Cross-language cross-modal retrieval method based on two levels of adaptable parameters

Abstract

The application discloses a cross-language cross-modal retrieval method based on adaptive parameters of two levels, which is divided into two stages. The first stage aims to realize cross-language migration between the source language and the target language. The second stage aims to realize cross-modal alignment between the target language and the image. In the first stage, the text encoder layer parameters are frozen, and a shared adapter module, a sentence-level adapter module and a token-level adapter module are added behind the text encoder layer of each layer of the target language branch. Then, the outputs of this layer are obtained by adding the weights after passing through the three modules respectively. The application uses a novel adaptive parameter generation adapter structure to realize cross-language cross-modal retrieval by using two levels of text information and shared text information.

Description

Technical Field

[0001] This invention relates to the field of cross-language and cross-modal retrieval, and more particularly to a cross-language and cross-modal retrieval method based on two levels of adaptable parameters. Background Technology

[0002] As the scale of data on the internet continues to grow, data types are becoming increasingly diverse. Users are no longer interested in a single data modality, and their retrieval needs are increasingly evolving from single-modality to cross-modal. Modality refers to the form of data expression, including text, images, video, and audio. This diversity and surge in quantity has spurred the demand for precise retrieval systems, enabling users to quickly find the content they need in the vast ocean of information. The rapid development of deep learning technology, especially in recent years, has greatly advanced the research and application of cross-modal retrieval technology.

[0003] Cross-modal retrieval involves searching between data from at least two modalities, typically using one modality as the query to retrieve relevant data from another modality. It achieves relatively accurate cross-matching by identifying potential associations between data from different modalities.

[0004] However, most current research on cross-modal retrieval focuses on using neural networks to process English data, because the paired data of English text and images / videos is not only abundant but also well-annotated. Cross-lingual cross-modal retrieval (CCR) becomes particularly important for building retrieval systems for other languages, especially for transferring resources from resource-rich source languages ​​(such as English) to target languages ​​that may lack sufficient or even no annotated data. While cross-modal retrieval has made significant progress in cross-lingual scenarios in recent years, a long-standing challenge for cross-modal retrieval systems is the lack of training data. Currently, mainstream annotated datasets are still limited to English, leading to a crucial issue: most languages ​​lack sufficient, high-quality annotated information. In non-English language domains, these languages ​​are referred to as low-resource languages ​​due to the lack of large-scale annotated datasets. Most CCR methods achieve cross-lingual transfer by constructing pseudo-parallel visual-language corpora using machine translation (MT). However, machine-translated sentences are generally inadequate in describing the corresponding visual content. Incorrectly assuming that pseudo-parallel data are correctly correlated will cause the network to overfit noisy correspondences. Furthermore, understanding multimodal and multilingual models allows for their application to a wider range of real-life scenarios. To build such multimodal and multilingual models, existing work attempts to integrate visual-language data from multiple languages ​​during pre-training. However, due to the large number of languages, these works often require enormous computational resources and cannot be flexibly extended to new languages, posing a significant challenge to training and improving cross-modal retrieval systems. To address this issue, some researchers have utilized multi-level alignment at the cross-linguistic stage and visual-target language alignment at the cross-modal stage. Researchers have introduced a single adapter in the target language branch for cross-linguistic transfer and cross-modal alignment, reducing computational cost and achieving good performance. However, this approach has the following drawbacks: 1. The ability of a single adapter is insufficient to fit some features unique to the language itself. 2. While the adapter saves resources, it only captures single features between samples, failing to consider features at different scales, leading to information loss. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing a cross-language and cross-modal retrieval method based on two levels of adaptive parameters. This method improves performance while maintaining a relatively small number of training parameters, thereby enabling cross-language and cross-modal retrieval.

[0006] The objective of this invention is achieved through the following technical solution: a cross-language, cross-modal retrieval method based on two levels of adaptable parameters. This method includes two training phases: a cross-language transfer phase and a cross-modal alignment phase, mainly comprising:

[0007] S1. Encode the features of the source language and the target language to obtain the embedding vector of the corresponding language text;

[0008] S2. Obtain the corresponding feature vector from the embedded vector through a text encoder. The text branch has two branches: the source language branch and the target language branch. Both branches contain a text encoder, which has several layers.

[0009] The source language branch is a pre-trained text encoder layer, and the obtained text information is used as the source language text output or the source language text input of the next layer.

[0010] The target language branch outputs the text from the pre-trained text encoder layer to the shared adapter module, sentence-level adapter module, and token-level adapter module, respectively. The text information output by the three modules is weighted to obtain the final text information, which is used as the target language text output or the target language input for the next layer.

[0011] S3. Based on the target language text features and source language text features output by the text encoder, the model is trained using MSE loss to achieve cross-language transfer.

[0012] S4. In the cross-modal alignment stage, the target language text is input into the trained text encoder to obtain the text features of the target language branch.

[0013] S5. Use a pre-trained image encoder to obtain image features, and use NCE loss calculation to realize a cross-modal alignment retrieval model of text features and image features;

[0014] S6. Input the text into the trained retrieval model to achieve cross-language and cross-modal retrieval.

[0015] Furthermore, the feature encoding of the source language and the target language includes: performing feature encoding on the source language text and the target language text to obtain the embedding vector of the corresponding language text.

[0016] Load the pre-trained mBERT tokenizer and perform token segmentation on the source and target language texts to obtain fixed-length tokenID vectors after segmentation. Where B represents the batch size and L represents the length of the word segmentation;

[0017] Using a multilingual word embedding matrix, word embedding operations are performed on the tokenID to obtain word embedding encoding vectors for the source language text and the target language text. Where H represents the hidden layer size, src represents the source language, and trg represents the target language;

[0018] Add positional encoding to both text word embedding vectors

[0019] Furthermore, the text encoder is the text encoder in the pre-trained CLIP model, and the image encoder is the image encoder in the pre-trained CLIP model. The text encoder includes 12 layers.

[0020] Furthermore, the shared adapter module specifically involves inputting text feature vectors into the ShareAdapter module to obtain common text information among text samples.

[0021] ShareAdapter = W upper (ReLU(W down X)+X

[0022]

[0023] in Where H represents the size of the hidden layer and I represents the size of the intermediate layer. This represents the matrix after the adapter has been applied.

[0024] Furthermore, the sentence-level adapter module includes: a selector, a downward projection layer, a nonlinear function layer, and an upward projection layer; specifically:

[0025] The [EOS] vector is selected based on the tokenID value of the obtained feature vector to obtain the global semantic information corresponding to the feature vector;

[0026] The global semantic information corresponding to the feature vector is input into the selector to obtain the global semantic related weight matrix and bias matrix; at the same time, the global semantic information corresponding to the feature vector is projected down, and after dimensionality reduction, the global semantic related feature matrix is ​​calculated through the output of the selector and input into the nonlinear function layer.

[0027] After passing through the nonlinear function layer, the output is again fed to the selector and the upprojection layer. The selector is used to obtain the weight matrix and the bias matrix. Then, the dot product of the weight matrix and the feature vector output by the nonlinear layer is added to the bias matrix to obtain the global semantically related feature matrix. The output is then passed through the upprojection layer to increase its dimension and then residually linked with the global semantic information to obtain the final output.

[0028] Furthermore, the operation of the selector specifically includes:

[0029] A selector is introduced into the input and noise is added. After normalization by softmax, the probability result of the selection is obtained. The value with the highest probability and the index are obtained from the selection result.

[0030] The obtained index is embedded to obtain a globally semantically related embedding vector. The embedding vector is multiplied by the maximum probability to make it differentiable, and the final globally semantically related unique pattern vector is obtained.

[0031] The obtained idiosyncratic pattern vectors are passed through a weighted linear layer to obtain a globally semantically related idiosyncratic pattern weight matrix, and the idiosyncratic pattern vectors are passed through a biased linear layer to obtain a globally semantically related idiosyncratic pattern bias matrix.

[0032] Furthermore, the token-level adapter module specifically involves: introducing the token-level information into a selector and adding noise, performing a normalization operation using softmax to obtain the probability result of selection, and obtaining the value and index of the highest probability from the selection result;

[0033] The value index is embedded and then multiplied with the probability vector to obtain the token-level unique pattern vector related to the input.

[0034] The token-level unique pattern vector is passed through a weighted linear layer and a biased linear layer to obtain a token-level weight matrix and a token-level bias matrix. Then, the dot product of the weight matrix and the token-level unique pattern vector is added to the bias matrix to obtain the final token-level feature matrix. After that, the token-level feature matrix is ​​increased in dimension by a linear layer, and the dimension is the same as the dimension before entering this module.

[0035] Furthermore, the weighting of the text information output by the three modules specifically involves:

[0036] Initialize a Fusion matrix. Concatenate the global semantic information-aware text features output by the sentence-level adapter module and the token-level text features output by the token-level adapter module along the third dimension to obtain the text features at both levels. Then, pass the Fusion matrix to obtain the fused text features.

[0037] Initialize the weight matrix G. The weight matrix will be updated as the model is trained. The weights of the common text features and the fused text features are added together.

[0038] Furthermore, the specific method for training the model using MSE loss is as follows:

[0039]

[0040] Where B is the batch size. Features of the source language text The weighted sum of the target language text features.

[0041] Furthermore, S5 specifically includes:

[0042] First, load the cross-language transfer-trained model, then crop the image from its original size to 224×224, then use a 2D convolutional layer to convert the input image into a feature map, and finally add positional encoding to obtain the initial features of the image.

[0043] The obtained initial features are input into the VisionTransformer of the pre-trained CLIP to obtain the final image features F. v ;

[0044] Calculate NCE loss for image and text features:

[0045]

[0046] F t To obtain the final target language text features by executing the target language branch in the text encoder.

[0047] The beneficial effects of this invention are:

[0048] 1. This invention freezes all parameters of the text encoder and introduces a shared adapter module, a global semantic information-aware adapter module, and a token-level aware adapter module for efficient parameter fine-tuning. The parameters of the sentence-level adapter module and the token-level adapter module are dynamically generated based on the text encoder's output. They can generate corresponding parameters according to different samples, find the unique patterns of the samples, and aim for the model to simultaneously consider the sentence context and more detailed token-level information for cross-language and cross-modal training. These two levels complement each other to obtain powerful text features with strong adaptability and reduced computational cost.

[0049] 2. This invention utilizes shared information between samples and sample-specific information to generate adaptive parameters, and integrates shared information and sample-specific information to improve cross-language and cross-modal retrieval capabilities. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of a cross-language, cross-modal retrieval method based on two levels of adaptable parameters according to the present invention.

[0051] Figure 2 This is a schematic diagram of the shared adapter module structure;

[0052] Figure 3 This is a schematic diagram of the sentence-level adaptive parameter generation adapter module structure of the present invention;

[0053] Figure 4 This is a schematic diagram of the token-level adaptive parameter generation adapter module structure of the present invention;

[0054] Figure 5 A schematic diagram of a cross-language, cross-modal retrieval device based on two levels of adaptable parameters provided in an embodiment of the present invention. Detailed Implementation

[0055] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0056] like Figure 1 As shown, the present invention provides a cross-language, cross-modal retrieval method based on two levels of adaptable parameters, the specific steps of which include:

[0057] S1. In the cross-language transfer stage, feature encoding is performed on the source language text and the target language text to obtain the embedding vector of the corresponding language text.

[0058] Load the pre-trained mBERT tokenizer and perform token segmentation on the source and target language texts to obtain fixed-length tokenID vectors after segmentation. Where B represents the batch size and L represents the length of the word segment.

[0059] Using a multilingual word embedding matrix, we perform word embedding operations on the previously obtained tokenID matrix to obtain word embedding encoding vectors for the source language text and the target language text. Where H represents the hidden layer size, src represents the source language, and trg represents the target language.

[0060] Add positional encoding to both text word embedding vectors

[0061] E src =Embedding(T src )+E pos

[0062] E trg =Embedding(T trg )+E pos

[0063] S2. Obtain the corresponding feature vector from the embedded vector through a text encoder. The text branch has two branches: the source language branch and the target language branch. Both branches contain a text encoder, which has several layers.

[0064] The source language branch is a pre-trained text encoder layer, and the obtained text information is used as the source language text output or the source language text input of the next layer.

[0065] The target language branch outputs the text from the pre-trained text encoder layer to the shared adapter module, sentence-level adapter module, and token-level adapter module, respectively. The text information output by the three modules is weighted to obtain the final text information, which is used as the target language text output or the target language input for the next layer.

[0066] The text encoder layer is specifically as follows:

[0067] Using the embedding vectors of the source and target language texts, the corresponding feature vectors are obtained through the text encoder of each layer of the pre-trained CLIP model. Where i represents the encoder of the i-th layer.

[0068]

[0069] here It is token-level and can be represented as

[0070] The shared adapter module is specifically:

[0071] like Figure 2 As shown, in the target language branch, the text feature vectors obtained by the text encoder are processed by a shared adapter module, ShareAdapter, to obtain the common text information among the text samples.

[0072] ShareAdapter(X) = W up (ReLU(W down X))+X

[0073]

[0074] in Where H represents the size of the hidden layer and I represents the size of the intermediate layer.

[0075] The sentence-level adapter module is specifically as follows:

[0076] The text feature vectors output by the text encoder layer are used in the target language branch. Text features with global semantic information perception are obtained after passing through a sentence-level adapter module. here The features represent sentence-level features of the target language, where trg_s represents the sentence-level features of the target language. This represents the characteristics after the adapter is applied.

[0077] Specifically, before entering the sentence-level adapter module, the feature vector after the text encoder is used. Based on the tokenID value, select the [EOS] vector from this feature vector to obtain the global semantic information corresponding to the feature vector.

[0078] like Figure 3 As shown, the sentence-level adapter module includes: a selector, a downward projection layer, a nonlinear function layer, and an upward projection layer; specifically:

[0079] The global semantic information corresponding to the feature vector is input into the selector to obtain the global semantic related weight matrix and bias matrix; at the same time, the global semantic information corresponding to the feature vector is projected down, and after dimensionality reduction, the global semantic related feature matrix is ​​calculated through the output of the selector and input into the nonlinear function layer.

[0080] After passing through the nonlinear function layer, the output is again fed to the selector and the upprojection layer. The weight matrix and bias matrix obtained by the selector are then used. The dot product of the weight matrix and the feature vector output by the nonlinear layer is then added to the bias matrix to obtain the global semantically related feature matrix. The output is then upscaled by the upprojection layer and residually linked with the global semantic information to obtain the final output.

[0081] The selector is specifically:

[0082] For the obtained global semantic information vector, a unique pattern vector that can represent the current input is adaptively selected based on the input. This unique pattern vector is adaptive and can dynamically change with different input samples to capture unique patterns in the samples. Then, noise is added to prevent extreme phenomena in the selection, such as all samples choosing the first one. After that, a softmax normalization operation is performed to obtain the selection probability results p1, p2, p3, p4...p k , where k represents k unique patterns.

[0083]

[0084] Where σ represents noise, R represents the selector, and p j (X) represents the probability of the j-th sample;

[0085] For the obtained selection results, retrieve the value with the highest probability and the index of that value.

[0086]

[0087] in This represents the maximum probability of the j-th sample. The index representing the maximum probability of the j-th sample;

[0088] The obtained indexes are embedded to obtain globally semantically relevant embedding vectors. The embedding vectors are then multiplied by the maximum probability of the samples to make them differentiable, resulting in the final globally semantically relevant pattern vectors. Here, S represents a unique pattern, and s represents the sentence level.

[0089] S s =Embedding(P index )P max

[0090] For the unique pattern vector S s After a weighted linear layer Obtain the global semantically related weight matrix For the unique pattern vector S s After a deviation linear layer Obtain the global semantic correlation bias matrix

[0091]

[0092] Here, 's' all represent sentence level.

[0093] The downward projection includes:

[0094] Because of the problem of a large number of parameters due to excessive dimensionality, we use a low-rank decomposition method, obtained using linear layer pairs. Dimensional reduction is performed to obtain the dimensionality-reduced result.

[0095]

[0096] in

[0097] The weight matrix obtained by the selector and the feature vector output by the text encoder are multiplied by a dot product, and then added to the bias matrix obtained by the selector to obtain the global semantically related feature matrix.

[0098]

[0099] The upward projection specifically includes:

[0100] After the feature matrix and weight matrix are multiplied by a nonlinear layer and added to the bias matrix, and because a low-rank decomposition was performed during the downward projection, the matrix is ​​then subjected to an upward projection linear layer for dimensionality increase before being combined with global semantic information. Perform residual connections to obtain the final global semantically relevant feature matrix.

[0101]

[0102] in Represents the characteristics after the adapter is applied.

[0103] Specifically, the token-level adapter module includes:

[0104] Utilizing text feature vectors from the text encoder layer in the target language branch The token-level information is obtained through a token-level adapter module, which then provides the text features. Here, 't' represents the token level. This represents the characteristics after the adapter is applied.

[0105] The processing of token-level unique pattern vectors is similar to that of sentence-level adapters. For the obtained token-level unique pattern vectors, a selector R adaptively selects the unique pattern vectors that best represent the current input. Noise is then added, followed by a softmax normalization operation to obtain the selection probability results p1, p2, p3, p4...p k Then, a TopK operation is performed to select the value with the highest probability and its index.

[0106]

[0107] The value index is embedded and then multiplied with the probability vector to obtain the token-level unique pattern vector S associated with the input. t .

[0108] S t =Embedding(P index )P max

[0109] The token-level unique pattern vector is processed through a weighted linear layer and a biased linear layer to obtain the token-level weight matrix and token-level bias matrix. The process is similar to that at the sentence level. Dimensionality reduction is performed, then the weight matrix and the reduced token-level specific pattern vector are multiplied by a dot product, and the bias matrix is ​​added to obtain the final token-level feature matrix. This token-level feature matrix is ​​then subjected to a linear layer for dimensionality increase, and then... Perform residual linking, with the same dimensions as before entering this module.

[0110]

[0111] here This represents the characteristics after the adapter is applied, and t represents the token level.

[0112] In the target language branch, shared text features obtained from different perception modules, global semantic information-perceived text features, and token-level text features are weighted and summed. Specifically:

[0113] Initialize a Fusion matrix W fusion The global semantic information-aware text features and token-level text features are concatenated in the third dimension to obtain two levels of text features, which are then processed by a Fusion matrix to obtain the fused text features. The goal of fusing the two matrices is to enable the model to learn both global semantic information and token-level information simultaneously.

[0114]

[0115] The weight matrix G is randomly initialized and updated as the model trains. The weights of the shared text features and the fused text features are summed.

[0116]

[0117] S3. Based on the target language text features and source language text features output by the text encoder, the model is trained using MSE loss to achieve cross-language transfer.

[0118]

[0119] Where B is the batch size, F src In order to feature the source language, F src For target language features.

[0120] S4. In the cross-modal alignment stage, load the model trained in the cross-language stage, input the target language text into the text encoder, and obtain the text features F of the target language branch. t ;

[0121] S5. Use a pre-trained image encoder to obtain image features, and use NCE loss calculation to implement a cross-modal alignment retrieval model for text features and image features.

[0122] Specifically, in the image branch, the image is first cropped from its original size to 224×224. Then, a 2D convolutional layer is used to convert the input image into a feature map. Finally, positional encoding is added to obtain the initial features of the image. B represents the batch size, and G×W represents the feature size.

[0123] The obtained initial features are input into the VisionTransformer of the pre-trained CLIP to obtain the final image features F. v .

[0124] The obtained image and text features are subjected to NCE loss calculation, and a cross-modal aligned retrieval model is trained:

[0125]

[0126] S6. Input the text into the trained retrieval model to achieve cross-language and cross-modal retrieval.

[0127] Corresponding to the aforementioned embodiment of a cross-language, cross-modal retrieval method based on two levels of adaptive parameters, the present invention also provides an embodiment of a cross-language, cross-modal retrieval device based on two levels of adaptive parameters.

[0128] Referring to Table 1, the experimental results of this invention and current cross-language, cross-modal retrieval methods under a zero-shot setting are shown. Zero-shot refers to the results of directly evaluating the model without fine-tuning on a downstream dataset. `trans` represents the machine-translated text data of the corresponding dataset. Because the alignment between images and text in the previous CC300K dataset was not high, it can be seen that the evaluation scores of this invention's cross-language, cross-modal retrieval method under the zero-shot setting are better than previous methods. This is because this invention can fully retain CLIP's strong performance in English, while traditional multilingual visual language models typically perform worse in English than their monolingual versions. It not only performs excellently in English but also surpasses previous methods in other languages.

[0129] Table 1

[0130]

[0131] Referring to Figure 2, this figure shows the experimental results of the present invention and current cross-language cross-modal retrieval methods under the condition of fine-tuning only the English setting. Fine-tuning only the English setting means that the visual language pre-trained model is first fine-tuned on English data, that is, the model is further trained on the downstream English dataset to adapt to the specific task and optimize its performance on English data. trans represents the machine translation data of the text in the corresponding dataset. Because the alignment between images and text in the previous CC300K data was not high, the two levels of adaptable parameters proposed in this invention were then added to this fine-tuned model to generate the cross-language cross-modal retrieval method. It can be seen that the method proposed in this invention is very competitive and has a significant performance improvement.

[0132] Table 2

[0133]

[0134] See Figure 5The present invention provides a cross-language and cross-modal retrieval device based on two levels of adaptive parameters, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement a cross-language and cross-modal retrieval method based on two levels of adaptive parameters as described in the above embodiment.

[0135] This invention provides an embodiment of a cross-language, cross-modal retrieval device based on two levels of adaptable parameters. This device can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data-processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 5 The diagram shown is a hardware structure diagram of any data processing-capable device that provides a cross-language, cross-modal retrieval device based on two levels of adaptable parameters, as provided by the present invention. (Except for...) Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0136] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0137] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0138] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements a cross-language, cross-modal retrieval method based on two levels of adaptable parameters as described in the above embodiments.

[0139] The computer-readable storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0140] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters.

[0141] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0142] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters, characterized in that, The method includes: Cross-language transfer phase: Feature encoding is performed on the source language and the target language to obtain the embedding vector of the corresponding language text; The embedded vector is processed by a text encoder to obtain the corresponding feature vector. The text branch has two branches: the source language branch and the target language branch. Both branches contain a text encoder, which consists of several layers. The source language branch is a pre-trained text encoder layer, and the obtained text information is used as the source language text output or the source language text input of the next layer. The target language branch is to output the text information of each layer of the pre-trained text encoder to the shared adapter module, the sentence-level adapter module and the token-level adapter module respectively. The text information output by the three modules is weighted to obtain the final text information, which is used as the target language text output or the target language input of the next layer. The shared adapter module specifically functions to input text feature vectors into the shared adapter module. Obtain the common text information among text samples : in , Where H represents the size of the hidden layer, I represents the size of the intermediate layer, and i represents the encoder of the i-th layer; The sentence-level adapter module includes: a selector, a down-projection layer, a non-linear function layer, and an up-projection layer, specifically: Based on the obtained feature vector The value is used to select the [EOS] vector, and the global semantic information corresponding to the feature vector is obtained; The global semantic information corresponding to the feature vector is input into the selector to obtain the global semantic related weight matrix and bias matrix; at the same time, the global semantic information corresponding to the feature vector is projected down, and after dimensionality reduction, the global semantic related feature matrix is ​​calculated through the output of the selector and input into the nonlinear function layer. After passing through the nonlinear function layer, the output is again fed to the selector and the upprojection layer. The weight matrix and bias matrix obtained by the selector are used. Then, the dot product of the weight matrix and the feature vector output by the nonlinear layer is added to the bias matrix to obtain the global semantic related feature matrix. The output is then passed through the upprojection layer to increase the dimension and then residually connected with the global semantic information to obtain the final output. The token-level adapter module specifically involves: introducing a selector and adding noise to the token-level feature matrix, followed by... After performing normalization, the probability results of the selection are obtained, and the value with the highest probability and its index are obtained from the selection results. The value index is embedded and then multiplied with the probability vector to obtain the token-level unique pattern vector related to the input. The token-level unique pattern vector is passed through a weighted linear layer and a biased linear layer to obtain the token-level weight matrix and the token-level bias matrix. Then, the dot product of the weight matrix and the token-level unique pattern vector is added to the bias matrix to obtain the final token-level feature matrix. After that, the token-level feature matrix is ​​passed through a linear layer to increase its dimension and then residually connected with the feature matrix that entered the selector at the beginning. The dimension is the same as the dimension before entering this module. The weighting of the text information output by the three modules specifically involves: Initialize a Fusion matrix. Concatenate the global semantic information-aware text features output by the sentence-level adapter module and the token-level text features output by the token-level adapter module in the third dimension to obtain text features at two levels. Then, pass the Fusion matrix to obtain the fused text features. Initialize the weight matrix The weight matrix is ​​updated as the model is trained, by adding the weights of the shared text features and the fused text features. Based on the target language text features and source language text features output by the text encoder, the model is trained using MSE loss to achieve cross-language transfer. Cross-modal alignment stage: The target language text is input into the trained text encoder to obtain the text features of the target language branch; Image features are obtained using a pre-trained image encoder, and a cross-modal alignment retrieval model for text features and image features is implemented using NCE loss calculation. Text is input into a trained retrieval model to achieve cross-language and cross-modal retrieval.

2. The method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters according to claim 1, characterized in that, The feature encoding of the source language and the target language includes: performing feature encoding on the source language text and the target language text to obtain the embedding vector of the corresponding language text; Load the pre-trained mBERT tokenizer and perform token segmentation on the source and target language texts to obtain fixed-length tokenID vectors after segmentation. , Where B represents the batch size and L represents the length of the word segment; Using a multilingual word embedding matrix, word embedding operations are performed on the tokenID to obtain word embedding encoding vectors for the source language text and the target language text. , Where H represents the size of the hidden layer. Represents the source language. Represents the target language; Add positional encoding to both text word embedding vectors .

3. The method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters according to claim 1, characterized in that, The text encoder is the text encoder in the pre-trained CLIP model, which includes 12 layers. The image encoder is the image encoder in the pre-trained CLIP model.

4. The method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters according to claim 1, characterized in that, The operation of the selector specifically includes: A selector is introduced into the input and noise is added, then... After performing normalization, the probability results of the selection are obtained, and the value with the highest probability and its index are obtained from the selection results. The obtained index is embedded to obtain a globally semantically related embedding vector. The embedding vector is multiplied by the maximum probability to make it differentiable, and the final globally semantically related unique pattern vector is obtained. The obtained idiosyncratic pattern vectors are passed through a weighted linear layer to obtain the globally semantically related idiosyncratic pattern weight matrix, and the idiosyncratic pattern vectors are passed through a biased linear layer to obtain the globally semantically related idiosyncratic pattern bias matrix.

5. The method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters according to claim 1, characterized in that, The specific method of using MSE loss to train the model is as follows: Where B is the batch size. Features of the source language text Features of the target language text after fusion.

6. The method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters according to claim 5, characterized in that, The retrieval model that uses a pre-trained image encoder to acquire image features and uses NCE loss to calculate cross-modal alignment of text and image features is specifically as follows: In the cross-modal alignment stage, the model trained in the cross-language transfer stage is first loaded, and the image is reduced and cropped from the original image size to 224×224. Then, a 2D convolutional layer is used to convert the input image into a feature map, and then positional encoding is added to obtain the initial features of the image. The initial features are input into the VisionTransformer of the pre-trained CLIP to obtain the final image features. ; Calculate NCE loss for image and text features: To obtain the final target language text features by executing the target language branch in the text encoder.

Images