Method for generating cross-language cross-modal retrieval based on adaptive parameters of two levels
By adopting two levels of adaptable parameter generation methods in the cross-language cross-modal retrieval system, combining shared adapter and sentence/token-level adapter, the problem of insufficient data in the low-resource language field is solved, and efficient cross-language cross-modal is achieved Improved state retrieval performance.
Patent Information
- Application Number
- CN202410927779.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-11
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-11
AI Technical Summary
The existing cross-language and cross-modal retrieval system lacks sufficient manual annotation data in the low-resource language field, resulting in insufficient model training data, and a single adapter cannot effectively fit multilingual characteristics and different scale features, resulting in information loss and waste of computing resources.
Two levels of adaptable parameter generation methods are adopted, and through the cross-language migration stage and the cross-modal alignment stage, the shared adapter module, the sentence-level adapter module and the token-level adapter module are used to dynamically generate parameters, combining shared information and sample-specific information. Implement cross-language and cross-modal retrieval.
While maintaining less training parameters, cross-language cross-modal retrieval performance is improved, computational volume is reduced, and the adaptability and cross-language cross-modal retrieval capability of the model are enhanced.
Smart Images

Figure CN118939859A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cross-language and cross-modal retrieval, and in particular to a cross-language and cross-modal retrieval method based on two levels of adaptable parameters. Background Art
[0002] As the scale of data on the Internet continues to grow, data types are becoming more and more diverse. The data modality that users are interested in is no longer single, and users' retrieval needs are increasingly showing a trend from single modality to cross-modality. Modality refers to the form of expression of data, including text, images, videos, and audio. The diversity and surge in the number of these contents have given rise to the need for accurate retrieval systems so that users can quickly find what they need in the vast ocean of information. The rapid development of deep learning technology, especially in recent years, has greatly promoted the research and application of cross-modal retrieval technology.
[0003] Cross-modal retrieval is the mutual retrieval of data from at least two modalities, usually using one modality as a query to retrieve related data from another modality. By finding the potential associations between data from different modalities, relatively accurate cross-matching can be achieved.
[0004] However, most of the current research on cross-modal retrieval focuses on processing English data using neural networks, because the paired data of English text and images / videos is not only large in quantity but also has been fully manually annotated. In order to build retrieval systems for other languages, cross-lingual cross-modal retrieval (CCR) becomes particularly important, especially to transfer resources from resource-rich source languages (such as English) to target languages, which may lack sufficient manually annotated data or even have no available annotated resources at all. Although cross-modal retrieval has made significant progress in cross-language scenarios in recent years, a long-term challenge facing cross-modal retrieval systems is the lack of training data. The current mainstream manually annotated datasets are still limited to English, which leads to an important problem: the data in most languages lack sufficient, high-quality manually labeled information. In the field of non-English languages, these languages are called low-resource languages due to the lack of large-scale manually annotated datasets. Most CCR methods achieve cross-language transfer by constructing pseudo-parallel visual-linguistic corpora through machine translation (MT). However, the translated sentences of machine translation are generally imperfect in describing the corresponding visual content. Incorrectly assuming that pseudo-parallel data is correctly correlated will cause the network to overfit the noisy correspondence. And understanding multimodal and multilingual models can be applied to a wider range of real-life scenarios. In order to build such multimodal and multilingual models, existing work attempts to integrate visual language data from multiple languages in pre-training. However, due to the large number of languages, these works often require huge computing resources and cannot be flexibly extended to new languages, which poses a major challenge to training and improving cross-modal retrieval systems. To solve this problem, some researchers use multi-level alignment in the cross-lingual stage and visual-target language alignment in the cross-modal stage. The researchers introduced a single adapter in the target language branch for cross-language transfer and cross-modal alignment, which reduced the amount of calculation and achieved good performance, but this method has the following defects: 1. The capacity of a single adapter is not enough to fit some characteristics unique to the language itself. 2. Although the adapter saves resources, it only captures a single feature between samples, and does not consider features of different scales, resulting in information loss. Summary of the invention
[0005] The purpose of the present invention is to address the deficiencies of the prior art and propose a cross-language and cross-modal retrieval method based on two levels of adaptable parameters to improve performance while maintaining a relatively small amount of training parameters, thereby achieving cross-language and cross-modal retrieval.
[0006] The object of the present invention is achieved through the following technical solution: a method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters, the method comprising two stages of training, the first stage being a cross-language migration stage, and the second stage being a cross-modal alignment stage, mainly comprising:
[0007] S1. Encode the features of the source language and the target language to obtain the embedding vector of the corresponding language text;
[0008] S2. Obtain a corresponding feature vector from the embedded vector through a text encoder. The text branch has two branches, a source language branch and a target language branch. Both branches contain a text encoder, and the text encoder has several layers.
[0009] The source language branch is a pre-trained text encoder layer, and the obtained text information is used as the source language text output or the next layer of source language text information input;
[0010] The target language branch is output to the shared adapter module, the sentence-level adapter module and the token-level adapter module in the pre-trained text encoder layer, and the text information output by the three modules is weighted to obtain the final text information as the target language text output or the target language input of the next layer;
[0011] S3. Based on the target language text features and source language text features output by the text encoder, the MSE loss is used to train the model to achieve cross-language migration.
[0012] S4, in the cross-modal alignment stage, input the target language text into the trained text encoder to obtain the text features of the target language branch;
[0013] S5. Use the pre-trained image encoder to obtain image features, and use NCE loss to calculate the retrieval model that realizes cross-modal alignment of text features and image features;
[0014] S6. Input the text into the trained retrieval model to achieve cross-language and cross-modal retrieval.
[0015] Furthermore, the feature encoding of the source language and the target language includes: feature encoding the source language text and the target language text to obtain embedding vectors of the corresponding language texts.
[0016] Load the pre-trained mBERT tokenizer and perform token segmentation on the source language text and the target language text to obtain a fixed-length tokenID vector after token segmentation. Where B represents the batch size, and L represents the length of the word segmentation;
[0017] Use a multilingual word embedding matrix to perform word embedding operations on tokenID to obtain the word embedding encoding vectors of the source language text and the target language text Where H represents the hidden layer size, src represents the source language, and trg represents the target language;
[0018] Add position encoding to both text word embedding vectors
[0019] Furthermore, the text encoder is a text encoder in a pre-trained model CLIP, the image encoder is an image encoder in a pre-trained CLIP, and the text encoder includes 12 layers.
[0020] Furthermore, the shared adapter module specifically includes: inputting the text feature vector into the shared adapter module ShareAdapter to obtain the text information shared by the text samples
[0021] ShareAdapter = W upper (ReLU(W down X)+X
[0022]
[0023] in Where H represents the hidden layer size, I represents the intermediate layer size, Represents the matrix after passing through the adapter.
[0024] Furthermore, the sentence-level adapter module includes: a selector, a downward projection layer, a nonlinear function layer and an upward projection layer; specifically:
[0025] Select the [EOS] vector according to the value of tokenID for the obtained feature vector to obtain the global semantic information corresponding to the feature vector;
[0026] The global semantic information corresponding to the feature vector is input into the selector to obtain the weight matrix and bias matrix related to the global semantics. At the same time, the global semantic information corresponding to the feature vector is projected downward, and after dimension reduction, the feature matrix related to the global semantics is calculated through the output of the selector and input into the nonlinear function layer.
[0027] After passing through the nonlinear function layer, it is output to the selector and the upward projection layer respectively. The selector is used to obtain the weight matrix and the bias matrix. Then, the dot product operation of the weight matrix and the feature vector output by the nonlinear layer is performed, and the dot product result is added to the bias matrix to obtain the feature matrix related to global semantics. The output is then dimensionalized by the upward projection layer and residually linked with the global semantic information to obtain the final output.
[0028] Furthermore, the operation of the selector specifically includes:
[0029] Introduce a selector to the input and add noise, then normalize it with softmax to get the probability of selection, and get the value and index with the highest probability for the selection result;
[0030] The obtained index is embedded to obtain an embedding vector related to global semantics, and the embedding vector is multiplied by the maximum probability value to make it differentiable, so as to obtain the final unique pattern vector related to global semantics;
[0031] The obtained unique pattern vector is passed through a weight linear layer to obtain a unique pattern weight matrix related to global semantics, and the unique pattern vector is passed through a bias linear layer to obtain a unique pattern bias matrix related to global semantics.
[0032] Furthermore, the token-level adapter module specifically includes: introducing a selector into the token-level information and adding noise, obtaining a probability result of the selection after performing a normalization operation through softmax, and obtaining a value and index with the maximum probability for the selection result;
[0033] The value index is embedded and then multiplied by the probability vector to obtain the unique pattern vector at the token level related to the input;
[0034] The token-level unique pattern vector passes through the weight linear layer and the bias linear layer to obtain the token-level weight matrix and the token-level bias matrix. Then the dot product of the weight matrix and the token-level unique pattern vector plus the bias matrix is added to obtain the final token-level feature matrix. After that, the token-level feature matrix passes through a linear layer to increase the dimension, and the dimension is the same as before entering this module.
[0035] Furthermore, the text information output by the three modules is weighted as follows:
[0036] Initialize a Fusion matrix, concatenate the global semantic information perception text features output by the sentence-level adapter module and the token-level text features output by the token-level adapter module in the third dimension, obtain two levels of text features, and then pass them through the Fusion matrix to obtain the fused text features.
[0037] Initialize the weight matrix G, which will be updated as the model is trained, and add the weights of the common text features and the fused text features.
[0038] Furthermore, the use of MSE loss to train the model is specifically as follows:
[0039]
[0040] Where B is the batch size, is the source language text feature, It is the target language text feature after weight addition.
[0041] Furthermore, the S5 is specifically:
[0042] First, load the model trained for cross-language transfer, then reduce and crop the image from the original image size to 224×224, then use a 2D convolutional layer to convert the input image into a feature map, and then add position encoding to obtain the initial features of the image;
[0043] The obtained initial features are input into the pre-trained CLIP VisionTransformer to the final image features F v ;
[0044] Calculate NCE loss for image features and text features:
[0045]
[0046] F t To execute the target language branch in the text encoder, the final target language text features are obtained.
[0047] Beneficial effects of the present invention:
[0048] 1. The present invention freezes all parameters of the text encoder, introduces a shared adapter module, a global semantic information perception adapter module and a token-level perception adapter module to fine-tune the parameters for high efficiency. The parameters of the sentence-level adapter module and the token-level adapter module are dynamically generated based on the output of the text encoder, and can generate corresponding parameters according to different samples to find the unique pattern of the sample. It is hoped that the model can simultaneously consider the context of the sentence and the more detailed token-level information for cross-language and cross-modal training. The two levels complement each other to obtain the final powerful text features, with strong adaptability and can reduce the amount of calculation.
[0049] 2. The present invention generates adaptive parameters with the help of the shared information between samples and the sample's own unique information, and integrates the shared information and the sample's unique information to improve the cross-language and cross-modal retrieval capabilities. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] Figure 1 A schematic diagram of a flow chart of a cross-language and cross-modal retrieval method based on two levels of adaptable parameters of the present invention;
[0051] Figure 2 It is a schematic diagram of the structure of the shared adapter module;
[0052] Figure 3 A schematic diagram of the structure of an adaptable parameter generation adapter module at the sentence level of the present invention;
[0053] Figure 4 A schematic diagram of the structure of an adapter module for generating adaptable parameters at the token level of the present invention;
[0054] Figure 5 A schematic diagram of a cross-language and cross-modal retrieval device based on two levels of adaptable parameters provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0055] The specific implementation modes of the present invention are further described in detail below with reference to the accompanying drawings.
[0056] like Figure 1 As shown, the present invention provides a method for generating a cross-language and cross-modal retrieval based on two levels of adaptable parameters, and the specific steps include:
[0057] S1. In the cross-language transfer stage, feature encoding is performed on the source language text and the target language text to obtain the embedding vector of the corresponding language text.
[0058] Load the pre-trained mBERT tokenizer and perform token segmentation on the source language text and the target language text to obtain a fixed-length tokenID vector after token segmentation. Where B represents the batch size and L represents the length of the word segmentation.
[0059] Use a multilingual word embedding matrix to perform word embedding operations on the previously obtained tokenID matrix to obtain the word embedding encoding vectors of the source language text and the target language text Where H represents the hidden layer size, src represents the source language, and trg represents the target language.
[0060] Add position encoding to both text word embedding vectors
[0061] E src =Embedding(T src )+E pos
[0062] E trg =Embedding(T trg )+E pos
[0063] S2. Obtain a corresponding feature vector from the embedded vector through a text encoder. The text branch has two branches, a source language branch and a target language branch. Both branches contain a text encoder, and the text encoder has several layers.
[0064] The source language branch is a pre-trained text encoder layer, and the obtained text information is used as the source language text output or the next layer of source language text information input;
[0065] The target language branch is output to the shared adapter module, the sentence-level adapter module and the token-level adapter module in the pre-trained text encoder layer, and the text information output by the three modules is weighted to obtain the final text information as the target language text output or the target language input of the next layer;
[0066] The text encoder layer is specifically:
[0067] The obtained embedding vectors of the source language text and the target language text are used to obtain the corresponding feature vectors through the text encoder of each layer of the pre-trained model CLIP. Where i represents the encoder of the i-th layer.
[0068]
[0069] here It is at the token level and can be expressed as
[0070] The shared adapter module is specifically:
[0071] like Figure 2 As shown in the figure, the text feature vector obtained by the text encoder in the target language branch is passed through a shared adapter module ShareAdapter to obtain the common text information between text samples.
[0072] ShareAdapter(X) = W up (ReLU(W down X))+X
[0073]
[0074] in Where H represents the hidden layer size and I represents the intermediate layer size.
[0075] The sentence level adapter module is specifically:
[0076] In the target language branch, use the text feature vector output by the text encoder layer The text features with global semantic information perception are obtained through a sentence-level adapter module. here represents the target language sentence level features, trg_s represents the target language sentence level, Represents the characteristics after passing through the adapter.
[0077] Specifically, before entering the sentence-level adapter module, the feature vector after the text encoder is used For this feature vector, select the [EOS] vector according to the value of tokenID to obtain the global semantic information corresponding to the feature vector
[0078] like Figure 3 As shown, the sentence-level adapter module includes: a selector, a downward projection layer, a nonlinear function layer and an upward projection layer; specifically:
[0079] The global semantic information corresponding to the feature vector is input into the selector to obtain the weight matrix and bias matrix related to the global semantics. At the same time, the global semantic information corresponding to the feature vector is projected downward, and after dimension reduction, the feature matrix related to the global semantics is calculated through the output of the selector and input into the nonlinear function layer.
[0080] After passing through the nonlinear function layer, it is output to the selector and the upward projection layer respectively. The weight matrix and bias matrix are obtained by the selector. The dot product operation of the weight matrix and the feature vector output by the nonlinear layer is then added to the bias matrix to obtain the feature matrix related to global semantics. The output is then dimensionalized by the upward projection layer and residually linked with the global semantic information to obtain the final output.
[0081] The selector is specifically:
[0082] For the obtained global semantic information vector, a unique pattern vector that can represent the current input is adaptively selected according to the input. Such a unique pattern vector is adaptive and can dynamically change with different input samples to capture the unique pattern in the sample. Then, noise is added to avoid extreme selection phenomena, such as focusing on the first one. After that, a softmax is normalized to obtain the probability results of the selection p1, p2, p3, p4...p k , k represents k unique patterns.
[0083]
[0084] Where σ represents noise, R represents the selector, and p j (X) represents the probability of the jth sample;
[0085] For the obtained selection results, obtain the value with the highest probability and the index of the value.
[0086]
[0087] in represents the maximum probability of the jth sample, The index representing the maximum probability of the jth sample;
[0088] The obtained index is embedded to obtain the global semantically relevant embedding vector, and the embedding vector is multiplied by the maximum probability of the sample to make it differentiable, and the final global semantically relevant unique pattern vector is obtained. Here S stands for unique mode and s stands for sentence level.
[0089] S s =Embedding(P index ) max
[0090] For the unique mode vector S s After a weighted linear layer Get the weight matrix of global semantic correlation For the unique mode vector S s After a biased linear layer Obtaining the global semantic correlation deviation matrix
[0091]
[0092] The s here stands for sentence level.
[0093] The downward projection includes:
[0094] Because of the large number of parameters caused by the large dimension, we use the low-rank decomposition method to obtain the linear layer. Perform dimensionality reduction to obtain the reduced dimensionality
[0095]
[0096] in
[0097] The weight matrix obtained by the selector is used to perform a dot multiplication operation with the feature vector output by the text encoder, and then added to the deviation matrix obtained by the selector to obtain a feature matrix related to global semantics.
[0098]
[0099] The upward projection specifically includes:
[0100] After the dot multiplication operation of the feature matrix and the weight matrix of the nonlinear layer, and then added to the bias matrix, because the low-rank decomposition operation is performed during the downward projection process, this side is further dimensionalized by an upward projection linear layer and the global semantic information is added. Perform residual connection to obtain the final global semantically relevant feature matrix.
[0101]
[0102] in Represents the characteristics after passing through the adapter,
[0103] Specifically, the token level adapter module includes:
[0104] Use the text feature vector of the text encoder layer in the target language branch Token-level information-aware text features are obtained through a token-level adapter module Here t represents the token level. Represents the characteristics after passing through the adapter.
[0105] The processing of the token-level unique pattern vector is similar to that of the sentence-level adapter. For the obtained token-level unique pattern vector, the selector R can adaptively select the unique pattern vector that can represent the current input according to the input, and then add noise. After that, a softmax is performed for normalization to obtain the selected probability results p1, p2, p3, p4...p k , and then perform a TopK operation to select the value with the highest probability and the index of the value.
[0106]
[0107] The value index is embedded and then multiplied by the probability vector to obtain the unique pattern vector S at the token level related to the input. t .
[0108] S t =Embedding(P index ) max
[0109] The token-level unique pattern vector is passed through the weight linear layer and the bias linear layer to obtain the token-level weight matrix and the token-level bias matrix. The same is true for the sentence level. The dimensionality reduction operation is performed, and then the weight matrix and the reduced-dimensional token-level unique pattern vector dot product are added to the bias matrix to obtain the final token-level feature matrix. After that, the token-level feature matrix is increased in dimension by a linear layer and Perform residual linking, the dimension is the same as before entering this module.
[0110]
[0111] here Represents the features after passing through the adapter, and t represents the token level.
[0112] In the target language branch, the common text features, global semantic information perception text features and token-level text features obtained from different perception modules are weighted and added. The details are as follows:
[0113] Initialize a Fusion matrix W fusion , the global semantic information perception text features and token level text features are spliced in the third dimension to obtain two levels of text features, and then the fused text features are obtained through the Fusion matrix The fusion of the two matrices is intended to enable the model to learn both global semantic information and token-level information.
[0114]
[0115] The weight matrix G is randomly initialized. The weight matrix will be updated as the model is trained, and the weights of the common text features and the fused text features are added.
[0116]
[0117] S3. Based on the target language text features and source language text features output by the text encoder, the MSE loss is used to train the model to achieve cross-language migration.
[0118]
[0119] Where B is the batch size, F src is the source language feature, F src for features in the target language.
[0120] S4. In the cross-modal alignment stage, load the model trained in the cross-language stage, input the target language text into the text encoder, and obtain the text feature F of the target language branch. t ;
[0121] S5. Use the pre-trained image encoder to obtain image features, and use NCE loss to calculate the retrieval model that implements cross-modal alignment of text features and image features.
[0122] Specifically, in the image branch, the image is first reduced and cropped from the original image size to 224×224, and then a 2D convolutional layer is used to convert the input image into a feature map, and then position encoding is added to obtain the initial features of the image. B is the batch size and G×W represents the feature size.
[0123] The obtained initial features are input into the pre-trained CLIP VisionTransformer to the final image features F v .
[0124] The NCE loss is calculated for the obtained image features and text features, and the cross-modal aligned retrieval model is trained:
[0125]
[0126] S6. Input the text into the trained retrieval model to achieve cross-language and cross-modal retrieval.
[0127] Corresponding to the aforementioned embodiment of a method for generating a cross-language and cross-modal retrieval based on two levels of adaptable parameters, the present invention further provides an embodiment of a device for generating a cross-language and cross-modal retrieval based on two levels of adaptable parameters.
[0128] Refer to Table 1, which is a graph of the experimental results of the present invention and the current cross-language and cross-modal retrieval method under the zero-sample setting. The zero-sample is the result of directly evaluating the model without fine-tuning on the downstream dataset. trans is the machine translation data of the text of the corresponding dataset. Because the alignment degree of the image and text in the previous CC300K data is not high, it can be seen that the evaluation scores of the cross-language and cross-modal retrieval method of the present invention under the zero-sample setting are worse than the previous methods. This is because the present invention can fully retain the powerful performance of CLIP in English, while traditional multilingual visual language models usually perform worse in English than their single-language versions. It not only performs well in English, but also surpasses previous methods in other languages.
[0129] Table 1
[0130]
[0131] Refer to Figure 2, which is an experimental result diagram of the present invention and the current cross-language and cross-modal retrieval method under the fine-tuning English setting only. Fine-tuning the English setting only means first fine-tuning the visual language pre-trained model for English data, that is, further training the model on the English downstream dataset to adapt to specific tasks and optimize its performance on English data. trans is the machine translation data of the text of the corresponding dataset. Because the alignment degree of pictures and texts in the previous CC300K data is not high, the two levels of adaptable parameters proposed by the present invention are added to this fine-tuned model to generate a cross-language and cross-modal retrieval method. It can be seen that the method proposed by the present invention is very competitive and the performance is greatly improved.
[0132] Table 2
[0133]
[0134] See also Figure 5An embodiment of the present invention provides a device for generating a cross-language and cross-modal retrieval based on two levels of adaptive parameters, including a memory and one or more processors, wherein the memory stores executable code, and when the processor executes the executable code, it is used to implement a method for generating a cross-language and cross-modal retrieval based on two levels of adaptive parameters in the above embodiment.
[0135] An embodiment of a cross-language and cross-modal retrieval device based on two levels of adaptive parameters provided by the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 5 As shown, it is a hardware structure diagram of any device with data processing capability where a cross-language and cross-modal retrieval device based on two levels of adaptable parameters provided by the present invention is located. Figure 5 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiments is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.
[0136] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0137] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.
[0138] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, a method for generating a cross-language and cross-modal retrieval method based on two levels of adaptable parameters in the above embodiment is implemented.
[0139] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.
[0140] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the cross-language and cross-modal retrieval method based on two levels of adaptable parameters.
[0141] Those skilled in the art will readily appreciate other embodiments of the present application after considering the description and practicing the contents disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The description and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.
[0142] It should be understood that the above general description and the detailed description below are only exemplary and explanatory and cannot limit the present application. The present application is not limited to the precise structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope thereof. The scope of the present application is limited only by the attached claims.
Claims
1. A cross-language and cross-modal retrieval method based on two levels of adaptable parameters, characterized in that: The method includes: Cross-language migration phase: Encode the features of the source language and the target language to obtain the embedding vectors of the corresponding language texts; The embedding vector is subjected to a text encoder to obtain a corresponding feature vector. The text branch has two branches, a source language branch and a target language branch. Both branches contain a text encoder, and the text encoder has several layers. The source language branch is a pre-trained text encoder layer, and the obtained text information is used as the source language text output or the next layer of source language text information input; The target language branch is output to the shared adapter module, the sentence-level adapter module and the token-level adapter module at each layer of the pre-trained text encoder, and the text information output by the three modules is weighted to obtain the final text information as the target language text output or the target language input of the next layer; According to the target language text features and source language text features output by the text encoder, the MSE loss is used to train the model to achieve cross-language migration; Cross-modal alignment phase: Input the target language text into the trained text encoder to obtain the text features of the target language branch; Use the pre-trained image encoder to obtain image features, and use the NCE loss to calculate the retrieval model that implements cross-modal alignment of text features and image features; Input the text into the trained retrieval model to achieve cross-language and cross-modal retrieval.
2. According to claim 1, a method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters is characterized in that: The feature encoding of the source language and the target language includes: feature encoding the source language text and the target language text to obtain an embedding vector of the corresponding language text; Load the pre-trained mBERT tokenizer and perform token segmentation on the source language text and the target language text to obtain a fixed-length tokenID vector after token segmentation. Where B represents the batch size, and L represents the length of the word segmentation; Use a multilingual word embedding matrix to perform word embedding operations on tokenID to obtain the word embedding encoding vectors of the source language text and the target language text Where H represents the hidden layer size, src represents the source language, and trg represents the target language; Add position encoding to both text word embedding vectors 3. According to claim 1, a method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters is characterized in that: The text encoder is a text encoder in a pre-trained model CLIP, and the text encoder includes 12 layers. The image encoder is an image encoder in a pre-trained CLIP.
4. According to claim 1, a method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters is characterized in that: The shared adapter module specifically inputs the text feature vector into the shared adapter module ShareAdapter to obtain the text information shared by the text samples. ShareAdapter=W upper (ReLU(W down X)+X in Where H represents the hidden layer size and I represents the intermediate layer size.
5. According to claim 1, a method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters, characterized in that: The sentence-level adapter module includes: selector, down-projection layer, nonlinear function layer and up-projection layer, specifically: Select the [EOS] vector according to the value of tokenID for the obtained feature vector to obtain the global semantic information corresponding to the feature vector; The global semantic information corresponding to the feature vector is input into the selector to obtain the weight matrix and bias matrix related to the global semantics. At the same time, the global semantic information corresponding to the feature vector is projected downward, and after dimension reduction, the feature matrix related to the global semantics is calculated through the output of the selector and input into the nonlinear function layer. After passing through the nonlinear function layer, it is output to the selector and the upward projection layer respectively. The weight matrix and bias matrix are obtained by the selector. The dot product operation of the weight matrix and the feature vector output by the nonlinear layer is then added to the bias matrix to obtain the feature matrix related to global semantics. The output is then dimensionalized by the upward projection layer and residually connected with the global semantic information to obtain the final output.
6. The method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters according to claim 5, characterized in that: The operations of the selector specifically include: Introduce a selector to the input and add noise, then normalize it with softmax to get the probability of selection, and get the value and index with the highest probability for the selection result; The obtained index is embedded to obtain an embedding vector related to global semantics, and the embedding vector is multiplied by the maximum probability value to make it differentiable, so as to obtain the final unique pattern vector related to global semantics; The obtained unique pattern vector is passed through a weight linear layer to obtain a unique pattern weight matrix related to global semantics, and the unique pattern vector is passed through a bias linear layer to obtain a unique pattern bias matrix related to global semantics.
7. The method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters according to claim 1, characterized in that: The token-level adapter module specifically includes: introducing a selector into the token-level feature matrix and adding noise, obtaining a probability result of the selection after a softmax normalization operation, and obtaining the value and index with the maximum probability for the selection result; The value index is embedded and then multiplied by the probability vector to obtain the unique pattern vector at the token level related to the input; The token-level specific pattern vector is passed through a weight linear layer and a bias linear layer to obtain a token-level weight matrix and a token-level bias matrix. The dot product of the weight matrix and the token-level specific pattern vector plus the bias matrix is then added to obtain the final token-level feature matrix. The token-level feature matrix is then passed through a linear layer for dimensionality increase and then residually connected to the feature matrix that initially entered the selector. The dimension is the same as before entering this module.
8. The method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters according to claim 1, characterized in that: The weighting of the text information output by the three modules is specifically as follows: Initialize a Fusion matrix, concatenate the global semantic information perception text features output by the sentence-level adapter module and the token-level text features output by the token-level adapter module in the third dimension, obtain two levels of text features, and then pass them through the Fusion matrix to obtain the fused text features; Initialize the weight matrix G, which will be updated as the model is trained, and add the weights of the common text features and the fused text features.
9. The method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters according to claim 1, characterized in that: The use of MSE loss to train the model is specifically as follows: Where B is the batch size, is the source language text feature, is the fused target language text feature.
10. The method for generating cross-language and cross-modal retrieval based on two levels of adaptable parameters according to claim 1, characterized in that: The retrieval model that uses a pre-trained image encoder to obtain image features and uses NCE loss calculation to achieve cross-modal alignment of text features and image features is specifically: In the cross-modal alignment phase, we first load the model trained in the cross-language transfer phase, reduce the image size from the original image to 224×224, and then use a 2D convolutional layer to convert the input image into a feature map. Then, we add position encoding to get the initial features of the image. The obtained initial features are input into the pre-trained CLIP VisionTransformer to the final image features F v ; Calculate NCE loss for image features and text features: F t To execute the target language branch in the text encoder, the final target language text features are obtained.
Citation Information
Patent Citations
Cross-language cross-modal retrieval method and device for noise robust learning
CN115438220A
Attention-based deep cross-modal hash retrieval method and apparatus, and related device
WO2022155994A1
Dam defect image-text cross-modal retrieval method and model
WO2022242388A1
Cited By
Cross-language cross-modal retrieval method based on semantic decoupling and dynamic parameter generation
CN119322859A
