Cross-modal retrieval method based on modality-specific adaptive scaling and attention network

Abstract

The present invention relates to a cross-modal retrieval method based on modality-specific adaptive scaling and attention network. First, a Transformer encoder with residual space reduction is used for the image modality, and a word-level feature attention module is used for the text modality to extract modality-invariant features respectively. Then, a modality embedding-level feature attention module is used to filter modality interference features. Then, an adaptive scaling network that integrates prior knowledge is used to map features to a multimodal common subspace for modality common feature learning. Finally, the multimodal common subspace features of new data to be queried and retrieved are calculated using the best performing network weights, and category-level reordering is performed to return the final cross-modal data retrieval. The present invention can effectively model multimodal data and efficiently and accurately complete cross-modal retrieval tasks in multimodal scenarios.

Description

Technical Field

[0001] The present invention relates to the fields of pattern recognition, computer vision and natural language processing, and in particular to a cross-modal retrieval method based on modality-specific adaptive scaling and attention network. Background Art

[0002] In the past few decades, with the rapid development of computer technology, especially data storage technology and network transmission technology, the total amount of data on the Internet has shown explosive growth. Text, audio, image, video, 3D model... More and more data forms have become the carriers of Internet information exchange. Single-modal retrieval tasks, such as text retrieval and image retrieval, focus on querying the desired results within a single modality and cannot model cross-modal relationships. There are huge differences in data distribution and feature representation of different modal data, which brings huge challenges to data retrieval. Summary of the invention

[0003] The purpose of the present invention is to provide a cross-modal retrieval method based on modality-specific adaptive scaling and attention network, which can effectively model multimodal data and efficiently and accurately complete cross-modal retrieval tasks in multimodal scenarios.

[0004] To achieve the above object, the technical solution of the present invention is: a cross-modal retrieval method based on modality-specific adaptive scaling and attention network, comprising the following steps:

[0005] Step S1: A Transformer encoder with residual space reduction is used for the image modality, and a word-level feature attention module is used for the text modality to extract modality-invariant features respectively;

[0006] Step S2, filtering the modal interference features using a modal embedding-level feature attention module with the same structure but independent parameters for the modal invariant features obtained in step S1;

[0007] Step S3, using an adaptive scaling network that integrates prior knowledge to map the features obtained in step S2 to a multimodal common subspace for modal common feature learning;

[0008] Step S4: train the neural network according to steps S1, S2, and S3, and retain the network weights with the best performance; use this group of network weights to calculate the multimodal common subspace features of new data to be queried and retrieved, and perform cross-modal data retrieval.

[0009] Compared with the prior art, the present invention has the following beneficial effects:

[0010] 1. Based on the evaluation of multimodal information density differences, the present invention proposes a modality-specific network adaptive scaling method that integrates prior knowledge, which greatly alleviates the problem of unified fitting between modalities caused by differences in modal fitting rates due to information density differences in multimodal tasks.

[0011] 2. Aiming at the interference features existing in the pre-trained model, the present invention proposes a modality embedding-level feature attention module to guide the migration of knowledge between the pre-trained source domain and the target domain, so that the neural network can inherit the good initialization state provided by the pre-trained model and be free from the harm of the interference features.

[0012] 3. The present invention constructs a word-level feature attention module to efficiently fuse word semantics through the attention mechanism, solving the problem of efficient fusion of text semantics in cross-modal retrieval tasks.

[0013] 4. The modality-invariant feature extraction method based on the Transformer structure can fully capture the internal correlation of the modality, construct a high-quality modality latent representation space and multimodal common subspace, and shape highly similar feature distributions for different modalities, thereby achieving better cross-modal retrieval performance and breaking through the performance bottleneck of multimodal tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a schematic diagram of the principle of the present invention. DETAILED DESCRIPTION

[0015] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0016] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0017] like Figure 1 As shown, a cross-modal retrieval method based on modality-specific adaptive scaling and attention network includes the following steps:

[0018] Step S1: A Transformer encoder with residual space reduction is used for the image modality, and a word-level feature attention module is used for the text modality to extract modality-invariant features respectively.

[0019] Step S2: Using a modality embedding-level feature attention module with the same structure but independent parameters to filter modality interference features on the modality-invariant features obtained in step S1.

[0020] Step S3: using an adaptive scaling network that integrates prior knowledge to map the features obtained in step S2 to a multimodal common subspace for modal common feature learning.

[0021] Step S4: Follow steps S1, S2, and S3 to train the neural network, retain the best performing network weights, and use this set of weights to calculate the multimodal common subspace features of new data to be queried and retrieved, and perform cross-modal data retrieval.

[0022] The step S1 comprises the following steps:

[0023] Step S11: For the image modality, the input image is evenly divided into N = 16 sub-images according to 4 rows and 4 columns, and the length and width of each sub-image are 1 / 4 of the original image. The block embedding x of each sub-image is extracted by convolution operation. 0 , x 1 ..., x N-1 , and an additional learnable block embedding x extra . Add position embedding to each block embedding according to formula (1-1) and formula (1-2):

[0024]

[0025]

[0026] Where pos is the position of the block, dim_i represents the dim_i-th dimension of the position embedding, and d is the dimension of the block embedding which is 256.

[0027] Embed each block of length 256 into x 0 , x 1 ..., x N-1 The feature map is scaled to 16×16, and all the block embeddings are superimposed in the channel dimension to obtain a N×16×16 multi-channel feature map. The feature map is enlarged using a layer normalization and N / 2 5×5 convolution kernels, reducing the number of channels by half. At the same time, an additional residual connection is added through an average pooling and zero padding. Then it is split according to the channel, and the feature map of each channel is remapped to a new block embedding. Since the sequence length has changed, the new block embedding is recalculated and the position embedding is added to update the relative position information to obtain the block embedding x′ after the residual space is reduced. 0 , x′ 1 ..., x′ N憠1 .

[0028] z0 =[x extra ; x′ 0 W P ; x′ 1 W P ; ... ; x′ N-1 W P ]+e pos (1-3)

[0029] z′ l =MHA(LN(z l-1 ))+z l-1 , l=1...L (1-4)

[0030] z l =MLP(LN(z′ l ))+z′ l , l=1...L (1-5)

[0031]

[0032] Calculate the image modality representation according to formula (1-3) to formula (1-6). Where z 0 , z′ l , z l is a temporary variable, L represents the number of image blocks, l represents the lth block, and W P represents the learnable parameter matrix, x extra is an additional learnable embedding, and its corresponding output The result after an MLP operation Represented as image modality z img .e pos Represents the added one-dimensional position embedding. LN represents layer normalization. MHA represents multi-head attention, which performs multiple self-attention operations on the input data. MLP stands for multi-layer perceptron, which contains a linear projection, GeLU activation function, and then another linear projection. In the model, an additional MLP head is used to obtain the output corresponding to the first embedding as the representation of the image modality.

[0033] Step S12: For text modality, the word-level feature attention module first uses the Word2Vec method to convert each word in the text description into a 300-dimensional word embedding, and then uses a weight generation unit with shared weights for all word embeddings, including a fully connected layer FC, a ReLU activation function, an average pooling layer AvgPooling and a Sigmoid layer. The weight generation unit generates a weight for each single embedding, and performs weighted averaging to obtain the representation of the text modality. Specifically, the following formula is followed:

[0034]

[0035]

[0036]

[0037]

[0038] Represents the word embedding corresponding to the n_jth word of the n_ith text data, is a temporary variable of 300 dimensions, Is a temporary variable of numeric type. is the weight vector, (e n_i )′ is the text representation after word embedding fusion.

[0039] The step S2 comprises the following steps:

[0040] Step S21, for the features of the image modality, a modality embedding-level feature attention module consisting of a linear layer 1, an activation function, a linear layer 2, and a NegSig function (as shown in formula (2-1), where Euler represents the Euler number, approximately equal to 2.71828) is used to capture the position of the interference features. For simplicity, the dimensions of the two linear layers are consistent with the dimensions of the input features. x represents the output of the second linear layer. The output of the NegSig function is dot-producted with the original feature input to obtain a negative interference feature. It is then added to the original input feature through a residual connection to shield the interference in the original input feature.

[0041]

[0042] Step S22: For the output of the text modality word-level feature attention module, similar operations are taken as in step S21, except that the module parameters used are independent.

[0043] The step S3 comprises the following steps:

[0044] Step S31: Before each training, a simple pre-classification network (such as MLP) is used to initialize the classification of each modality feature extracted by the pre-trained backbone network. The classification accuracy is acc modal_i It is expressed as the pre-classification accuracy of the modal_i modality. Let the matrix A = (acc 1 …acc m ), where m is the number of modes. The mode-specific scaling factor s modal_i As shown in the following formula:

[0045]

[0046] Where θ is an m-dimensional vector that can be learned during training. During initialization, the function can be initialized as a non-negative monotonically decreasing random function between the domain [0, 1], combining the artificial prior that "more complex tasks require larger networks, and simpler tasks require smaller networks". modal_i As a scaling factor for adaptive scaling encoders and decoders.

[0047] Step S32: As shown in formula (3-2), the scaling loss of the modal_i modality is minimized during the training process. Among them, acc (including the classification accuracy acc of the modal_i modality modal_i And the classification accuracy acc of the modal_j modality modal_j ) is the classification accuracy of the auxiliary classifier, and m is the number of modalities:

[0048]

[0049] Step S33: Calculate the parameter s according to the following formula modal_i Update:

[0050]

[0051] Among them, s modal_i ′ is s modal_i The updated value, lr is the learning rate. It should be noted that acc will change with the training of the network. We only update s modal_i acc is used when acc is not updated.

[0052] The step S4 comprises the following steps:

[0053] Step S41: During the training process, the loss function of the modal_i-th mode is:

[0054]

[0055] in, is the supervision loss of the modal_i modality, is the reconstruction loss of the modal_i modality. λ is a balance parameter between the supervision loss and the reconstruction loss, and λ is a trade-off factor.

[0056] Follow steps S1, S2, and S3 to train the neural network, and use the gradient descent algorithm to minimize the loss function L during the training process. modal_iAfter each training, update the matrix A according to the classification results, calculate the adaptive scaling loss according to formula (3-2), and update s according to formula (3-3) and formula (3-4). modal_i . Set the maximum number of training iterations iter max as a termination condition.

[0057] Step S42: Select the model with the highest average precision mAP@ALL on the validation set, discard the network weights of the decoder part, use the model to calculate the common subspace embedding for all the data to be queried and retrieved, use the cosine distance to calculate the similarity, and sort in descending order.

[0058] Step S43: re-sort the retrieval results of each input data by category. Taking the input data as an image as an example, the K retrieved txt The kth text data is used as input data to reversely retrieve the image. The reverse retrieval result of the kth text is recorded as a binary string Among them, matching tags If the reverse retrieval result belongs to the same category as the input image query, the value of the matching label is 1, otherwise it is 0 (there may be multiple matching labels that are 1). Convert to decimal value Sort the text data in descending order based on this value. If the values ​​are the same, the text with the highest ranking will be given priority. The same operation is applied to the case where the input data is text. Finally, the search results after re-sorting at the category level are returned.

[0059] In particular, the present invention proposes a modality-specific network adaptive scaling method that integrates prior knowledge based on the evaluation of multimodal information density differences, which greatly alleviates the problem of unified fitting between modalities caused by differences in modal fitting rates due to differences in information density in multimodal tasks. Aiming at the interference features existing in the pre-trained model, the present invention proposes a modality embedding-level feature attention module to guide the migration of knowledge between the pre-training source domain and the target domain, so that the neural network can inherit the good initialization state provided by the pre-training model and be free from the harm of interference features. The present invention constructs a word-level feature attention module to efficiently fuse word semantics through the attention mechanism, solving the problem of efficient fusion of text semantics in cross-modal retrieval tasks. The modality-invariant feature extraction method based on the Transformer structure can fully capture the internal correlation of the modality, construct a high-quality modality latent representation space and a multimodal common subspace, and shape highly similar feature distributions for different modalities, thereby achieving better cross-modal retrieval performance and breaking through the performance bottleneck of multimodal tasks.

[0060] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. A cross-modal retrieval method based on modality-specific adaptive scaling and attention network, characterized in that: The steps include: Step S1: A Transformer encoder with residual space reduction is used for the image modality, and a word-level feature attention module is used for the text modality to extract modality-invariant features respectively; Step S2, filtering the modal interference features using a modal embedding-level feature attention module with the same structure but independent parameters for the modal invariant features obtained in step S1; Step S3, using an adaptive scaling network that integrates prior knowledge to map the features obtained in step S2 to a multimodal common subspace for modal common feature learning; Step S4, training the neural network according to steps S1, S2, and S3, retaining the best performing network weights; using the set of network weights to calculate the multimodal common subspace features of new data to be queried and retrieved, and performing cross-modal data retrieval; The step S1 is specifically implemented as follows: Step S11: For the image modality, the input image is evenly divided into N = 16 sub-images according to 4 rows and 4 columns, and the length and width of each sub-image are 1 / 4 of the original image; a convolution operation is used to extract the block embedding x0, x1..., x N-1 , and an additional learnable block embedding x extra ; Add position embedding to each block embedding according to formula (1-1) and formula (1-2): Where pos is the position of the block, dim_i represents the dim_i-th dimension of the position embedding, and d is the dimension of the block embedding, which is 256; Embed each block of length 256 into x0, x1, ..., x N-1 The feature map is scaled to 16×16, and all the block embeddings are superimposed in the channel dimension to obtain a N×16×16 multi-channel feature map; a layer normalization and N / 2 5×5 convolution kernels are used to expand the feature map, reducing the number of channels by half; at the same time, an additional residual connection is added through an average pooling and zero padding; then the feature map is split according to the channel, and the feature map of each channel is remapped to a new block embedding; due to the change in sequence length, the new block embedding is recalculated and the position embedding is added to update the relative position information to obtain the block embedding x′0, x′1..., x′ after the residual space is reduced. N-1 ; Calculate the image modality representation according to formula (1-3) to formula (1-6): z0=[x extra ;x′0W P ;x′1W P ;...;x′ N-1 W P ]+e pos (1-3) z′ l =MHA(LN(z l-1 ))+z l-1 ,l=1...L (1-4) With l =MLP(LN(z′ l ))+z′ l ,l=1...L (1-5) Among them, z0, z′ l , z l is a temporary variable, L represents the total number of image blocks, l represents the lth block, and W P represents the learnable parameter matrix, x extra is an additional learnable block embedding, and its corresponding output The result after an MLP operation Represented as image modality z img ;e pos represents the added one-dimensional position embedding; LN represents layer normalization; MHA represents multi-head attention, which performs multiple self-attention operations on the input data; MLP represents multi-layer perceptron, which includes linear projection 1, GeLU activation function, and linear projection 2; in the model, an additional MLP head is used to obtain the output corresponding to the first block embedding as the representation of the image modality; Step S12: For the text modality, the word-level feature attention module first uses the Word2Vec method to convert each word in the text description into a 300-dimensional word embedding, and then uses a weight generation unit with shared weights for all word embeddings, including a fully connected layer FC, a ReLU activation function, an average pooling layer AvgPooling and a Sigmoid layer; the weight generation unit generates a weight for each word embedding, and performs weighted averaging to obtain the representation of the text modality; specifically, the following formula is followed: Represents the word embedding corresponding to the n_jth word of the n_ith text data, is a temporary variable of 300 dimensions, Is a temporary variable of numeric type. is the weight vector, (e n_i )′ is the text representation after word embedding fusion; The step S3 is specifically implemented as follows: Step S31: Before each training, a pre-classification network is used to initialize the classification of each modality feature extracted by the pre-trained backbone network. The classification accuracy is acc modal_i It is expressed as the pre-classification accuracy of the modal_i modality; let the matrix A = (acc1 ... acc m ), where m is the number of modes; the mode-specific scaling factor s modal_i As shown in the following formula: Among them, θ is an m-dimensional vector that can be learned during training. During initialization, combined with the artificial prior that "more complex tasks require larger networks, and simpler tasks require smaller networks", the above function is initialized to a non-negative monotonically decreasing random function between the domain [0, 1]. The modality-specific scaling factor s modal_i As a scaling factor for adaptive scaling encoders and decoders; Step S32: As shown in formula (3-2), the scaling loss of the modal_i modality is minimized during the training process. Where acc includes the classification accuracy acc of the modal_i modality modal_i And the classification accuracy acc of the modal_j modality modal_j is the classification accuracy of the auxiliary classifier, and m is the number of modalities: Step S33: Calculate the parameter s according to the following formula modal_i Update: Among them, s modal_i′ Yes modal_i The updated value, lr is the learning rate; acc will change with the training of the network and will only change when updating s modal_i acc is used when acc is not updated.

2. The cross-modal retrieval method based on modality-specific adaptive scaling and attention network according to claim 1, characterized in that: The step S2 is specifically implemented as follows: Step S21, for the features of the image modality, a modality embedding-level feature attention module composed of a linear layer 1, an activation function, a linear layer 2, and a NegSig function is used to capture the position of the interference features; for simplicity, the dimensions of the two linear layers are consistent with the dimensions of the input features; x represents the output of the linear layer 2; the output of the NegSig function is dot-producted with the original feature input to obtain a negative interference feature; and then, through a residual connection, it is added to the original input feature to shield the interference in the original input feature; The NegSig function is shown in formula (2-1): Where Euler represents the Euler number: Step S22: For the output of the text modality word-level feature attention module, similar operations are taken as in step S21, except that the module parameters used are independent.

3. The cross-modal retrieval method based on modality-specific adaptive scaling and attention network according to claim 1, characterized in that: The step S4 is specifically implemented as follows: Step S41: During the training process, the loss function of the modal_i-th mode is: in, is the supervision loss of the modal_i modality, is the reconstruction loss of the modal_i-th mode; λ is the balance parameter between the supervision loss and the reconstruction loss, and λ is a trade-off factor; Follow steps S1, S2, and S3 to train the neural network, and use the gradient descent algorithm to minimize the loss function L during the training process. modal_i After each training, update the matrix A according to the classification results, calculate the adaptive scaling loss according to formula (3-2), and update s according to formula (3-3) and formula (3-4). modal_i ; Set the maximum number of training iterations iter max As a termination condition; Step S42: Select the model with the highest mean average precision mAP@ALL of all results on the validation set, and discard the network weights of the decoder part; use the model to calculate the common subspace embedding for all the data to be queried and retrieved, use the cosine distance to calculate the similarity, and sort them in descending order; Step S43: re-sort the retrieval results of each input data by category; if the input data is an image, sort the K txt The kth text data is used as input data to reversely retrieve the image; the reverse retrieval result of the kth text is recorded as a binary string Among them, matching tags If the reverse retrieval result belongs to the same category as the input image query, the value of the matching label is 1, otherwise it is 0. There may be multiple matching labels that are 1; the reverse retrieval binary results of all texts Convert to decimal value Sort the text data in descending order according to the value; if the values ​​are the same, the text that was originally ranked higher will take precedence; the same operation is performed when the input data is text; and finally, the search results after reordering at the category level are returned.

Images