Deep cross-modal retrieval method and system based on multilevel structure self-sensing

Abstract

The invention discloses a depth cross-modal retrieval method and system based on multilevel structure self-sensing, and the method comprises the steps: processing an original sample into an original sample sequence, the original sample comprising paired images and text entities; respectively inputting the original sample sequences into a cross-modal network, and calculating cross-modal comparison loss and cross-modal matching loss; inputting the original sample sequence into a single-mode teacher network, and calculating representation distillation loss to enable a cross-mode network to learn optimal cross-mode representation of a single-mode representation level; calculating respective sample similarity matrixes of the single-mode network and the cross-mode network, fusing the similarity matrixes of the two single-mode networks, and calculating structural self-sensing relation distillation loss with the similarity matrix of the cross-mode network; and performing parameter updating on the whole model to train the model. According to the method, through the cross-modal matching loss and the multi-layer structure self-sensing distillation loss, excellent cross-modal consistency representation is learned, the structural integrity of a single mode is reserved, and the cross-modal retrieval effect is improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a deep cross-modal retrieval method and system based on multi-level structure self-perception. Background Art

[0002] Visual language retrieval aims to search for similar instances in one modality based on a query from another modality. However, in real applications, it is more common to search for unimodal and cross-modal instances simultaneously. In real applications, large information retrieval websites always display cross-modal and unimodal instances to users, that is, for a given text / image query, similar text and visual instances will be returned. In experiments with multiple benchmark models, we observed that the existing visual language retrieval models have a widespread decline in unimodal retrieval performance compared to unimodal independent training. The main reason for this decline in unimodal performance is that cross-modal joint learning affects the unimodal structure. Therefore, a distillation model is proposed to learn cross-modal representations while keeping the unimodal structure intact. Summary of the invention

[0003] The purpose of the present invention is to provide a deep cross-modal retrieval method and system based on multi-level structural self-perception, which uses an excellent unimodal model as a teacher model and performs multi-level distillation on the cross-modal retrieval model to retain the unimodal retrieval performance and promote cross-modal retrieval capabilities.

[0004] The technical solution to achieve the purpose of the present invention is as follows: In the first aspect, the present invention provides a deep cross-modal retrieval method based on multi-level structure self-perception, comprising the following steps:

[0005] Step 1: Process the original samples into original sample sequences, where the original samples include paired images and text entities;

[0006] Step 2: Input the original sample sequences into the cross-modal network and calculate the cross-modal contrast loss and cross-modal matching loss;

[0007] Step 3: Input the original sample sequence into the unimodal teacher network and calculate the representation distillation loss so that the cross-modal network can learn the optimal cross-modal representation at the unimodal representation level;

[0008] Step 4: Calculate the sample similarity matrix of the unimodal network and the cross-modal network, fuse the two unimodal network similarity matrices, and calculate the structural self-aware relationship distillation loss with the cross-modal network similarity matrix;

[0009] Step 5: Update the parameters of the overall model according to the above loss function to train the model.

[0010] In a second aspect, the present invention provides a deep cross-modal retrieval system based on multi-level structure self-perception, which is used to implement the method described in the first aspect, and the system includes:

[0011] The first module is used to process the original samples into original sample sequences, where the original samples include paired images and text entities;

[0012] The second module is used to input the original sample sequences into the cross-modal network and calculate the cross-modal contrast loss and the cross-modal matching loss;

[0013] The third module is used to input the original sample sequence into the unimodal teacher network and calculate the representation distillation loss so that the cross-modal network can learn the optimal cross-modal representation at the unimodal representation level;

[0014] The fourth module is used to calculate the sample similarity matrix of the unimodal network and the cross-modal network, and then fuse the similarity matrices of the two unimodal networks and calculate the structural self-perceived relationship distillation loss with the cross-modal network similarity matrix;

[0015] In the fifth module, the parameters of the overall model are updated according to the above loss function to train the model.

[0016] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect when executing the program.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0018] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which implements the steps of the method described in the first aspect when executed by a processor.

[0019] Compared with the prior art, the significant advantages of the present invention are as follows: the present invention proposes a cross-modal retrieval method for comprehensive visual language retrieval that retains the unimodal structure while learning cross-modal consistent representations. The optimal unimodal model is obtained through contrast loss, and the cross-modal retrieval model is distilled at multiple levels using the unimodal model. A novel cross-modal retrieval model is proposed, which effectively solves the problem that the representation learned by the traditional cross-modal learning method has poor performance in unimodal retrieval and is difficult to learn a good cross-modal consistent representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is an overall flow chart of the deep cross-modal retrieval method based on multi-level structure self-perception of the present invention.

[0021] Figure 2 This is a network framework diagram of the deep cross-modal retrieval method based on multi-level structure self-perception.

[0022] Figure 3 This is the relationship distillation flow chart.

[0023] Figure 4 Sub-flowchart for model training steps. DETAILED DESCRIPTION

[0024] Combination Figure 1 to Figure 4 The present invention provides a deep cross-modal retrieval method based on multi-level structure self-perception, comprising the following steps:

[0025] (1) Processing the original samples into original sample sequences, where the original samples include paired images and text entities;

[0026] ① Initialize text and image. The specific form is:

[0027]

[0028] Where L T is the number of texts, d is the dimension of the text vector, L I =HW / P 2 ,(P×P) represents the size of each block of the image, T is the original text, I is the original image, C, H, and W are the number of channels, width, and height of the image respectively.

[0029] (2) Input the original sample sequence into the cross-modal network respectively and calculate the cross-modal contrast loss and cross-modal matching loss.

[0030] ① Use the attention-based image encoder to encode the image and obtain the first feature representation of the image modality, which is in the form of:

[0031]

[0032] Where X I is the image modality feature;

[0033] ② Use the attention-based text encoder to encode the image and obtain the first feature representation of the text modality, which is in the form of:

[0034]

[0035] Where X T is the text modality feature;

[0036] ③ Use the cross-modal attention mechanism and the feed-forward network to obtain the first feature representation of the cross-modal module. The specific form is:

[0037]

[0038] MultiAtt(X)=[Att(X)1,...,Att(X)M ]W M

[0039] FFN=(MultiAtt(X))=max(0,MultiAtt(X)W1+b1)W2+b2

[0040] where x I With x T The dimensions are the same; M is the number of multi-head attention heads, d M =d / M, d is the dimension of the first feature shared subspace; X represents X I or X T ; W M , W1, W2 are the two learning parameters of the multi-head attention and feedforward networks respectively; Att is the attention flag; Q T , Q I For image and text queries, For the key of image and text, V I and V T is the value; MultiAtt is multi-head attention, FFN is the feedforward network, max means taking the maximum, b1 and b2 are the two biases of the feedforward network, and softmax is the activation function.

[0041] ④ Calculate the cross-modal contrast loss, the specific form is:

[0042]

[0043] where d(I,T) = cos(g i (I CLS ),g t (T CLS )) and d(T,I)=cos(g t (T CLS ),g i (I CLs )) is the similarity function, g represents the encoder, p is the model prediction value, CE is the cross entropy loss, τ is the temperature scale coefficient, I CLS and T CLS is the [CLS] token of the vision and language encoder, exp stands for expectation, is the expected loss of image text, y i2t (I) retrieves the true value of the text for image I, y t2i (T) retrieves the true value of the image for the text T, p i2t (I) and p t2i (T) is the image retrieval text and text retrieval image prediction, J is the number of images or texts in a batch, and Represent the retrieval prediction of the kth image or text in a batch, respectively.

[0044] ⑤ Calculate the cross-modal matching loss, the specific form is:

[0045] l itm =[CE(y match ,p matc )]

[0046] where y match is the one-dimensional one-hot encoding representing the true value, p match =φ(IT CLS ) is the matching prediction probability output by the two-dimensional classifier, and φ represents the binary classifier.

[0047] (3) Input the original sample sequence and use the contrast loss and distance maximum regularization loss to train the optimal unimodal model.

[0048]

[0049] R v =||min((D v -△ + )⊙(D v -△ - ),0)||0

[0050] R w =||min((D w -△ + )⊙(D w -△ - ),0)||0

[0051] Where s(·) is used to measure the similarity between two instances; D v (I i ,I j )=s(I i ,I j ) and D w (I i ,I j )=s(I i ,I j ) is a similarity measure; negative example I - , T - Sampling from the same batch, positive example I + , T + After perturbing each instance, we get: R v With R w is the regularization of the distance; K is the number of batches, △ + =γ + 1 N×N and - =γ - 1 N×Nis the threshold parameter, where γ is the marginal value for distinguishing the similarity of data pairs, l ince is the nce loss of the image, l tnce nce loss for the text.

[0052] (4) The original sample sequence is input into the unimodal teacher network and the representation distillation loss is calculated so that the cross-modal network can learn the optimal cross-modal representation at the unimodal representation level.

[0053] ① Calculate the visual modality representation distillation loss:

[0054]

[0055] in represents the visual-to-visual similarity within the same batch, For expectation, y i2i (I) is the true value of the image retrieval image, p i2i (I) is the predicted value of the image for image retrieval.

[0056] ② Calculate the text modality representation distillation loss:

[0057]

[0058] in Indicates the similarity between texts in the same batch.

[0059] (5) Calculate the sample similarity matrix of the unimodal network and the cross-modal network respectively, fuse the similarity matrices of the two unimodal networks and calculate the structural self-aware relationship distillation loss with the cross-modal network similarity matrix, as follows:

[0060] Calculate the relationship similarity matrix S between the two single-modal models and the cross-modal model in the same batch respectively I , S T With S IT ;

[0061] According to the hyperparameter γ, the mixed similarity matrix S of the two single modes is obtained o :

[0062] S o =γS I +(1-λ)S T

[0063] λ is a hyperparameter;

[0064] Calculate the distillation loss of the mixed similarity matrix and the cross-modal model similarity matrix:

[0065]

[0066] (6) Calculate cross-modal loss and distillation loss, and update the model.

[0067] The specific loss function of the overall model is as follows:

[0068] ① Calculate cross-modal loss:

[0069] l cr = l itc +l itm

[0070] ② Calculate the multi-size distillation loss:

[0071] l md = l iic +l ttc +l rd

[0072] ③Overall loss function:

[0073] l=l cr +l md

[0074] Based on the same inventive concept, the present invention also proposes a deep cross-modal retrieval system based on multi-level structure self-perception, which is used to implement the above-mentioned deep cross-modal retrieval method. The system includes:

[0075] The first module is used to process the original samples into original sample sequences, where the original samples include paired images and text entities;

[0076] The second module is used to input the original sample sequences into the cross-modal network and calculate the cross-modal contrast loss and the cross-modal matching loss;

[0077] The third module is used to input the original sample sequence into the unimodal teacher network and calculate the representation distillation loss so that the cross-modal network can learn the optimal cross-modal representation at the unimodal representation level;

[0078] The fourth module is used to calculate the sample similarity matrix of the unimodal network and the cross-modal network, and then fuse the similarity matrices of the two unimodal networks and calculate the structural self-perceived relationship distillation loss with the cross-modal network similarity matrix;

[0079] In the fifth module, the parameters of the overall model are updated according to the above loss function to train the model.

[0080] The specific implementation methods of the first to fifth modules are the same as the aforementioned method steps and will not be repeated here.

[0081] The present invention uses the attention network as the backbone network, and uses contrast loss and distance maximum regularization loss to train a unimodal model to extract image text features. Through relation distillation with a unimodal model as a teacher and a cross-modal retrieval model as a student, it has excellent performance in unimodal retrieval and cross-modal retrieval tasks.

[0082] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A deep cross-modal retrieval method based on multi-level structure self-perception, characterized in that: The steps include: Step 1: Process the original samples into original sample sequences, where the original samples include paired images and text entities; Step 2: Input the original sample sequences into the cross-modal network and calculate the cross-modal contrast loss and cross-modal matching loss; Step 3: Input the original sample sequence into the unimodal teacher network and calculate the representation distillation loss so that the cross-modal network can learn the optimal cross-modal representation at the unimodal representation level; Step 4: Calculate the sample similarity matrix of the unimodal network and the cross-modal network, fuse the two unimodal network similarity matrices, and calculate the structural self-aware relationship distillation loss with the cross-modal network similarity matrix; Step 5: Update the parameters of the overall model according to the above loss function to train the model.

2. The deep cross-modal retrieval method based on multi-level structure self-perception as claimed in claim 1, characterized in that: In step 1, obtaining the original sample sequence includes the following steps: Initialize text and images in the following form: Where L T is the number of texts, d is the dimension of the text vector, L I =HW / P 2 ,(P×P) represents the size of each block of the image, T is the original text, I is the original image, C, H, and W are the number of channels, width, and height of the image respectively.

3. The deep cross-modal retrieval method based on multi-level structure self-perception as claimed in claim 2, characterized in that: In step 2, the original sample sequence is input into the cross-modal network and the cross-modal contrast loss and the cross-modal matching loss are calculated, which specifically includes: ① Use the attention-based image encoder to encode the image and obtain the first feature representation of the image modality, which is in the form of: Where X I is the image modality feature; ② Use the attention-based text encoder to encode the image and obtain the first feature representation of the text modality, which is in the form of: Where X T is the text modality feature; ③ Use the cross-modal attention mechanism and the feed-forward network to obtain the first feature representation of the cross-modal module. The specific form is: MultiAtt(X)=[Att(X)1,...,Att(X) M ]W M FFN=(MultiAtt(X))=max(0,MultiAtt(X)W1+b1)W2+b2 where x I With x T The dimensions are the same; M is the number of multi-head attention heads, d M =d / M, d is the dimension of the first feature shared subspace; X represents X I or X T ; W M , W1, W2 are the two learning parameters of the multi-head attention and feedforward networks respectively; Att is the attention flag; Q T , Q I For image and text queries, For the key of image and text, V I and V T is the value; MultiAtt is multi-head attention, FFN is the feedforward network, max means taking the maximum, b1 and b2 are the two biases of the feedforward network, and softmax is the activation function; ④ Calculate the cross-modal contrast loss, the specific form is: where d(I,T) = cos(g i (I CLS ),g t (T CLS )) and d(T,I)=cos(g t (T CLS ),g i (I CLS )) is the similarity function, g represents the encoder, p is the model prediction value, CE is the cross entropy loss, τ is the temperature scale coefficient, I CLS and T CLS is the [CLS] token of the vision and language encoder, exp stands for expectation, is the expected loss of image text, y i2t (I) Retrieve the text truth value for image I, y t2i (T) retrieves the true value of the image for the text T, p i2t (I) and p t2i (T) is the image retrieval text and text retrieval image prediction, J is the number of images or texts in a batch, and Represents the retrieval prediction of the kth image or text in a batch respectively; ⑤ Calculate the cross-modal matching loss, the specific form is: it itm =[CE(y matc ,p match )] where y match is the one-dimensional one-hot encoding representing the true value, p match =φ(IT CLS ) is the matching prediction probability output by the two-dimensional classifier, and φ represents the binary classifier.

4. The deep cross-modal retrieval method based on multi-level structure self-perception as claimed in claim 3, characterized in that: In step 3, the original sample sequence is input into the unimodal teacher network for learning and the representation distillation loss is calculated so that the cross-modal network learns the optimal cross-modal representation at the unimodal representation level, including the following steps: ① The original sample sequence is input into the unimodal network to obtain the optimal unimodal model, which includes the following steps: R v =||min((D v -△ + )⊙(D v -△ - ),0)||0 R w =||min((D w -△ + )⊙(D w -△ - ),0)||0 Where s(·) is used to measure the similarity between two instances; D v (I i ,I j )=s(I i ,I j ) and D w (I i ,I j )=s(I i ,I j ) is a similarity measure; negative example I - 、T - Sampling from the same batch, positive example I + 、T + After perturbing each instance, we get: R v With R w is the regularization of the distance; K is the number of batches, △ + =γ + 1 N×N and - =γ - 1 N×N is the threshold parameter, where γ is the marginal value for distinguishing the similarity of data pairs, l ince is the nce loss of the image, k tnce nce loss for the text; ② Calculate the visual modality representation distillation loss: in represents the visual-to-visual similarity within the same batch, For expectation, y i2i (I) is the true value of the image retrieval image, p i2i (I) is the predicted value of the image for image retrieval; ③ Calculate the text modality representation distillation loss: in Indicates the similarity between texts in the same batch.

5. The deep cross-modal retrieval method based on multi-level structure self-perception as claimed in claim 4, characterized in that: In the step 4, the sample similarity matrices of the unimodal network and the cross-modal network are calculated, and after fusing the two unimodal network similarity matrices, the structural self-perceived relationship distillation loss is calculated with the cross-modal network similarity matrix, including the following steps: ① Calculate the relationship similarity matrix S between the two single-modal models and the cross-modal model in the same batch respectively I , S T With S IT ; ②According to the hyperparameter λ, the mixed similarity matrix S of the two single modes is obtained O : S O =λS I +(1-λ)S T λ is a hyperparameter; ③ Calculate the distillation loss of the mixed similarity matrix and the cross-modal model similarity matrix:

6. The deep cross-modal retrieval method based on multi-level structure self-perception as claimed in claim 5, characterized in that: In step 5, the parameters of the overall model are updated according to the above loss function to train the model: ① Calculate cross-modal loss: l cr ＝l itc +l itm ② Calculate the multi-size distillation loss: l md ＝l iic +l ttc +l rd ③Overall loss function: l＝l cr +l md 。 7. A deep cross-modal retrieval system based on multi-level structure self-perception, characterized in that: For implementing the method described in any one of claims 1 to 6, the system comprises: The first module is used to process the original samples into original sample sequences, where the original samples include paired images and text entities; The second module is used to input the original sample sequences into the cross-modal network and calculate the cross-modal contrast loss and the cross-modal matching loss; The third module is used to input the original sample sequence into the unimodal teacher network and calculate the representation distillation loss so that the cross-modal network can learn the optimal cross-modal representation at the unimodal representation level; The fourth module is used to calculate the sample similarity matrix of the unimodal network and the cross-modal network, and then fuse the similarity matrices of the two unimodal networks and calculate the structural self-perceived relationship distillation loss with the cross-modal network similarity matrix; In the fifth module, the parameters of the overall model are updated according to the above loss function to train the model.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of any method described in claims 1-6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are implemented.

Images