Optimization Method for Cross-Modal Retrieval Model Based on Image-Text Importance Consistency

Abstract

The present invention discloses a cross-modal retrieval model optimization method based on image-text importance consistency. The method provided by the present invention mines fine-grained semantic information in image-text pairs, realizes finer-grained alignment between different modalities, and effectively improves the retrieval accuracy of the cross-modal retrieval model.

Description

Technical Field

[0001] The present invention relates to the field of image-text retrieval, and in particular to a cross-modal retrieval model optimization method based on image-text importance consistency. Background Art

[0002] Cross-modal image-text retrieval, as an important downstream task of cross-modal representation learning, aims to build a bidirectional retrieval process for images and texts, that is, given a query sample of a certain modality, retrieve semantically similar samples of the other modality. In the prior art, a cross-modal retrieval model is pre-trained to embed image and text data into a feature space and calculate the similarity between the two. However, this work only explores the coarse-grained alignment of the two modalities and does not mine the fine-grained semantic information in the image-text pair. The retrieval performance of the model is not high. Therefore, the prior art needs to be improved and enhanced. Summary of the invention

[0003] In view of the above-mentioned defects of the prior art, the present invention provides a cross-modal retrieval model optimization method based on image-text importance consistency, aiming to solve the problem of low performance of cross-modal image-text modeling models in the prior art.

[0004] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0005] A first aspect of the present invention provides a method for optimizing a cross-modal retrieval model based on image-text importance consistency, the method comprising:

[0006] Acquire an initial cross-modal retrieval model, wherein the initial cross-modal retrieval model includes a target self-attention module;

[0007] A target batch is selected from the training set, the target batch includes at least one first image-text pair, the first image-text pair includes a first sample image and a first sample text that match each other, and a connection feature of an image feature set corresponding to the first sample image and a text feature set corresponding to the first sample text is input into a target self-attention module, wherein the image feature set includes features corresponding to a plurality of regions of the first sample image, and the text feature set includes features corresponding to a plurality of words in the first sample text;

[0008] Obtain a self-attention matrix in the target self-attention module, wherein the self-attention matrix is ​​obtained based on the query vector and the key vector in the target self-attention module, and the self-attention matrix is ​​divided according to the number of features in the image feature set and the number of features in the text feature set to obtain four sub-matrices, namely, a first sub-matrix, a second sub-matrix, a third sub-matrix and a fourth sub-matrix, wherein the size of the first sub-matrix is ​​N V *NV , the size of the second submatrix is ​​N V *N L , the size of the third submatrix is ​​N L *N V , the size of the fourth submatrix is ​​N L *N L , N V is the number of features in the image feature set, N L is the number of features in the text feature set;

[0009] Acquire the first modality intra-semantic importance distribution and the first inter-modality semantic importance distribution corresponding to the first sample image according to the first submatrix and the third submatrix, and acquire the second inter-modality semantic importance distribution and the second modality intra-semantic importance distribution corresponding to the first sample text according to the second submatrix and the fourth submatrix;

[0010] The training loss is determined according to the consistency between the semantic importance distribution within the first modality and the semantic importance distribution between the first modalities, and the consistency between the semantic importance distribution within the second modality and the semantic importance distribution between the second modalities, and the parameters of the initial cross-modal retrieval model are optimized according to the training loss.

[0011] The cross-modal retrieval model optimization method based on image-text importance consistency, wherein the initial cross-modal retrieval model is a pre-trained model.

[0012] In the cross-modal retrieval model optimization method based on image-text importance consistency, the image area corresponding to the features in the image feature set is the area including the target obtained after target detection is performed on the first sample image.

[0013] The cross-modal retrieval model optimization method based on image-text importance consistency, wherein after updating the parameters of the initial cross-modal retrieval model according to the training loss, comprises:

[0014] The step of selecting a target batch from the training set is repeated until the parameters of the initial cross-modal retrieval model converge.

[0015] The cross-modal retrieval model optimization method based on image-text importance consistency, wherein the step of obtaining the first modality intra-semantic importance distribution and the first inter-modality semantic importance distribution corresponding to the first sample image according to the first submatrix and the third submatrix includes:

[0016] Performing a sum operation on the elements in each column of the first submatrix to obtain a first modality intra-semantic importance score, and performing a normalization operation on each of the first modality intra-semantic importance scores to obtain a first modality intra-semantic importance distribution;

[0017] A sum operation is performed on the elements in each column of the third submatrix to obtain a first inter-modal semantic importance score, and a normalization operation is performed on each of the first inter-modal semantic importance scores to obtain a first inter-modal semantic importance distribution.

[0018] The cross-modal retrieval model optimization method based on image-text importance consistency, wherein the second inter-modal semantic importance distribution and the second intra-modal semantic importance distribution corresponding to the first sample text according to the second sub-matrix and the fourth sub-matrix include:

[0019] Performing a sum operation on the elements in each column of the second submatrix to obtain a second inter-modal semantic importance score, and performing a normalization operation on each of the second inter-modal semantic importance scores to obtain a second inter-modal semantic importance distribution;

[0020] A sum operation is performed on the elements in each column of the fourth submatrix to obtain a second modality intra-semantic importance score, and a normalization operation is performed on each of the second modality intra-semantic importance scores to obtain a second modality intra-semantic importance distribution.

[0021] The cross-modal retrieval model optimization method based on image-text consistency, wherein the target batch also includes at least one second image-text pair, the second image-text pair includes a second sample image and a second sample text that do not match each other, and the training loss is determined according to the consistency between the semantic importance distribution within the first modality and the semantic importance distribution between the first modalities, and the consistency between the semantic importance distribution within the second modality and the semantic importance distribution between the second modalities, including:

[0022] Determining a first loss according to a relative entropy between the first intra-modality semantic importance distribution and the first inter-modality semantic importance distribution;

[0023] determining a second loss according to a relative entropy between the second intra-modality semantic importance distribution and the second inter-modality semantic importance distribution;

[0024] Obtaining an optimization loss, wherein the optimization loss is obtained based on the first loss and the second loss corresponding to the at least one first image-text pair;

[0025] Obtaining the predicted similarity of each image-text pair in the target batch output by the initial cross-modal retrieval model, and obtaining a third loss according to the similarity difference corresponding to each image-text pair in the target batch, wherein the similarity difference corresponding to the image-text pair is constructed by the image and text matching relationship included in the image-text pair, and the predicted similarity corresponding to the image-text pair;

[0026] The training loss is determined according to the optimization loss and the third loss.

[0027] A second aspect of the present invention provides a cross-modal retrieval model optimization device based on image-text importance consistency, the device comprising:

[0028] A model acquisition module, used to acquire an initial cross-modal retrieval model, wherein the initial cross-modal retrieval model includes a target self-attention module;

[0029] a feature acquisition module, configured to select an image-text pair in a training set, wherein the image-text pair includes a first sample image and a first sample text, and input a connection feature of an image feature set corresponding to the first sample image and a text feature set corresponding to the first sample text into a target self-attention module in the at least one attention module, wherein the image feature set includes features corresponding to a plurality of regions of the first sample image, and the text feature set includes features corresponding to a plurality of words in the first sample text;

[0030] A matrix segmentation module is used to obtain the self-attention matrix in the target self-attention module, wherein the self-attention matrix is ​​obtained based on the query vector and the key vector in the target self-attention module, and the self-attention matrix is ​​segmented according to the number of features in the image feature set and the number of features in the text feature set to obtain four sub-matrices, namely, the first sub-matrix, the second sub-matrix, the third sub-matrix and the fourth sub-matrix, wherein the size of the first sub-matrix is ​​N V *N V , the size of the second submatrix is ​​N V *N L , the size of the third submatrix is ​​N L *N V , the size of the fourth submatrix is ​​N L *N L , N V is the number of features in the image feature set, N L is the number of features in the text feature set;

[0031] An importance distribution determination module is used to obtain a first modality intra-semantic importance distribution and a first inter-modality semantic importance distribution corresponding to the first sample image according to the first submatrix and the third submatrix, and a second inter-modality semantic importance distribution and a second modality intra-semantic importance distribution corresponding to the first sample text according to the second submatrix and the fourth submatrix;

[0032] A parameter optimization module is used to determine the training loss according to the semantic importance distribution within the first modality, the semantic importance distribution between the first modalities, the semantic importance distribution within the second modality, and the semantic importance distribution between the second modalities, and optimize the parameters of the initial cross-modal retrieval model according to the training loss.

[0033] According to a third aspect of the present invention, a terminal is provided, comprising a processor and a computer-readable storage medium communicatively connected to the processor, wherein the computer-readable storage medium is suitable for storing a plurality of instructions, and the processor is suitable for calling the instructions in the computer-readable storage medium to execute the steps of implementing any of the above-mentioned cross-modal retrieval model optimization methods based on image-text importance consistency.

[0034] A fourth aspect of the present invention provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps of any of the above-mentioned methods for optimizing a cross-modal retrieval model based on image-text importance consistency.

[0035] Compared with the prior art, the present invention provides a cross-modal retrieval model optimization method based on image-text importance consistency. The method provided by the present invention is applicable to a model with a self-attention module. The self-attention matrix in the self-attention module is split according to the number of image region features and the number of text word features to obtain four sub-matrices. The sub-matrices are used to measure the inter-modal semantic importance and intra-modal semantic importance of visual and text fragmented semantics, and the parameters of the model are optimized according to the consistency constraints between the inter-modal semantic importance distribution and the inter-modal semantic importance distribution. The fine-grained semantic information in the image-text pair is mined, and a finer-grained alignment between different modalities is achieved, effectively improving the retrieval accuracy of the cross-modal retrieval model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of an embodiment of a cross-modal retrieval model optimization method based on image-text importance consistency provided by the present invention;

[0037] Figure 2 A schematic diagram of a module for optimizing parameters of a cross-modal retrieval model in a cross-modal retrieval model optimization method based on image-text importance consistency provided by the present invention;

[0038] Figure 3 A schematic diagram of the process of obtaining the intra-modal semantic importance distribution and inter-modal semantic importance distribution corresponding to an image in the cross-modal retrieval model optimization method based on image-text importance consistency provided by the present invention;

[0039] Figure 4 A schematic diagram of a process for obtaining the intra-modal semantic importance distribution and inter-modal semantic importance distribution corresponding to a text in a cross-modal retrieval model optimization method based on image-text importance consistency provided by the present invention;

[0040] Figure 5 A structural principle diagram of an embodiment of a cross-modal retrieval model optimization device based on image-text importance consistency provided by the present invention;

[0041] Figure 6 A schematic diagram of the principles of an embodiment of a terminal provided by the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] The cross-modal retrieval model optimization method based on image-text importance consistency provided by the present invention can be applied to a terminal with computing capabilities. The terminal can execute the cross-modal retrieval model optimization method based on image-text importance consistency provided by the present invention to optimize the parameters of the cross-modal retrieval model. The terminal can be but is not limited to various computers, mobile terminals, smart home appliances, wearable devices, etc.

[0044] Embodiment 1

[0045] like Figure 1 As shown, in one embodiment of the cross-modal retrieval model optimization method based on image-text importance consistency, the steps include:

[0046] S100. Obtain an initial cross-modal retrieval model, wherein the initial cross-modal retrieval model includes a target self-attention module.

[0047] The present invention is directed to the optimization of a cross-modal retrieval model with a self-attention module. The initial cross-modal retrieval model may be a pre-trained model or a model that has only undergone parameter initialization and has not been trained. The initial cross-modal retrieval model includes a target self-attention module, and the initial cross-modal retrieval model may also include other modules, such as a feature extraction module, a convolution module, etc. The input of the initial cross-modal retrieval model is an image-text pair, and the output is the similarity between the image and the text included in the image-text pair, thereby realizing cross-modal retrieval of images and texts. During the pre-training process of the initial cross-modal retrieval model, an image-text pair with a similarity label is input into the model, and the parameters of the model are updated according to the difference between the predicted similarity output by the model and the similarity label to realize training.

[0048] S200. Select a target batch from a training set, wherein the target batch includes at least one first image-text pair, wherein the first image-text pair includes a first sample image and a first sample text that match each other, and input connection features of an image feature set corresponding to the first sample image and a text feature set corresponding to the first sample text into a target self-attention module in the at least one attention module, wherein the image feature set includes features corresponding to multiple regions of the first sample image, and the text feature set includes features corresponding to multiple words in the first sample text.

[0049] The training set includes multiple images and multiple texts. The target batch can be formed by selecting an image and at least one text matching the image and several texts not matching the image from the training set. The target batch also includes at least one second image-text pair. The second image-text pair includes a second sample image and a second sample text that do not match each other. The image in the target batch and each text matching the image form a first image-text pair, and each text matching the image forms a second image-text pair. When the semantics of the image and the text are consistent, the two are said to match each other. If the semantics of the image and the text are inconsistent, the two are said to not match each other. The matching relationship between the images and texts in the training set can be obtained by manual annotation.

[0050] Specifically, the image feature set corresponding to the first sample image can be obtained by dividing the first sample image into multiple regions and then extracting features from each region, or the image feature set corresponding to the first sample image can be obtained by extracting the region containing the target in the first sample image through the target detection module in advance, and then extracting features from the region containing the target to obtain the image feature set corresponding to the first sample image, or further processing the features extracted from the region containing the target to obtain the image feature set corresponding to the first sample image, that is, the image region corresponding to the features in the image feature set is the region including the target obtained after target detection on the first sample image. The target detection module can adopt the structure of an existing target detection model (such as Fast-RCNN, etc.). The text feature set can be obtained by tokenizing multiple words in the first sample text using a pre-trained corpus or by further extracting features after tokenization.

[0051] In the initial cross-modal retrieval model, after obtaining the image feature set and the text feature set, the image feature set and the text feature set are connected to obtain a connection feature, that is, a feature with N V The feature set X of the local area of ​​the image V , and N L The feature set X of the local area of ​​the image L , after connection, the length is N = N V +N L The connection feature X = [X V |X L ], input the connected features into the target self-attention module, in which each feature is multiplied by the query (Q) matrix to obtain a query vector, each feature is also multiplied by the key (K) matrix to obtain a key vector, the query vector is multiplied by the transposed key vector to obtain a self-attention matrix, each feature is also multiplied by the value (V) matrix to obtain a value vector, the self-attention matrix is ​​processed and normalized to obtain the importance score of a single feature, and then multiplied by the value vector to obtain the output of the target self-attention module, and the similarity score between the first sample image and the first sample text is obtained based on the output of the target self-attention module. In the self-attention module of the initial cross-modal retrieval model, the features after the image features and the text features are connected are processed as a whole to obtain the importance score of a single feature, and a finer-grained semantic information importance alignment is not achieved.

[0052] The inventors studied the operation process of the self-attention mechanism and found that the self-attention matrix in the self-attention module after feature connection can be divided according to the number of image features and the number of text features, and a finer-grained semantic importance consistency constraint is achieved based on the divided self-attention matrix. Specifically, the method provided in this embodiment also includes the steps of:

[0053] S300, obtaining the self-attention matrix in the target self-attention module, the self-attention matrix is ​​obtained based on the query vector and the key vector in the target self-attention module, and the self-attention matrix is ​​divided according to the number of features in the image feature set and the number of features in the text feature set to obtain four sub-matrices, namely the first sub-matrix, the second sub-matrix, the third sub-matrix and the fourth sub-matrix, wherein the size of the first sub-matrix is ​​N V *N V , the size of the second submatrix is ​​N V *N L , the size of the third submatrix is ​​N L *N V , the size of the fourth submatrix is ​​N L *N L , N V is the number of features in the image feature set, N L is the number of features in the text feature set.

[0054] According to the calculation formula of the self-attention mechanism, the inventors found that the query vector in the target self-attention module can be expressed as: The key vector can be expressed as: Among them, W Q and W K is the learnable parameter in the attention module, then the self-attention matrix S can be divided into S VV , S VL , S LV and S LL Four sub-matrices:

[0055]

[0056] Among them, S vv is the first sub-matrix, S VL is the second sub-matrix, S LV is the third sub-matrix, S LL is the fourth submatrix, that is, the first submatrix is ​​the self-attention matrix with a size of N in the upper left corner V *N V The second submatrix is ​​the self-attention matrix with a size of N in the upper right corner. V *NL The third submatrix is ​​the self-attention matrix with a size of N at the bottom left. L *N V The fourth submatrix is ​​the self-attention matrix with a size of N at the bottom right. L *N L According to the splitting process described above, it can be seen that the importance scores in the first submatrix all come from the features of the first sample image, the importance scores in the third submatrix all come from the features of the first sample image and the first sample text, the importance scores in the second submatrix all come from the features of the first sample text and the first sample image, and the importance scores in the fourth submatrix all come from the features of the first sample text. In order to achieve more fine-grained semantic importance alignment, the method provided in this embodiment further includes the steps of:

[0057] S400, obtaining a first modality intra-semantic importance distribution and a first inter-modality semantic importance distribution corresponding to the first sample image according to the first submatrix and the third submatrix, and a second inter-modality semantic importance distribution and a second modality intra-semantic importance distribution corresponding to the first sample text according to the second submatrix and the fourth submatrix;

[0058] S500, determine the training loss based on the consistency between the semantic importance distribution within the first modality and the semantic importance distribution between the first modalities, and the consistency between the semantic importance distribution within the second modality and the semantic importance distribution between the second modalities, and optimize the parameters of the initial cross-modal retrieval model based on the training loss.

[0059] Specifically, obtaining the first intra-modality semantic importance distribution and the first inter-modality semantic importance distribution corresponding to the first sample image according to the first sub-matrix and the third sub-matrix includes:

[0060] Performing a sum operation on the elements in each column of the first submatrix to obtain a first modality intra-semantic importance score, and performing a normalization operation on each of the first modality intra-semantic importance scores to obtain a first modality intra-semantic importance distribution;

[0061] A sum operation is performed on the elements in each column of the third submatrix to obtain a first inter-modal semantic importance score, and a normalization operation is performed on each of the first inter-modal semantic importance scores to obtain a first inter-modal semantic importance distribution.

[0062] The second inter-modal semantic importance distribution and the second intra-modal semantic importance distribution corresponding to the first sample text according to the second sub-matrix and the fourth sub-matrix include:

[0063] Performing a sum operation on the elements in each column of the second submatrix to obtain a second inter-modal semantic importance score, and performing a normalization operation on each of the second inter-modal semantic importance scores to obtain a second inter-modal semantic importance distribution;

[0064] A sum operation is performed on the elements in each column of the fourth submatrix to obtain a second modality intra-semantic importance score, and a normalization operation is performed on each of the second modality intra-semantic importance scores to obtain a second modality intra-semantic importance distribution.

[0065] like Figure 3 and Figure 4 As shown, in order to achieve a more fine-grained importance consistency constraint, in this embodiment, the four sub-matrices are added on the columns as the fine-grained item semantic importance. The calculation process of the intra-modal semantic importance and inter-modal semantic importance of visual items is taken as an example: the intra-modal semantic importance of visual items is calculated by calculating S VV Add up the columns to get the semantic importance scores of all visual items, such as Figure 2 As shown in Figure 2, these scores are normalized using softmax to obtain the intra-modal semantic importance distribution A of the visual item. V , and the inter-modal semantic importance distribution of visual items is B V Yes S LV The calculation process of text mode is the same as that of visual mode.

[0066] For more fine-grained items, their importance should remain consistent, that is, the importance of items within a modality and between modalities should be consistent. Therefore, in this embodiment, after obtaining the importance distribution within the first modality, the importance distribution between the first modalities, the importance distribution within the second modality, and the importance distribution between the second modalities, a consistency constraint between the importance distribution within the first modality and the importance distribution between the first modalities, and a consistency constraint between the importance distribution within the second modality and the importance distribution between the second modalities are constructed to optimize the parameters of the model so that the cross-modal retrieval model learns a more reasonable embedding representation.

[0067] The determining of the training loss according to the consistency between the first modality intra-semantic importance distribution and the first inter-modality semantic importance distribution, and the consistency between the second modality intra-semantic importance distribution and the second inter-modality semantic importance distribution, comprises:

[0068] Determining a first loss according to a relative entropy between the first intra-modality semantic importance distribution and the first inter-modality semantic importance distribution;

[0069] determining a second loss according to a relative entropy between the second intra-modality semantic importance distribution and the second inter-modality semantic importance distribution;

[0070] Obtaining an optimization loss, wherein the optimization loss is obtained based on the first loss and the second loss corresponding to the at least one first image-text pair;

[0071] Obtaining the predicted similarity of each image-text pair in the target batch output by the initial cross-modal retrieval model, and obtaining a third loss according to the similarity difference corresponding to each image-text pair in the target batch, wherein the similarity difference corresponding to the image-text pair is constructed by the image and text matching relationship included in the image-text pair, and the predicted similarity corresponding to the image-text pair;

[0072] The training loss is determined according to the optimization loss and the second loss.

[0073] Specifically, each training batch corresponds to one training loss, that is, each image-text pair in the target batch is respectively input into the initial cross-modal retrieval model to obtain the predicted similarity of each image-text pair output by the initial cross-modal retrieval model, wherein for the second image-text pair, the importance distribution calculation process of the previous text is not performed, while for the first image-text pair, the importance distribution calculation process of the previous text is performed. In the process of determining the training loss according to the first loss and the second loss, the image-text pair similarity prediction performance of the initial cross-modal retrieval model is further aggregated. For each of the first image-text pairs in the target batch, a first loss and a second loss can be calculated respectively, and the first loss and the second loss are summed to obtain the optimized score loss corresponding to the first image-text pair, and the optimized score losses corresponding to all the first image-text pairs are summed to obtain the optimized loss. For each image-text pair in the target batch (including the first image-text pair and the second image-text pair), the similarity difference corresponding to the image-text pair can be constructed based on the predicted similarity output by the model after the image-text pair is input into the initial cross-modal retrieval model, and the matching relationship between the image and the text included in the image-text pair. The similarity difference corresponding to each image-text pair in the target batch is summed to obtain the third loss. The optimization loss and the third loss are weightedly summed to obtain the training loss, and the learnable parameters of the initial cross-modal retrieval model are optimized based on the optimization goal of minimizing the training loss.

[0074] In this embodiment, the parameters of the initial cross-modal retrieval model are optimized in an iterative manner, that is, each time the parameters of the initial cross-modal retrieval model are optimized according to a training batch, a new training batch is selected to continue to optimize the parameters on the basis of the initial cross-modal retrieval model optimized last time, and when the parameters of the initial cross-modal retrieval model converge, the optimization is terminated, that is, after the parameters of the initial cross-modal retrieval model are updated according to the training loss, it includes:

[0075] The step of selecting a target batch from the training set is repeated until the parameters of the initial cross-modal retrieval model converge.

[0076] In summary, this embodiment provides a cross-modal retrieval model optimization method based on image-text importance consistency. For the self-attention matrix in the self-attention module, it is split according to the number of image region features and the number of text word features to obtain four sub-matrices. The sub-matrices are used to measure the inter-modal semantic importance and intra-modal semantic importance of the fragmented semantics of vision and text, and the parameters of the model are optimized according to the consistency constraints between the inter-modal semantic importance distribution and the inter-modal semantic importance distribution. The fine-grained semantic information in the image-text pair is mined, and a finer-grained alignment between different modalities is achieved, effectively improving the retrieval accuracy of the cross-modal retrieval model.

[0077] It should be understood that, although the various steps in the flowcharts given in the accompanying drawings of the present specification are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0078] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0079] Embodiment 2

[0080] Based on the above embodiments, the present invention also provides a cross-modal retrieval model optimization device based on image-text importance consistency, such as Figure 5 As shown, the cross-modal retrieval model optimization device based on image-text importance consistency includes:

[0081] A model acquisition module, used to acquire an initial cross-modal retrieval model, wherein the initial cross-modal retrieval model includes a target self-attention module, as specifically described in Embodiment 1;

[0082] A feature acquisition module is used to select a target batch from a training set, wherein the target batch includes at least one first image-text pair, wherein the first image-text pair includes a first sample image and a first sample text that match each other, and input a connection feature of an image feature set corresponding to the first sample image and a text feature set corresponding to the first sample text into a target self-attention module in the at least one attention module, wherein the image feature set includes features corresponding to a plurality of regions of the first sample image, and the text feature set includes features corresponding to a plurality of words in the first sample text, as described in Embodiment 1;

[0083] A matrix segmentation module is used to obtain the self-attention matrix in the target self-attention module, wherein the self-attention matrix is ​​obtained based on the query vector and the key vector in the target self-attention module, and the self-attention matrix is ​​segmented according to the number of features in the image feature set and the number of features in the text feature set to obtain four sub-matrices, namely, the first sub-matrix, the second sub-matrix, the third sub-matrix and the fourth sub-matrix, wherein the size of the first sub-matrix is ​​N V *N V , the size of the second submatrix is ​​N V *N L , the size of the third submatrix is ​​N L *N V , the size of the fourth submatrix is ​​N L *N L , N V is the number of features in the image feature set, N L is the number of features in the text feature set, as specifically described in Embodiment 1;

[0084] An importance distribution determination module is used to obtain a first modality intra-semantic importance distribution and a first inter-modality semantic importance distribution corresponding to the first sample image according to the first submatrix and the third submatrix, and a second inter-modality semantic importance distribution and a second modality intra-semantic importance distribution corresponding to the first sample text according to the second submatrix and the fourth submatrix, as specifically described in Embodiment 1;

[0085] A parameter optimization module is used to determine the training loss based on the semantic importance distribution within the first modality, the semantic importance distribution between the first modalities, the semantic importance distribution within the second modality, and the semantic importance distribution between the second modalities, and optimize the parameters of the initial cross-modal retrieval model based on the training loss, as specifically described in Example 1.

[0086] Embodiment 3

[0087] Based on the above embodiments, the present invention also provides a terminal, such as Figure 6 As shown, the terminal includes a processor 10 and a memory 20. Figure 6 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0088] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a cross-modal retrieval model optimization program 30 based on image-text importance consistency is stored on the memory 20, and the cross-modal retrieval model optimization program 30 based on image-text importance consistency can be executed by the processor 10, thereby realizing the cross-modal retrieval model optimization method based on image-text importance consistency in the present application.

[0089] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other chip, used to run the program code or process data stored in the memory 20, such as executing the cross-modal retrieval model optimization method based on image-text importance consistency.

[0090] In one embodiment, when the processor 10 executes the cross-modal retrieval model optimization program 30 based on image-text importance consistency in the memory 20, the following steps are implemented:

[0091] Acquire an initial cross-modal retrieval model, wherein the initial cross-modal retrieval model includes a target self-attention module;

[0092] A target batch is selected from a training set, wherein the target batch includes at least one first image-text pair, wherein the first image-text pair includes a first sample image and a first sample text that match each other, and a connection feature of an image feature set corresponding to the first sample image and a text feature set corresponding to the first sample text is input into a target self-attention module, wherein the image feature set includes features corresponding to a plurality of regions of the first sample image, and the text feature set includes features corresponding to a plurality of words in the first sample text;

[0093] Obtain a self-attention matrix in the target self-attention module, wherein the self-attention matrix is ​​obtained based on the query vector and the key vector in the target self-attention module, and the self-attention matrix is ​​divided according to the number of features in the image feature set and the number of features in the text feature set to obtain four sub-matrices, namely, a first sub-matrix, a second sub-matrix, a third sub-matrix and a fourth sub-matrix, wherein the size of the first sub-matrix is ​​N V *N V , the size of the second submatrix is ​​N V *N L , the size of the third submatrix is ​​N L *N V , the size of the fourth submatrix is ​​N L *N L , N V is the number of features in the image feature set, N L is the number of features in the text feature set;

[0094] Acquire the first modality intra-semantic importance distribution and the first inter-modality semantic importance distribution corresponding to the first sample image according to the first submatrix and the third submatrix, and acquire the second inter-modality semantic importance distribution and the second modality intra-semantic importance distribution corresponding to the first sample text according to the second submatrix and the fourth submatrix;

[0095] The training loss is determined according to the consistency between the semantic importance distribution within the first modality and the semantic importance distribution between the first modalities, and the consistency between the semantic importance distribution within the second modality and the semantic importance distribution between the second modalities, and the parameters of the initial cross-modal retrieval model are optimized according to the training loss.

[0096] Embodiment 4

[0097] The present invention also provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the cross-modal retrieval model optimization method based on image-text importance consistency as described above.

[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A cross-modal retrieval model optimization method based on image-text importance consistency, characterized in that: The method comprises: Acquire an initial cross-modal retrieval model, wherein the initial cross-modal retrieval model includes a target self-attention module; A target batch is selected from a training set, wherein the target batch includes at least one first image-text pair, wherein the first image-text pair includes a first sample image and a first sample text that match each other, and a connection feature of an image feature set corresponding to the first sample image and a text feature set corresponding to the first sample text is input into the target self-attention module, wherein the image feature set includes features corresponding to a plurality of regions of the first sample image, and the text feature set includes features corresponding to a plurality of words in the first sample text; Obtain the self-attention matrix in the target self-attention module, the self-attention matrix is ​​obtained based on the query vector and the key vector in the sub-target attention module, and the self-attention matrix is ​​divided according to the number of features in the image feature set and the number of features in the text feature set to obtain four sub-matrices, namely the first sub-matrix, the second sub-matrix, the third sub-matrix and the fourth sub-matrix, wherein the size of the first sub-matrix is ​​N V *N V , the size of the second submatrix is ​​N V *N L , the size of the third submatrix is ​​N L *N V , the size of the fourth submatrix is ​​N L *N L , N V is the number of features in the image feature set, N L is the number of features in the text feature set; Obtaining a first modality intra-semantic importance distribution and a first inter-modality semantic importance distribution corresponding to the first sample image according to the first submatrix and the third submatrix, and obtaining a second inter-modality semantic importance distribution and a second modality intra-semantic importance distribution corresponding to the first sample text according to the second submatrix and the fourth submatrix; Determining a training loss according to the consistency between the semantic importance distribution within the first modality and the semantic importance distribution between the first modalities, and the consistency between the semantic importance distribution within the second modality and the semantic importance distribution between the second modalities, and optimizing the parameters of the initial cross-modal retrieval model according to the training loss; The acquiring, according to the first submatrix and the third submatrix, the first intra-modality semantic importance distribution and the first inter-modality semantic importance distribution corresponding to the first sample image includes: Performing a sum operation on the elements in each column of the first submatrix to obtain a semantic importance score within the first modality, and performing a normalization operation on each of the semantic importance scores within the first modality to obtain a semantic importance distribution within the first modality; Performing a sum operation on the elements in each column of the third submatrix to obtain a first inter-modal semantic importance score, and performing a normalization operation on each of the first inter-modal semantic importance scores to obtain a first inter-modal semantic importance distribution; The second inter-modal semantic importance distribution and the second intra-modal semantic importance distribution corresponding to the first sample text according to the second sub-matrix and the fourth sub-matrix include: Performing a sum operation on the elements in each column of the second submatrix to obtain a semantic importance score within the second modality, and performing a normalization operation on each of the semantic importance scores within the second modality to obtain a semantic importance distribution between the second modalities; A sum operation is performed on the elements in each column of the fourth submatrix to obtain a second inter-modal semantic importance score, and a normalization operation is performed on each of the second inter-modal semantic importance scores to obtain a second intra-modal semantic importance distribution.

2. The cross-modal retrieval model optimization method based on image-text importance consistency according to claim 1, characterized in that: The initial cross-modal retrieval model is a pre-trained model.

3. The cross-modal retrieval model optimization method based on image-text importance consistency according to claim 1, characterized in that: The image region corresponding to the features in the image feature set is a region including the target obtained after target detection is performed on the first sample image.

4. The cross-modal retrieval model optimization method based on image-text importance consistency according to claim 1, characterized in that: After updating the parameters of the initial cross-modal retrieval model according to the training loss, the method further comprises: The step of selecting a target batch from the training set is repeated until the parameters of the initial cross-modal retrieval model converge.

5. The cross-modal retrieval model optimization method based on image-text importance consistency according to claim 1, characterized in that: The target batch also includes at least one second image-text pair, the second image-text pair includes a second sample image and a second sample text that do not match each other, and the determining of the training loss according to the consistency between the semantic importance distribution within the first modality and the semantic importance distribution between the first modalities, and the consistency between the semantic importance distribution within the second modality and the semantic importance distribution between the second modalities, includes: Determining a first loss according to a relative entropy between the first intra-modality semantic importance distribution and the first inter-modality semantic importance distribution; determining a second loss according to a relative entropy between the second intra-modality semantic importance distribution and the second inter-modality semantic importance distribution; Obtaining an optimization loss, wherein the optimization loss is obtained based on the first loss and the second loss corresponding to the at least one first image-text pair; Obtaining the predicted similarity of each image-text pair in the target batch output by the initial cross-modal retrieval model, and obtaining a third loss according to the similarity difference corresponding to each image-text pair in the target batch, wherein the similarity difference corresponding to the image-text pair is constructed by the image and text matching relationship included in the image-text pair, and the predicted similarity corresponding to the image-text pair; The training loss is determined according to the optimization loss and the third loss.

6. A cross-modal retrieval model optimization device based on image-text importance consistency, characterized in that: The device comprises: A model acquisition module, used to acquire an initial cross-modal retrieval model, wherein the initial cross-modal retrieval model includes a target self-attention module; A feature acquisition module is used to select a target batch in a training set, wherein the target batch includes at least one first image-text pair, wherein the first image-text pair includes a first sample image and a first sample text that match each other, and input a connection feature of an image feature set corresponding to the first sample image and a text feature set corresponding to the first sample text into a target self-attention module in at least one attention module, wherein the image feature set includes features corresponding to a plurality of regions of the first sample image, and the text feature set includes features corresponding to a plurality of words in the first sample text; A matrix segmentation module is used to obtain the self-attention matrix in the target self-attention module, the self-attention matrix is ​​obtained based on the query vector and the key vector in the sub-target attention module, and the self-attention matrix is ​​segmented according to the number of features in the image feature set and the number of features in the text feature set to obtain four sub-matrices, namely the first sub-matrix, the second sub-matrix, the third sub-matrix and the fourth sub-matrix, wherein the size of the first sub-matrix is ​​N V *N V , the size of the second submatrix is ​​N V *N L , the size of the third submatrix is ​​N L *N V , the size of the fourth submatrix is ​​N L *N L , N V is the number of features in the image feature set, N L is the number of features in the text feature set; An importance distribution determination module is used to obtain a first modality intra-semantic importance distribution and a first inter-modality semantic importance distribution corresponding to the first sample image according to the first submatrix and the third submatrix, and a second inter-modality semantic importance distribution and a second modality intra-semantic importance distribution corresponding to the first sample text according to the second submatrix and the fourth submatrix; The acquiring, according to the first submatrix and the third submatrix, the first intra-modality semantic importance distribution and the first inter-modality semantic importance distribution corresponding to the first sample image includes: Performing a sum operation on the elements in each column of the first submatrix to obtain a semantic importance score within the first modality, and performing a normalization operation on each of the semantic importance scores within the first modality to obtain a semantic importance distribution within the first modality; Performing a sum operation on the elements in each column of the third submatrix to obtain a first inter-modal semantic importance score, and performing a normalization operation on each of the first inter-modal semantic importance scores to obtain a first inter-modal semantic importance distribution; The second inter-modal semantic importance distribution and the second intra-modal semantic importance distribution corresponding to the first sample text according to the second sub-matrix and the fourth sub-matrix include: Performing a sum operation on the elements in each column of the second submatrix to obtain a semantic importance score within the second modality, and performing a normalization operation on each of the semantic importance scores within the second modality to obtain a semantic importance distribution between the second modalities; Performing a sum operation on the elements in each column of the fourth submatrix to obtain a second inter-modal semantic importance score, and performing a normalization operation on each of the second inter-modal semantic importance scores to obtain a second intra-modal semantic importance distribution; A parameter optimization module is used to determine the training loss according to the semantic importance distribution within the first modality, the semantic importance distribution between the first modalities, the semantic importance distribution within the second modality, and the semantic importance distribution between the second modalities, and optimize the parameters of the initial cross-modal retrieval model according to the training loss.

7. A terminal, characterized in that: The terminal includes: a processor, a computer-readable storage medium communicatively connected to the processor, the computer-readable storage medium being suitable for storing a plurality of instructions, and the processor being suitable for calling the instructions in the computer-readable storage medium to execute the steps of the cross-modal retrieval model optimization method based on image-text importance consistency as described in any one of claims 1 to 5 above.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the cross-modal retrieval model optimization method based on image-text importance consistency as described in any one of claims 1-5.

Images