A supervised cross-modal retrieval method, device, equipment and medium

Abstract

The present invention discloses a supervised cross-modal retrieval method, device, equipment and medium, the method includes: extracting features from training sample data of image modality and text modality; mapping the extracted image data features and text data features to a common representation space; respectively calculating the loss of the label space, the loss within each modality and between different modalities in the common representation space, and the invariance loss between the image and text modalities, and adding different weights to obtain the loss function of the retrieval model; optimizing the parameters of the retrieval model by minimizing the loss function; mapping the target retrieval data to the common representation space using the optimized retrieval model, calculating the similarity between the target retrieval data and the data in the image and text data set, and obtaining the corresponding retrieval ranking result. In this way, the discriminability of different semantic data samples and the semantic information of the original data are retained, the correlation between cross-modal data can be calculated more effectively, and the retrieval accuracy is higher.

Description

Technical Field

[0001] The present invention relates to the field of data retrieval technology, and more specifically, to a supervised cross-modal retrieval method, device, equipment and medium. Background Art

[0002] With the rapid development of science and technology, the forms of generation and acquisition of scientific and technological information are becoming increasingly diverse. The data representation forms of various types of scientific and technological information are rich and diverse, and have gradually changed from single text data to mixed data types of other modalities such as pictures and videos with more vivid expressions and richer content. Traditional single-modal retrieval methods have good query effects on a single modality, but due to the possible feature heterogeneity and weak correlation between data of different modalities, the feature vectors of data of different modalities cannot be directly involved in the calculation due to different dimensions and attributes, making single-modal retrieval unsuitable for retrieval between multiple modalities. Cross-modal retrieval methods use the semantic similarity between different modalities to retrieve similar content between multi-modal data. Through cross-modal retrieval methods, the needs of multi-angle intelligent analysis of multi-modal scientific and technological information can be met.

[0003] The core of cross-modal retrieval lies in how to measure the content similarity between different modal data, that is, to solve the heterogeneity between different modal data. Representation learning is a general method to overcome the heterogeneity between different modal data. This method aims to design a suitable function to map different modal data to a common representation space. In this representation space, since the dimensions of the data are consistent, the similarity between different modal data can be directly solved. In order to construct an appropriate representation space, researchers have proposed many methods for designing mapping functions.

[0004] Traditional methods use statistical correlation analysis to learn linear functions by optimizing target statistics, but the correlation between multimodal data in the real world is complex, and linear functions cannot fully model the mapping relationship. Due to the excellent performance of deep neural networks in the field of representation learning, a large number of deep learning-based methods are used to learn the common representation space of multimodal data. Compared with unsupervised methods, supervised deep learning methods can be used to learn more discriminative representation features, so that different categories of data can be better separated in the common representation space. Existing supervised cross-modal retrieval methods include using label information to learn discriminative features between multimodal data, using classification information to learn semantics or discriminability within each modality, etc. Although these methods use classification information, this classification information is only used to learn discriminative features within each modality or between multiple modalities, so these cross-modal methods do not fully utilize the semantic information in the original data. Summary of the invention

[0005] In view of the defects of the prior art and the need for improvement, the present invention provides a supervised cross-modal retrieval method, apparatus, device and medium, which aims to solve the technical problem that the existing cross-modal retrieval method does not fully utilize the semantic information in the original data, resulting in low retrieval accuracy.

[0006] To achieve the above-mentioned purpose, the present invention provides a supervised cross-modal retrieval method, comprising the following steps: S1: extracting features from training sample data of image modality and text modality respectively, wherein the training sample data is obtained from a picture and text data set; S2: mapping the extracted image data features and text data features to a common representation space; S3: calculating the loss of the label space, the loss within each modality and between different modalities in the common representation space, and the invariance loss between the image and text modalities respectively, and adding different weights to obtain a loss function of the retrieval model; S4: optimizing the parameters of the retrieval model by minimizing the loss function to obtain an optimized retrieval model; S5: mapping the target retrieval data to the common representation space using the optimized retrieval model, calculating the similarity between the target retrieval data and the data in the picture and text data set, thereby obtaining a retrieval ranking result corresponding to the target retrieval data;

[0007] The loss function is expressed as: L = λL1 + μL2 + ηL3, where

[0008] L1 is the loss of label space, n is the number of image-text data pairs, ||·|| F represents the Frobenius norm, P is the projection matrix of the linear classifier, α1 and β1 are the weights corresponding to the image and text prediction labels, respectively, U, V, and Y are the representation matrices of the image, text, and corresponding labels in the common representation space, respectively;

[0009]

[0010] The likelihood function that measures the similarity between modalities is:

[0011]

[0012] L2 is the loss within each mode and between different modes in the common representation space, Γ ij =cos(u i ,v j ), Φ ij =cos(u i ,u j ), Θ ij = cos(v i ,v j ), Among them, cos is the cosine function, which is used to measure the similarity; sgn is the sign function, which is 1 if the two elements belong to the same class, otherwise it is 0; For mapping image modalities, Used to map text modalities, where is the i-th image sample, is the jth text sample, Υ α and β are learnable parameters;

[0013] L3 is the invariance loss between image and text modalities;

[0014] λ, μ, and η are the weight coefficients of L1, L2, and L3 respectively;

[0015] In step S5, the similarity between the target retrieval data and the data in the image and text data set is calculated, including: by weighted averaging the cross-modal data similarity and the same-modal data similarity, the similarity S between the target retrieval data and the data in the image and text data set is calculated, and the calculation formula is:

[0016] S=α2Similarity(x,U)+β2Similarity(x,V)

[0017] Among them, α2 and β2 are weights, Similarity(·) is a function to measure similarity, and x is the input image or text data; then they are sorted, and the top-ranked data is taken as the retrieval ranking result corresponding to the target retrieval data.

[0018] Furthermore, the step S1 includes: S11: using a deep convolutional neural network to perform feature extraction on the training sample data of the image modality, and adding a first fully connected layer after the image extraction subnetwork; S12: using a natural language processing model to perform feature extraction on the training sample data of the text modality, and adding a second fully connected layer after the text extraction subnetwork.

[0019] Furthermore, the step S2 includes: adding a third fully connected layer after the first fully connected layer and the second fully connected layer, and mapping the extracted image data features and text data features to a common representation space through the third fully connected layer.

[0020] Furthermore, a linear classifier is added after the third fully connected layer to predict the categories of the image and text, and compared with the actual categories, so as to calculate the loss of the label space.

[0021] Another aspect of the present invention provides a supervised cross-modal retrieval device, which is used to execute any of the supervised cross-modal retrieval methods described above, and the supervised cross-modal retrieval device comprises:

[0022] A feature extraction module, used to extract features from training sample data of image modality and text modality respectively, wherein the training sample data is obtained from the image and text dataset;

[0023] A common representation space learning module, used for mapping the extracted image data features and text data features to a common representation space;

[0024] A loss function calculation module, used to respectively calculate the loss of the label space, the loss within each modality and between different modalities in the common representation space, and the invariance loss between the image and text modalities, and add different weights to obtain the loss function of the retrieval model;

[0025] A retrieval model optimization module, used to optimize the parameters of the retrieval model by minimizing the loss function to obtain an optimized retrieval model;

[0026] The retrieval result determination module is used to map the target retrieval data to the public representation space using the optimized retrieval model, calculate the similarity between the target retrieval data and the data in the graphic data set, and thus obtain the retrieval ranking result corresponding to the target retrieval data.

[0027] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of any one of the above-mentioned supervised cross-modal retrieval methods when executing the computer program.

[0028] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-mentioned supervised cross-modal retrieval methods are implemented.

[0029] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0030] (1) The present invention maps the extracted image data features and text data features to a common representation space; calculates the loss of the label space, the loss within each modality and between different modalities in the common representation space, and the invariance loss between the image and text modalities, and adds different weights to obtain the loss function of the retrieval model; optimizes the parameters of the retrieval model by minimizing the loss function to obtain the optimized retrieval model. An end-to-end supervised cross-modal deep learning framework is adopted, which makes full use of the classification information of multimodal data to learn the common representation space, which not only retains the discriminability of different semantic data samples, but also eliminates the differences between cross-modal data. Compared with the traditional hash-based retrieval method, the real-valued cross-modal representation learning method is adopted, which retains the semantic information of the original data, can more effectively calculate the correlation between cross-modal data, eliminates the influence of heterogeneity between different modalities, and has a higher retrieval accuracy.

[0031] (2) The present invention makes full use of classification information to learn the common representation space. At the same time, the present method is a real-valued cross-modal representation learning method. Compared with the binary representation learning method, or the cross-modal hash retrieval method, the present method retains the information of the original data and has a higher retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flowchart of a supervised cross-modal retrieval method provided by the present invention;

[0033] Figure 2 It is a structural block diagram of a supervised cross-modal retrieval device provided by the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The characteristics, operations or features described in the specification can be combined in any appropriate manner to form various implementation methods. At the same time, the steps or actions in the method description can also be replaced or adjusted in order in a manner that is obvious to those skilled in the art. Therefore, the various sequences in the specification and the accompanying drawings are only for the purpose of clearly describing a certain embodiment and do not mean that they are necessary sequences, unless otherwise specified that a certain sequence must be followed. In addition, the technical features involved in the various implementation methods of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0035] On the one hand, the present invention provides a cross-modal retrieval method based on supervision, such as Figure 1 As shown, the following steps are included:

[0036] S1: performing feature extraction on training sample data of image modality and text modality respectively, wherein the training sample data is obtained from an image and text dataset;

[0037] S2: Mapping the extracted image data features and text data features to a common representation space;

[0038] S3: respectively calculating the loss of the label space, the loss within each modality and between different modalities in the common representation space, and the invariance loss between the image and text modalities, and adding different weights to obtain the loss function of the retrieval model;

[0039] S4: Optimizing the parameters of the retrieval model by minimizing the loss function to obtain an optimized retrieval model;

[0040] S5: Mapping the target retrieval data to the public representation space using the optimized retrieval model, calculating the similarity between the target retrieval data and the data in the graphic and text data set, thereby obtaining a retrieval ranking result corresponding to the target retrieval data.

[0041] The present invention supervises the model to learn discriminative features by minimizing the discriminative loss in both the label space and the common representation space. At the same time, the modality-invariant features in the common representation space are learned by minimizing the invariance loss between different modalities and using a weight sharing strategy. According to this learning strategy, both paired label information and classification information are utilized as fully as possible to ensure that the learned representation is discriminative in semantic structure and invariant in all modalities.

[0042] Specifically, the following steps are included:

[0043] 1. Data preprocessing and image text feature extraction

[0044] The present invention can be applied to cross-modal retrieval between different modalities. The introduction mainly takes image and text data as an example. Assuming there are n image-text data pairs, defined as in is the i-th image sample, is the i-th text sample. Each image-text pair corresponds to a semantic label vector c represents the number of directories. If the i-th instance belongs to the j-th directory, the corresponding component in the label vector is 1, otherwise it is 0.

[0045] Image data preprocessing includes resizing, cropping, normalization, etc. Text preprocessing includes denoising, word segmentation, stop word filtering, etc. The following introduces the feature extraction method.

[0046] For image data, a deep convolutional neural network is used to extract the feature vector of the image. An element-by-element multiplication and sum operation is performed between the convolution kernel in the convolutional neural network and the input image to project the information in the receptive field into an element in the feature map, thereby completing the feature extraction of the image. This part controls the dimensions of each layer and finally represents the image as a feature vector of a specific dimension. Generally speaking, convolutional neural networks have multiple layers, in which each subsequent layer of the neural network represents the image data in a more abstract way, so the output of the penultimate layer can be used as the feature vector. That is, using a specific convolutional neural network, a single-modal feature extraction of the image can be performed, and then the feature vector of the image is represented for the next step of representation learning. Therefore, the fc7 layer in the 19-layer VGGNet is first used to extract a 4096-dimensional high-level feature vector as the semantic vector of the image. Then, a fully connected layer is used to perform public representation learning to obtain the public representation vector u of the image. i .

[0047] For text data, a feature extraction algorithm based on language model and neural network is used to convert text information from an unstructured raw text into a high-level semantic feature vector representation. For the text sub-network, the BERT model (BERT-base, Chinese) is used to extract the feature vector of Chinese text. BERT uses the Transformer structure to construct a multi-layer bidirectional encoder network, which can directly convert the original text into a high-level semantic vector form with semantic features. Then, a fully connected layer is also used for public representation learning to obtain the public representation vector v of the text. j .

[0048] The feature extraction module is used to extract high-level semantic feature vectors of scientific information images and texts. For the image sub-network and text sub-network, one is used for image modality and the other is used for text modality, and both adopt an end-to-end training mode.

[0049] 2. Common Representation Space Learning

[0050] For the common representation learning module, in order to ensure that the two subnetworks can learn a common representation space, a fully connected layer is used at the end of the model to perform weight sharing learning. Then, based on the assumption that the representation vector in the common space is suitable for classification, a linear classifier with a parameter matrix P is connected to the end of the model to learn discriminative features by utilizing label information. Therefore, this method can well learn cross-modal correlations while accurately distinguishing the representation features of the common space.

[0051] Since image and text data are in different representation spaces, the similarity between the two cannot be directly compared. Therefore, this method learns two mapping functions to map images and text to a common representation space. Definition To map the image modality, To map the text mode, where Υ α and β are learnable parameters.

[0052] The image representation matrix in the representation space is defined as U = [u1,u2,...,u n ], the text representation matrix is ​​V = [v1,v2,...,v n ], the corresponding label representation matrix is ​​Y = [y1,y2,...,y n ].

[0053] 3. Model training and objective function

[0054] The goal of this model is to learn the semantic structure of the data, that is, to learn a common space where samples from the same semantic category should be similar, even though these data may come from different modalities, and samples from different semantic categories should be dissimilar. In order to learn the discriminative representation features of multimodal data, this method minimizes the discriminative loss in the label space and the common representation space. At the same time, this method minimizes the distance between the representations of each image-text pair to reduce cross-modal differences.

[0055] In order to preserve the distinction between samples of different categories after feature projection, this method assumes that the common representation features are very suitable for classification, and uses a linear classifier to predict the semantic labels of sample features in the common representation space. Specifically, a fully connected layer is used to connect the end of the image modality network and the text modality network. The classifier classifies the representation features of the training data in the common space to generate a predicted label for each data sample.

[0056] The objective function can be divided into three parts. The first part is to measure the loss of the label space, as shown in formula (1).

[0057]

[0058] Among them, ||·|| F represents the Frobenius norm, P is the projection matrix of the linear classifier, and α1 and β1 are the weights corresponding to the image and text prediction labels. Since the way of extracting high-level semantic vectors for images and texts is different, the prediction loss of their features mapped to the common representation space is inconsistent, so different weights are given to the image and text prediction labels to balance the difference in prediction loss.

[0059] The role of the second part of the objective function is to directly measure the discrimination loss between different modal data and within each modal data in the common representation interval, as shown in formula (2).

[0060]

[0061] Among them, Γ ij =cos(u i ,v j ), Φ ij =cos(u i ,u j ), Θ ij = cos(v i ,v j ), cos is the cosine function, which is used to measure similarity; sgn is the sign function, which is 1 if the two elements belong to the same class, otherwise it is 0.

[0062] The likelihood function that measures the similarity between modalities is shown in formula (3).

[0063]

[0064] Since the logarithm of the maximum likelihood function is the minimum likelihood function with a negative sign, formula (3) can be simplified to the first line of formula (2). It can be deduced that the cosine similarity cos(u i ,v j ) is larger, Γ ij The larger the value, the probability p(1|u i ,v j ), that is, the public representation space is classified by similarity. Similarly, the second and third lines in formula (2) measure the similarity within the image and text samples respectively. Therefore, this formula is a reasonable method to measure the similarity of the public representation space and a criterion for learning discriminative features.

[0065] To eliminate cross-modal differences, the third part of the objective function minimizes the distance between all image-text pair representations, as shown in Formula (4).

[0066]

[0067] By combining (1), (2), and (4) and applying different weights respectively, we can get the overall objective function of the model, as shown in formula (5).

[0068] L=λL1+μL2+ηL3 (5)

[0069] 4. Similarity calculation and result display

[0070] Finally, the model is trained and used. The target retrieval data is mapped to the common representation space using the trained model, and the similarity between it and the data in the scientific and technological image and text dataset is calculated. In actual use, the similarity of the same modality data can be added to improve the accuracy of the retrieval, as shown in formula (6).

[0071] S=α2Similarity(x,U)+β2Similarity(x,V) (6)

[0072] Among them, α2 and β2 are weights, Similarity(·) is a function to measure similarity, x is the input image or text data, S is the final return result, and then they are sorted, and the data with the highest sorting is taken as the final result.

[0073] Another aspect of the present invention provides a supervised cross-modal retrieval device, comprising:

[0074] A feature extraction module 11 is used to extract features from training sample data in image mode and text mode respectively, wherein the training sample data is obtained from the image and text data set;

[0075] A common representation space learning module 21, used for mapping the extracted image data features and text data features to a common representation space;

[0076] A loss function calculation module 31 is used to calculate the loss of the label space, the loss within each modality and between different modalities in the common representation space, and the invariance loss between the image and text modalities, and add different weights to obtain the loss function of the retrieval model;

[0077] A retrieval model optimization module 41, used to optimize the parameters of the retrieval model by minimizing the loss function to obtain an optimized retrieval model;

[0078] The search result determination module 51 is used to map the target search data to the public representation space using the optimized search model, calculate the similarity between the target search data and the data in the graphic data set, and thus obtain the search ranking result corresponding to the target search data.

[0079] The division of the various modules in the above-mentioned supervised cross-modal retrieval device is only for illustration. In other embodiments, the supervised cross-modal retrieval device may be divided into different modules as needed to complete all or part of the functions of the above-mentioned device.

[0080] The implementation of each module in the supervised cross-modal retrieval device provided in the embodiment of the present application can be in the form of a computer program. The computer program can be run on a terminal or a server. The program modules constituted by the computer program can be stored in the memory of an electronic device. When the computer program is executed by a processor, the steps of the method described in the embodiment of the present application are implemented.

[0081] The embodiment of the present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, enable the processors to perform the steps of the supervised cross-modal retrieval method.

[0082] A computer program product comprising instructions, when executed on a computer, causes the computer to perform a supervision-based cross-modal retrieval method.

[0083] Any reference to memory, storage, database or other medium used in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0084] It is easy for those skilled in the art to understand that the above is only a preferred embodiment of the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the technical field to which the present invention belongs, according to the idea of ​​the present invention, they can also make some simple deductions, deformations or substitutions. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A supervised cross-modal retrieval method, characterized in that: The following steps are involved: S1: performing feature extraction on training sample data of image modality and text modality respectively, wherein the training sample data is obtained from an image and text dataset; S2: Mapping the extracted image data features and text data features to a common representation space; S3: respectively calculating the loss of the label space, the loss within each modality and between different modalities in the common representation space, and the invariance loss between the image and text modalities, and adding different weights to obtain the loss function of the retrieval model; S4: Optimizing the parameters of the retrieval model by minimizing the loss function to obtain an optimized retrieval model; S5: Mapping the target retrieval data to the public representation space using the optimized retrieval model, calculating the similarity between the target retrieval data and the data in the graphic and text data set, thereby obtaining a retrieval ranking result corresponding to the target retrieval data; The loss function is expressed as: L = λL1 + μL2 + ηL3, where L1 is the loss of label space, n is the number of image-text data pairs, ||·|| F represents the Frobenius norm, P is the projection matrix of the linear classifier, α1 and β1 are the weights corresponding to the image and text prediction labels, respectively, U, V, and Y are the representation matrices of the image, text, and corresponding labels in the common representation space, respectively; The likelihood function that measures the similarity between modalities is: L2 is the loss within each mode and between different modes in the common representation space, Γ ij =cos(u i ,v j ), Φ ij =cos(u i ,u j ), Θ ij = cos(v i ,v j ), Among them, cos is the cosine function, which is used to measure the similarity; sgn is the sign function, which is 1 if the two elements belong to the same class, otherwise it is 0; For mapping image modality, Used to map text modalities, where is the i-th image sample, is the jth text sample, Υ α and β are learnable parameters; L3 is the invariance loss between image and text modalities; λ, μ, and η are the weight coefficients of L1, L2, and L3 respectively; In step S5, the similarity between the target retrieval data and the data in the image and text data set is calculated, including: by weighted averaging the cross-modal data similarity and the same-modal data similarity, the similarity S between the target retrieval data and the data in the image and text data set is calculated, and the calculation formula is: S=α2Similarity(x,U)+β2Similarity(x,V) Among them, α2 and β2 are weights, Similarity(·) is a function to measure similarity, and x is the input image or text data; then they are sorted, and the top-ranked data is taken as the retrieval ranking result corresponding to the target retrieval data.

2. The supervised cross-modal retrieval method according to claim 1, characterized in that: The step S1 includes: S11: Use a deep convolutional neural network to extract features from the training sample data of the image modality, and add the first fully connected layer after the image extraction subnetwork; S12: Use the natural language processing model to extract features from the training sample data of the text modality, and add a second fully connected layer after the text extraction subnetwork.

3. The supervised cross-modal retrieval method according to claim 2, characterized in that: The step S2 comprises: A third fully connected layer is added after the first fully connected layer and the second fully connected layer, and the extracted image data features and text data features are mapped to a common representation space through the third fully connected layer.

4. The supervised cross-modal retrieval method according to claim 3, characterized in that: A linear classifier is added after the third fully connected layer to predict the categories of images and texts, and compared with the actual categories to calculate the loss of the label space.

5. A supervised cross-modal retrieval device, characterized in that: Used to execute the supervised cross-modal retrieval method according to any one of claims 1 to 4, the supervised cross-modal retrieval device comprising: A feature extraction module, used to extract features from training sample data of image modality and text modality respectively, wherein the training sample data is obtained from the image and text dataset; A common representation space learning module, used for mapping the extracted image data features and text data features to a common representation space; A loss function calculation module, used to respectively calculate the loss of the label space, the loss within each modality and between different modalities in the common representation space, and the invariance loss between the image and text modalities, and add different weights to obtain the loss function of the retrieval model; A retrieval model optimization module, used to optimize the parameters of the retrieval model by minimizing the loss function to obtain an optimized retrieval model; The retrieval result determination module is used to map the target retrieval data to the public representation space using the optimized retrieval model, calculate the similarity between the target retrieval data and the data in the graphic data set, and thus obtain the retrieval ranking result corresponding to the target retrieval data.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images