Cross-modal retrieval method, device, equipment and storage medium

Abstract

The present invention provides a cross-modal retrieval method, device, equipment and storage medium, the method comprising: obtaining data to be retrieved, the data to be retrieved being text or image; inputting the data to be retrieved into a trained neural network model to obtain a hash code of the data to be retrieved output by the neural network, wherein the neural network model comprises a text network and an image network, the text network is used to extract the global features of the text, and generate a corresponding hash code according to the global features of the text, and the image network is used to extract the global features and local features of the image, and generate a corresponding hash code according to the global features and local features of the image; based on the hash code of the data to be retrieved, cross-modal retrieval is performed on the data to be retrieved in the data set to be retrieved. The present invention can improve the cross-modal retrieval performance.

Description

Technical Field

[0001] The present invention relates to the field of computer software technology, and in particular to a cross-modal retrieval method, device, equipment and storage medium. Background Art

[0002] With the explosive growth of multimedia data on the Internet, cross-modal retrieval has gradually become one of the hot issues in the field of artificial intelligence. Cross-modal retrieval refers to using sample data of one modality as a query to retrieve data of another modality, that is, the query data and the retrieved data belong to two different modalities. Therefore, the key to cross-modal retrieval is to break the semantic gap between data of different modalities, so as to map data of different modalities into the same feature space.

[0003] At present, most cross-modal retrieval methods are based on the approximate nearest neighbor search method, which sorts data by calculating the similarity of different modal data features. However, these methods have the problems of too much storage data and low retrieval efficiency. The emergence of cross-modal hash retrieval solves the above two problems. Cross-modal hash retrieval first maps different modal data into hash codes in the same feature space, then calculates the similarity between data through the XOR operation between hash codes, and sorts the retrieval results according to the similarity.

[0004] However, current cross-modal hash retrieval methods are usually based on two assumptions: first, images with similar text descriptions have similar contents. Second, images of the same category have good global similarity. However, data in real datasets often cannot meet the above two assumptions, thus resulting in a decrease in the performance of cross-modal hash retrieval. Summary of the invention

[0005] Embodiments of the present invention provide a cross-modal retrieval method, apparatus, device and storage medium to solve the current problem of low retrieval performance.

[0006] In a first aspect, an embodiment of the present invention provides a cross-modal retrieval method, comprising:

[0007] Obtain the data to be retrieved, where the data to be retrieved is text or image;

[0008] Input the data to be retrieved into the trained neural network model to obtain the hash code of the data to be retrieved output by the neural network, wherein the neural network model includes a text network and an image network, the text network is used to extract the global features of the text and generate the corresponding hash code according to the global features of the text, and the image network is used to extract the global features and local features of the image and generate the corresponding hash code according to the global features and local features of the image;

[0009] Based on the hash code of the data to be retrieved, cross-modal retrieval is performed on the data set to be retrieved.

[0010] In a possible implementation, the overall objective function of the neural network model includes a global consistency loss function within the text modality, a local and global consistency loss function between modalities, and a classification adversarial loss function;

[0011] The global consistency loss function within the text modality is used to keep the features of texts of the same category consistent, and to keep the hash codes generated by texts of the same category consistent;

[0012] The inter-modal local and global consistency loss function is used to make the global hash code and local hash code of the image consistent with the hash code of the text corresponding to the image, and to make the global hash code and local hash code generated by images of the same category consistent;

[0013] The classification adversarial loss function includes the classification adversarial loss function of images and the classification adversarial loss function of texts. The classification adversarial loss function of images is used to make the global hash codes and local hash codes generated by images of the same category consistent. The classification adversarial loss function of texts is used to make the hash codes generated by texts of the same category consistent.

[0014] In one possible implementation, the overall objective function of the neural network model is:

[0015] J=J tg +J gl +J vc +J tc ;

[0016] in,

[0017]

[0018]

[0019] J tg represents the global consistency loss function within the text modality, J gl represents the local and global consistency loss function between modalities, J vc represents the classification adversarial loss function of the image, J tc Denotes the classification adversarial loss function of the text, H t1 , H t2 and H t Represents the text hash code of different texts, B t1 and B t2 They represent the K-bit binary matrices generated after inputting two text pairs, H vg and Hvl Represent the global hash code of the text corresponding to the image and the local hash code of the text corresponding to the image, B vg and B vl Represent the global binary code and local binary code of the image respectively, L vc Representatives use H vgl The result of classification, L represents the true category of the image, L tc Representatives use H t The classification result, H vgl Represents the global-local hash code of the image.

[0020] In one possible implementation, the overall objective function of the neural network model is:

[0021] J=w1J tg +w2J gl +w3J vc +w3J tc ;

[0022] in,

[0023]

[0024]

[0025] J tg represents the global consistency loss function within the text modality, J gl represents the local and global consistency loss function between modalities, J vc represents the classification adversarial loss function of the image, J tc Denotes the classification adversarial loss function of the text, H t1 , H t2 and H t Represents the text hash code of different texts, B t1 and B t2 They represent the K-bit binary matrices generated after inputting two text pairs, H vg and H vl Represent the global hash code of the text corresponding to the image and the local hash code of the text corresponding to the image, B vg and B vl Represent the global binary code and local binary code of the image respectively, L vc Representatives use H vgl The result of classification, L represents the true category of the image, L tc Representatives use H t The classification result, H vgl represents the global and local hash codes of the image, and w1, w2, and w3 represent the weights of the loss function respectively.

[0026] In a possible implementation, before inputting the data to be retrieved into the trained neural network model, the process further includes:

[0027] A cross-modal dataset is selected and divided into a training dataset and a test dataset; wherein the cross-modal dataset includes a plurality of text and image pairs and category information corresponding thereto;

[0028] The neural network model is trained based on the error back propagation algorithm, the overall objective function and the training data set. When the overall objective function converges and passes the test of the test data set, the trained neural network model is obtained.

[0029] In a possible implementation, the image network includes an image feature generation module and an image hash generation module, the image feature generation module is used to extract global features and local features of the image, and the image hash generation module is used to generate corresponding hash codes according to the global features and local features of the image;

[0030] The text network includes a text feature generation module, a text hash generation module and an input layer based on a bag-of-words model. The text feature generation module is used to extract the global features of the text, and the text hash generation module is used to generate corresponding hash codes according to the global features of the text.

[0031] In a possible implementation, the image feature generation module includes a first fully connected layer for generating global features of the image and a second fully connected layer for generating local features of the image, and the image hash generation module includes a third fully connected layer for generating a global feature hash code of the image, a fourth fully connected layer for generating a local feature hash code of the image, a fifth fully connected layer for generating a global local hash code of the image, and a sixth fully connected layer for classifying the same;

[0032] The text feature generation module includes a seventh fully connected layer for extracting global features of the text, and the text hash generation module includes an eighth fully connected layer for mapping the global features of the text into global feature hash codes of the text and a ninth fully connected layer for classifying the global feature hash codes of the text.

[0033] In a second aspect, an embodiment of the present invention provides a cross-modal retrieval device, including:

[0034] A data acquisition module is used to acquire the data to be retrieved, where the data to be retrieved is text or image;

[0035] A hash generation module, used for inputting the data to be retrieved into the trained neural network model to obtain the hash code of the data to be retrieved output by the neural network, wherein the neural network model includes a text network and an image network, the text network is used for extracting the global features of the text and generating the corresponding hash code according to the global features of the text, and the image network is used for extracting the global features and local features of the image and generating the corresponding hash code according to the global features and local features of the image;

[0036] The cross-modal retrieval module is used to perform cross-modal retrieval on the data to be retrieved in the data set to be retrieved based on the hash code of the data to be retrieved.

[0037] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements the steps of the method described in the first aspect or any possible implementation manner of the first aspect.

[0038] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect or any possible implementation method of the first aspect are implemented.

[0039] The embodiment of the present invention provides a cross-modal retrieval method, apparatus, device and storage medium, which first obtains the data to be retrieved, then inputs the data to be retrieved into a trained neural network model to obtain a hash code of the data to be retrieved output by the neural network. Finally, based on the hash code of the data to be retrieved, a cross-modal retrieval is performed on the data to be retrieved in the data set to be retrieved.

[0040] Since the trained neural network model is used in cross-modal retrieval, not only text features can be extracted, but also the global features and local features of the image can be extracted, and the corresponding hash codes can be generated according to the global features and local features of the image, so as to obtain better cross-modal hash codes, thereby improving the retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1A and Figure 1B is an example diagram provided by an embodiment of the present invention;

[0043] Figure 2 is a flow chart of an implementation of a cross-modal retrieval method provided by an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of a neural network structure provided by an embodiment of the present invention;

[0045] Figures 4A-4C is a PR curve diagram of a query from image to text using different algorithms provided by an embodiment of the present invention;

[0046] Figures 4D-4F It is a PR curve diagram of a query from text to image using different algorithms provided by an embodiment of the present invention;

[0047] Figure 5 is a schematic diagram of the structure of a cross-modal retrieval device provided by an embodiment of the present invention;

[0048] Figure 6 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0050] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0051] The image shown in Figure 1 often cannot satisfy the two assumptions described in the background technology in real data sets. Figure 1A The texts of the two images in the figure are similar, but the images are less similar from a visual perspective. Figure 1B Both images in the example belong to the boy category, but the contents of the two images can only partially match, which leads to reduced retrieval performance.

[0052] In order to solve the problems in the prior art, an embodiment of the present invention provides a cross-modal search method, a device and a storage medium. The cross-modal search method provided by an embodiment of the present invention is first introduced below.

[0053] The executor of the cross-modal retrieval method may be a cross-modal retrieval device, and the cross-modal retrieval device may be an electronic device having a processor and a memory, which is not specifically limited in the embodiment of the present invention.

[0054] See also Figure 2, which shows a flow chart of implementing the cross-modal retrieval method provided by an embodiment of the present invention, and is described in detail as follows:

[0055] Step S210: Obtain data to be retrieved.

[0056] The data to be retrieved can be text or image. For example, the image is used as the query data and the text is used as the retrieved data; or the text is used as the query data and the image is used as the retrieved data. There is no limitation here, and the user can select the type of data to be retrieved according to the actual retrieval requirements.

[0057] Step S220: input the data to be retrieved into the trained neural network model to obtain the hash code of the data to be retrieved output by the neural network.

[0058] The neural network model includes a text network and an image network. The text network is used to extract the global features of the text and generate the corresponding hash code according to the global features of the text. The image network is used to extract the global features and local features of the image and generate the corresponding hash code according to the global features and local features of the image.

[0059] The inventors found in the process of retrieval that, first, due to the uncertainty characteristics of image modal data, image data with similar texts are not necessarily similar. Second, images of the same category may not necessarily maintain global feature consistency, but may also maintain local consistency. However, text has high-level semantic information, that is, similar texts have similar semantic information. Therefore, the present invention uses text as the basis for guiding the generation of cross-modal hash codes, and utilizes the correspondence between the local information of the image, the global information of the image, and the global information of the text to generate a cross-modal hash code.

[0060] In some embodiments, in order to use text features as a guide, local features and global features of the image are embedded into the text feature space to facilitate obtaining better cross-modal hash codes, and the overall objective function of the neural network model is set.

[0061] Optionally, the overall objective function of the neural network model includes a global consistency loss function within the text modality, a local and global consistency loss function between modalities, and a classification adversarial loss function.

[0062] First, the global consistency loss function within the text modality is used to maintain consistency between the features of texts of the same category, and to maintain consistency between the hash codes generated for texts of the same category.

[0063] Specifically, in order to make two texts of similar categories have strong similarity after being mapped to the same hash space, the global consistency loss function within the text modality can be expressed as:

[0064]

[0065] Among them, H t1 ∈R K×r , H t2 ∈R K×r , B t1 =sign(H t1 ), B t2 =sign(H t2 ), r represents the number of input text pairs. t1 and B t2 Represents the K-bit binary code matrix generated after inputting the text pair, H t1 With H t2 Represents the text hash code generated after the text pair is input into the text network.

[0066] Among them, J pair (H t1 ,H t2 ) is to make the text features of the same category consistent. The formula is as follows:

[0067] Specifically, Negative log-likelihood of text hash code similarity.

[0068] We want to make the generated text hash code as similar as possible to the corresponding binary code.

[0069] Second, the inter-modal local and global consistency loss function is used to make the global hash code and local hash code of the image consistent with the hash code of the text corresponding to the image, and to make the global hash code and local hash code generated by images of the same category consistent.

[0070] Specifically, in cross-modal retrieval, the image pairs corresponding to two categories of similar texts often have different contents, which makes it impossible to accurately implement the intra-modal similarity measurement of image hash codes. However, under normal circumstances, there is a global or local correspondence between images and corresponding texts. The present invention adopts inter-modal local and global consistency loss functions, takes text as guiding information, and uses the characteristics that text content can correspond to local information or global information of images to improve the performance of cross-modal retrieval. The formula is:

[0071]

[0072] Among them, H t ∈R K×m Represents the hash code of the text, H vg ∈R K×m and H vl ∈R K×m Represent the global hash code of the text corresponding to the image and the local hash code of the text corresponding to the image. vg=sign(H vg ) and B vl =sign(H vl ) are the global and local binary codes of the image, and m is the number of input image-text pairs.

[0073] Among them, J pair (H t ,H vg ) and J pair (H t ,H vl ) indicates that it is desired to make both the global hash code and the local hash code of the image similar to the text hash code, that is, to use the text information as a guide to achieve simultaneous matching of the text hash code with the global hash code and the local hash code of the image. and It indicates that we want to make the generated global hash code of the image close to the corresponding binary code of the local hash code of the image.

[0074] Third, the classification adversarial loss function includes the classification adversarial loss function of images and the classification adversarial loss function of texts. The classification adversarial loss function of images is used to make the global hash codes and local hash codes generated by images of the same category consistent. The classification adversarial loss function of texts is used to make the hash codes generated by texts of the same category consistent.

[0075] Since images of the same category have differences between global and local features, it is difficult to construct a measure of global and local category consistency of an image. However, the global hash code H of an image vg and the local hash code H vl Having the same category information, the global-local hash code generated by the fusion of the two can better represent the image category. The global consistency loss function in the text modality can be expressed as:

[0076] J c =J vc +J tc ;

[0077] Among them, the classification adversarial loss function of the image

[0078] Text classification adversarial loss function

[0079] On the one hand, the present invention trains the image feature generation module G in an adversarial manner I and image hash generation module D I , for an image set v, through G I The global image feature F can be generated vg and local image features F vl , that is, {F vg , Fvl =G I (v) D I F vg and F vl Mapped to the global feature hash code H vg and the local feature hash code H vl Next, H vg and H vl Connect and map to the global-local hash code H through the fully connected layer vgl , that is, {H vg , H vl , H vgl =D I (F vg , F vl ).

[0080] The training process needs to be based on the following two principles: first, D I Hope H vgl It can be used to accurately judge the category, so it is necessary to minimize the image classification adversarial loss function. I Hope to find more that can not make H vgl To accurately classify an image, we need to maximize the image classification adversarial loss function. vc Representatives use H vgl The result of classification, L represents the true category of the image.

[0081] On the other hand, the present invention trains the text feature generation module G in an adversarial manner T and text hashing module D T Composition, for a text set t, G T Can generate text features F t , that is, {F t =G T (t), D t F t Mapped to text feature hash code H t , that is {H t =D T (F t ). Among them, L tc Representatives use H t The result of classification. Its adversarial method is the same as that of the image network.

[0082] Finally, the overall objective function of the neural network is as follows:

[0083] J=J tg +J gl +J vc +J tc ;

[0084] In some embodiments, in order to make the results of cross-modal retrieval more accurate, another overall objective function of the neural network is provided as follows:

[0085] J=w1J tg +w2J gl +w3J vc +w3J tc ;

[0086] By assigning different weights to the global consistency loss function within the text modality, the local and global consistency loss function between modalities, and the classification adversarial loss function, the hash code obtained by cross-modal retrieval can be made more accurate, making the retrieval results more similar to the queried data.

[0087] In some embodiments, a cross-modal dataset needs to be selected and divided into a training dataset and a test dataset to train the neural network, wherein the cross-modal dataset includes a plurality of text and image pairs and category information corresponding thereto.

[0088] The neural network model is trained based on the error back propagation algorithm, the overall objective function and the training data set. When the overall objective function converges and passes the test of the test data set, the trained neural network model is obtained.

[0089] By using the error back propagation algorithm and iterative optimization, the neural network is trained using the training data set until the overall objective function converges. The trained neural network is tested using the test data set. According to the test results, the similarity between the output results of the trained neural network and the real results can be seen.

[0090] The specific training process is as follows:

[0091]

[0092] in,

[0093] In the training phase, we first optimize J tg +J tc , used to update the text feature generation module G T Parameters and text hashing module D T Parameters Then, in the fixed and Under this premise, the text hash code is used as a guide to optimize J gl +J vc , used to update the image feature generation module G I Parameters and image hash generation module D I parameter It should be noted that They are all image feature generation modules G I , Image hash generation module D I , text feature generation module G T and text hashing module D T The parameters that need to be optimized will not be described here.

[0094] In this way, the local and global features of the image can be embedded into the text feature space under the guidance of text, and a better cross-modal hash code can be obtained.

[0095] In some embodiments, Figure 3 As shown, the image network includes an image feature generation module and an image hash generation module. The image feature generation module is used to extract the global features and local features of the image, and the image hash generation module is used to generate corresponding hash codes according to the global features and local features of the image.

[0096] Reference Figure 3 In terms of image network, for example, on the basis of the DCMH network structure, namely, 5 convolutional layers conv1-conv5 and 2 fully connected layers fc6-fc7, a fully connected layer for extracting local features of the image, a fully connected layer for local feature hash code and a fully connected layer for global local hash code are added. A fully connected layer fc7 for generating global features of the image and fully connected layers fc8 and fc9 for generating local features of the image, the image hash generation module includes a third fully connected layer fc11 for generating global feature hash codes of the image, a fourth fully connected layer fc10 for generating local feature hash codes of the image, a fifth fully connected layer fc12 for generating global local hash codes of the image and a sixth fully connected layer fc13 for classifying them.

[0097] In addition, this embodiment also uses a maximum activations of convolutions (MAC) method to extract convolution features. First, feature maps are extracted from the conv5 layer, and then the maximum response values ​​of different feature maps are connected as the convolution features of the image. This feature can better reflect the content of the object. The convolution features generate local features of the image after passing through two fully connected layers fc8-fc9, where the dimensions of the fc8 layer and the fc9 layer are 2635 and 512 respectively. Then the local feature hash code of the image is generated through the fc10 layer. The global feature hash code of the image is connected with the local feature hash code and input into the fully connected layer fc12 to obtain a global local hash code that fuses the global and local features of the image. The fc13 layer is used to classify the global-local hash code of the image.

[0098] The text network includes a text feature generation module, a text hash generation module and an input layer based on a bag-of-words model. The text feature generation module is used to extract the global features of the text, and the text hash generation module is used to generate corresponding hash codes according to the global features of the text.

[0099] Reference Figure 3 , exemplarily, the text feature generation module can use the bag of words model to represent the text as a vector as the network input, and then use two fully connected layers fc1 and fc2 to extract the global features of the text, where the dimensions of the fc1 layer and the fc2 layer are 8192 and 5138 respectively, and fc2 is the seventh fully connected layer. The fc3 layer is the eighth fully connected layer that maps the global features of the text to the global feature hash code of the text, and the fc4 layer is the ninth fully connected layer as a classification layer for classifying the global feature hash code of the text.

[0100] Among them, the activation functions of the third fully connected layer fc11, the fourth fully connected layer fc10, the fifth fully connected layer fc12 and the eighth fully connected layer fc3 are all tanh, the activation functions of the sixth fully connected layer fc13 and the ninth fully connected layer fc4 are all sigmoid, and the activation functions of the first fully connected layer fc7, the second fully connected layer fc9 and the seventh fully connected layer fc2 are all relu.

[0101] Step S230: Based on the hash code of the data to be retrieved, a cross-modal search is performed on the data to be retrieved in the data set to be retrieved.

[0102] According to the hash code of the data to be retrieved output by the neural network, the similarity between the data is calculated through the XOR operation between the hash codes, and the retrieval results are sorted according to the similarity to obtain the cross-modal retrieval results.

[0103] The cross-modal retrieval method provided by the present invention first obtains the data to be retrieved, and then inputs the data to be retrieved into a trained neural network model to obtain a hash code of the data to be retrieved output by the neural network. Finally, based on the hash code of the data to be retrieved, a cross-modal retrieval is performed on the data to be retrieved in the data set to be retrieved.

[0104] Since the trained neural network model is used in cross-modal retrieval, not only text features can be extracted, but also the global features and local features of the image can be extracted, and the corresponding hash codes can be generated according to the global features and local features of the image, so as to obtain better cross-modal hash codes, thereby improving the retrieval performance.

[0105] As mentioned in the background technology, traditional cross-modal hash retrieval methods are usually based on two assumptions. First, the image contents described by similar texts are also similar. Second, images of the same category have good global similarity. However, real data often cannot meet the above two assumptions, which leads to a decrease in the performance of the cross-modal hash retrieval model. The embodiment of the present invention proposes a cross-modal retrieval method based on text-guided adversarial hashing (Text-guided AdversarialHashing for Cross-Modal Retrieval, TAH). This method uses the text hash code as the basis for training the image network on the basis of the constructed neural network, and combines the local features of the image with the global features to represent the image content. In addition, global consistency loss within the text modality, local and global consistency loss between modalities, and classification adversarial loss are specifically proposed for training cross-modal networks. Experiments have shown that TAH can achieve good retrieval performance in three data sets.

[0106] Based on the constructed neural network, the text hash code is used as the basis for training the image network, and the local features of the image are combined with the global features to represent the image content. This method uses text features as a guide to embed the local features and global features of the image into the text feature space, which can obtain better cross-modal hash codes and thus improve the retrieval performance.

[0107] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0108] The following is an exemplary experiment to illustrate the performance of the cross-modal retrieval method provided by an embodiment of the present invention.

[0109] For the sake of ease of record, the cross-modal retrieval method provided in an embodiment of the present invention is referred to as the TAH method. Here, the retrieval performance of the TAH method is compared with that of the CCA method, SePH method, DCMH method, SSAH method, MDCH method and DSSAH method in three data sets: MIRFLICKR-25K, IAPR TC-12 and NUS-WIDE.

[0110] The MIRFLICKR-25K dataset contains 25,000 images collected from the Flicker website. Each image has a corresponding text description, and these image and text pairs are divided into 24 categories. In order to compare with other algorithms, we represent the text as a bag-of-words model BoW vector with a dimension of 1386 as the input of the text network.

[0111] IAPR TC-12 contains 2000 image-text pairs belonging to 255 categories, and each image is described in three different languages. This paper only uses the English description text. The BoW vector dimension of the text is 2912.

[0112] The NUS-WIDE dataset contains 269,648 image-text pairs, where the images are obtained from real-world photography. Each image-text pair can belong to one or more categories. In the experiment, we use 195,834 images containing 21 most common categories for training and testing. The text is represented as a 1000-dimensional BoW vector at the input stage.

[0113] The data in the above datasets are divided into three parts, namely the query set, the retrieval set and the training set. In the retrieval process, based on the image-text pairs contained in the query set, the data (retrieval set) in the image set other than the query set is retrieved. In addition, the algorithm extracts a part of the data from the retrieval set as the training set for training the cross-modal neural network. Among them, the number of query set samples in the MIRFLICKR-25K dataset and the IAPR TC-12 dataset is 2000, and the number of training set samples is 10000. The number of query set samples in the NUS-WIDE dataset is 2100, and the number of training set samples is 10500. During the training process, the batch size is set to 64. In addition, we use two commonly used performance metrics to measure the performance of the algorithm, namely Mean Average Precision (MAP) and Precision-Recall Curve (PR).

[0114] Table 1 compares the MAP values ​​of TAH with other excellent algorithms, which are the comparisons of MAP values ​​when the hash code length is 16 bits, 32 bits, and 64 bits. "I→T" represents that the query set is in image mode, while the retrieval set is in text mode. On the contrary, "T→I" represents that the query set is in text mode, while the retrieval set is in image mode. From the comparison results of MAP values, it can be found that TAH can obtain the best retrieval effect in all image libraries under different hash code lengths.

[0115] Unlike the other five methods that use deep learning to extract features, CCA and SePH use artificially designed features to represent images. Therefore, the retrieval performance of CCA and SePH is lower than that of other methods. DCMH can keep the semantic consistency of features within different modalities, but it does not take into account the problem that images of the same category may not have similar features. TAH fully considers this feature of images and uses text as a guide to generate better cross-modal hash codes. Therefore, TAH is about 6% higher than the MAP of DCMH. SSAH, based on DCMH, uses label information and designed adversarial networks to obtain better retrieval performance. However, SSAH still does not take into account the differences between images of the same category, so its performance is still lower than TAH. Attention mechanism is the most common method for discovering important features in different modalities. MDCH uses attention mechanism, inter-modal loss and intra-modal loss to train cross-modal networks, but compared with TAH, it does not consider the local information of different modal data, so its retrieval performance is lower than TAH. Different from the DCMH, SSAH and MDCH methods, the DSSAH method takes into account the differences between images of the same category and uses the adversarial mechanism and semantic consistency between modalities to generate the best MAP value except TAH.

[0116] Table 1

[0117]

[0118]

[0119] As can be seen from the table, the retrieval performance of TAH in the three datasets is better than that of six excellent cross-modal hashing retrieval methods.

[0120] In order to verify the effectiveness of TAH, we replaced the CNN-F network used to extract image features in TAH with Vgg16 and Vgg19. The results are shown in Table 2. By comparison, it can be found that TAH_Vgg19 can achieve the best retrieval effect, which is about 0.3% higher than the MAP value of TAH, while TAH_Vgg16 has the lowest MAP. The experimental results show that under the network framework of TAH, using different network structures will affect the retrieval results.

[0121] Table 2

[0122]

[0123] In addition, the effects of the combination of three different losses on the MAP value of the TAH method are also analyzed. It should be noted that the original intention of TAH is to use text features as a guide for training cross-modal networks. Therefore, in all cases listed in Table 3, J loss is used. tgIt should be noted that J in Table 2 tg +J gl +J c This is the TAH method proposed in the text. By comparison, it can be found that when the loss function used is J tg +J c or J tg +J gl The MAP values ​​are lower than J tg +J gl +J c , indicating J tg , J gl and J c All of them play a positive role in training cross-modal networks. In addition, the loss function is J tg +J gl The MAP value is higher than J tg +J c This indicates that during the training process of TAH, J gl The effect is greater than J c Finally, we try to improve the MAP value by weighting the loss, that is, setting the loss to w1J tg +w2J gl +w3J c , where w1, w2 and w3 represent the weights of the three losses respectively. The optimal MAP value can be obtained when w1 = 0.7, w2 = 0.2, and w3 = 0.1.

[0124] Table 3

[0125]

[0126] also, 4A to 4F The PR curves of TAH and the above six different algorithms on the MIRFLICKR-25K dataset, IAPR TC-12 dataset and NUS-WIDE dataset were also compared. FIG. 4A to FIG. 4C For image-to-text queries, FIG. 4D to FIG. 4F is a text-to-image query. It can be found that as the recall rate increases, the accuracy of all algorithms decreases. In addition, TAH can achieve the best retrieval performance under different recall rates.

[0127] The embodiment of the present invention proposes a TAH method for generating more reasonable cross-modal hash codes. Different from traditional cross-modal hashing methods, TAH maps image features to the feature space of text, and combines the local features and global features of the image with the adversarial mechanism to ensure the similarity of the images. Experiments have shown that the retrieval performance of TAH in three data sets is better than that of six excellent cross-modal hashing retrieval methods.

[0128] Based on the cross-modal retrieval method provided in the above embodiment, the present invention also provides a specific implementation of a cross-modal retrieval device applied to the cross-modal retrieval method. Please refer to the following embodiment.

[0129] like Figure 5 As shown, a cross-modal retrieval device 500 is provided, and the device 500 includes:

[0130] The data acquisition module 510 is used to acquire the data to be retrieved, where the data to be retrieved is text or image;

[0131] A hash generation module 520, which is used to input the data to be retrieved into the trained neural network model to obtain the hash code of the data to be retrieved output by the neural network, wherein the neural network model includes a text network and an image network, the text network is used to extract the global features of the text and generate the corresponding hash code according to the global features of the text, and the image network is used to extract the global features and local features of the image and generate the corresponding hash code according to the global features and local features of the image;

[0132] The cross-modal retrieval module 530 is used to perform a cross-modal retrieval on the data to be retrieved in the data set to be retrieved based on the hash code of the data to be retrieved.

[0133] In a possible implementation, the overall objective function of the neural network model includes a global consistency loss function within the text modality, a local and global consistency loss function between modalities, and a classification adversarial loss function;

[0134] The global consistency loss function within the text modality is used to keep the features of texts of the same category consistent, and to keep the hash codes generated by texts of the same category consistent;

[0135] The inter-modal local and global consistency loss function is used to make the global hash code and local hash code of the image consistent with the hash code of the text corresponding to the image, and to make the global hash code and local hash code generated by images of the same category consistent;

[0136] The classification adversarial loss function includes the classification adversarial loss function of images and the classification adversarial loss function of texts. The classification adversarial loss function of images is used to make the global hash codes and local hash codes generated by images of the same category consistent. The classification adversarial loss function of texts is used to make the hash codes generated by texts of the same category consistent.

[0137] In one possible implementation, the overall objective function of the neural network model is:

[0138] J=J tg +J gl +Jvc +J tc ;

[0139] in,

[0140]

[0141]

[0142] J tg represents the global consistency loss function within the text modality, J gl represents the local and global consistency loss function between modalities, J vc represents the classification adversarial loss function of the image, J tc Denotes the classification adversarial loss function of the text, H t1 , H t2 and H t Represents the text hash code of different texts, B t1 and B t2 They represent the K-bit binary matrices generated after inputting two text pairs, H vg and H vl Represent the global hash code of the text corresponding to the image and the local hash code of the text corresponding to the image, B vg and B vl Represent the global binary code and local binary code of the image respectively, L vc Representatives use H vgl The result of classification, L represents the true category of the image, L tc Representatives use H t The classification result, H vgl Represents the global-local hash code of the image.

[0143] In one possible implementation, the overall objective function of the neural network model is:

[0144] J=w1J tg +w2J gl +w3J vc +w3J tc ;

[0145] in,

[0146]

[0147]

[0148] J tg represents the global consistency loss function within the text modality, J gl represents the local and global consistency loss function between modalities, Jvc represents the classification adversarial loss function of the image, J tc Denotes the classification adversarial loss function of the text, H t1 , H t2 and H t Represents the text hash code of different texts, B t1 and B t2 They represent the K-bit binary matrices generated after inputting two text pairs, H vg and H vl Represent the global hash code of the text corresponding to the image and the local hash code of the text corresponding to the image, B vg and B vl Represent the global binary code and local binary code of the image respectively, L vc Representatives use H vgl The result of classification, L represents the true category of the image, L tc Representatives use H t The classification result, H vgl represents the global and local hash codes of the image, and w1, w2, and w3 represent the weights of the loss function respectively.

[0149] In a possible implementation, the hash generation module 520 is further configured to:

[0150] A cross-modal dataset is selected and divided into a training dataset and a test dataset; wherein the cross-modal dataset includes a plurality of text and image pairs and category information corresponding thereto;

[0151] The neural network model is trained based on the error back propagation algorithm, the overall objective function and the training data set. When the overall objective function converges and passes the test of the test data set, the trained neural network model is obtained.

[0152] In a possible implementation, the image network includes an image feature generation module and an image hash generation module, the image feature generation module is used to extract global features and local features of the image, and the image hash generation module is used to generate corresponding hash codes according to the global features and local features of the image;

[0153] The text network includes a text feature generation module, a text hash generation module and an input layer based on a bag-of-words model. The text feature generation module is used to extract the global features of the text, and the text hash generation module is used to generate corresponding hash codes according to the global features of the text.

[0154] In a possible implementation, the image feature generation module includes a first fully connected layer for generating global features of the image and a second fully connected layer for generating local features of the image, and the image hash generation module includes a third fully connected layer for generating a global feature hash code of the image, a fourth fully connected layer for generating a local feature hash code of the image, a fifth fully connected layer for generating a global local hash code of the image, and a sixth fully connected layer for classifying the same;

[0155] The text feature generation module includes a seventh fully connected layer for extracting global features of the text, and the text hash generation module includes an eighth fully connected layer for mapping the global features of the text into global feature hash codes of the text and a ninth fully connected layer for classifying the global feature hash codes of the text.

[0156] Figure 6 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 6 As shown, the electronic device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. When the processor 60 executes the computer program 62, the steps in the above-mentioned transformer riser and bushing monitoring method embodiments are implemented, for example Figure 2 Alternatively, when the processor 60 executes the computer program 62, the functions of the modules in the above-mentioned device embodiments are realized, such as the functions of the modules 410 to 430 shown in FIG. 4 .

[0157] Exemplarily, the computer program 62 may be divided into one or more modules, which are stored in the memory 61 and executed by the processor 60 to implement the present invention. The one or more modules may be a series of computer program instruction segments capable of implementing specific functions, which are used to describe the execution process of the computer program 62 in the electronic device 6. For example, the computer program 62 may be divided into Figure 5 Modules 510 to 530 are shown.

[0158] The electronic device 6 may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will appreciate that Figure 6 It is only an example of the electronic device 6 and does not constitute a limitation of the electronic device 6. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0159] The processor 60 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0160] The memory 61 may be an internal storage unit of the electronic device 6, such as a hard disk or memory of the electronic device 6. The memory 61 may also be an external storage device of the electronic device 6, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 6. Further, the memory 61 may also include both an internal storage unit and an external storage device of the electronic device 6. The memory 61 is used to store the computer program and other programs and data required by the electronic device. The memory 61 may also be used to temporarily store data that has been output or is to be output.

[0161] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0162] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0163] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0164] In the embodiments provided by the present invention, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0165] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0167] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned cross-modal retrieval method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0168] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A cross-modal retrieval method, characterized in that: include: Acquire data to be retrieved, wherein the data to be retrieved is text or image; Inputting the data to be retrieved into a trained neural network model to obtain a hash code of the data to be retrieved output by the neural network, wherein the neural network model includes a text network and an image network, the text network is used to extract global features of the text and generate corresponding hash codes according to the global features of the text, and the image network is used to extract global features and local features of the image and generate corresponding hash codes according to the global features and local features of the image; Based on the hash code of the data to be retrieved, performing a cross-modal search on the data to be retrieved in the data set to be retrieved; Among them, the overall objective function of the neural network model includes a global consistency loss function within the text modality, a local and global consistency loss function between modalities, and a classification adversarial loss function; the global consistency loss function within the text modality is used to maintain consistency between the features of texts of the same category, and to maintain consistency between the hash codes generated by texts of the same category; the local and global consistency loss function between modalities is used to maintain consistency between the global hash code and the local hash code of the image and the hash code of the text corresponding to the image, and to maintain consistency between the global hash code and the local hash code generated by images of the same category; the classification adversarial loss function includes a classification adversarial loss function of the image and a classification adversarial loss function of the text, the classification adversarial loss function of the image is used to maintain consistency between the global hash code and the local hash code generated by images of the same category, and the classification adversarial loss function of the text is used to maintain consistency between the hash codes generated by texts of the same category; The overall objective function of the neural network model is: J=J tg +J gl +J vc +J tc ; or J = w1J tg +w2J gl +w3J vc +w3J tc ; in, J tg represents the global consistency loss function within the text modality, J gl represents the local and global consistency loss function between modalities, J vc represents the classification adversarial loss function of the image, J tc Denotes the classification adversarial loss function of the text, H t1 , H t2 and H t Represents the text hash code of different texts, B t1 and B t2 They represent the K-bit binary matrices generated after inputting two text pairs, H vg and H vl Represent the global hash code of the text corresponding to the image and the local hash code of the text corresponding to the image, B vg and B vl Represent the global binary code and local binary code of the image respectively, L vc Representatives use H vgl The result of classification, L represents the true category of the image, L tc Representatives use H t The classification result, H vgl represents the global and local hash codes of the image, and w1, w2, and w3 represent the weights of the loss function respectively.

2. A cross-modal retrieval method as claimed in claim 1, characterized in that: Before inputting the data to be retrieved into the trained neural network model, the method further includes: Selecting a cross-modal dataset and dividing it into a training dataset and a test dataset; wherein the cross-modal dataset includes a plurality of text and image pairs and corresponding category information; The neural network model is trained based on the error back propagation algorithm, the overall objective function of the neural network model and the training data set. When the overall objective function converges and passes the test of the test data set, a trained neural network model is obtained.

3. A cross-modal retrieval method as claimed in claim 1, characterized in that: The image network includes an image feature generation module and an image hash generation module, wherein the image feature generation module is used to extract global features and local features of an image, and the image hash generation module is used to generate corresponding hash codes according to the global features and local features of the image; The text network includes a text feature generation module, a text hash generation module and an input layer based on a bag-of-words model. The text feature generation module is used to extract global features of the text, and the text hash generation module is used to generate corresponding hash codes according to the global features of the text.

4. A cross-modal retrieval method as claimed in claim 3, characterized in that: The image feature generation module includes a first fully connected layer for generating global features of an image and a second fully connected layer for generating local features of an image, and the image hash generation module includes a third fully connected layer for generating a global feature hash code of an image, a fourth fully connected layer for generating a local feature hash code of an image, a fifth fully connected layer for generating a global local hash code of an image, and a sixth fully connected layer for classifying the same; The text feature generation module includes a seventh fully connected layer for extracting global features of the text, and the text hash generation module includes an eighth fully connected layer for mapping the global features of the text into global feature hash codes of the text and a ninth fully connected layer for classifying the global feature hash codes of the text.

5. A cross-modal retrieval device, characterized in that: include: A data acquisition module is used to acquire data to be retrieved, wherein the data to be retrieved is text or image; A hash generation module, used for inputting the data to be retrieved into a trained neural network model to obtain a hash code of the data to be retrieved output by the neural network, wherein the neural network model includes a text network and an image network, the text network is used for extracting global features of the text and generating corresponding hash codes according to the global features of the text, and the image network is used for extracting global features and local features of the image and generating corresponding hash codes according to the global features and local features of the image; A cross-modal retrieval module, configured to perform a cross-modal retrieval on the data to be retrieved in the data set to be retrieved based on a hash code of the data to be retrieved; Among them, the overall objective function of the neural network model includes a global consistency loss function within the text modality, a local and global consistency loss function between modalities, and a classification adversarial loss function; the global consistency loss function within the text modality is used to maintain consistency between the features of texts of the same category, and to maintain consistency between the hash codes generated by texts of the same category; the local and global consistency loss function between modalities is used to maintain consistency between the global hash code and the local hash code of the image and the hash code of the text corresponding to the image, and to maintain consistency between the global hash code and the local hash code generated by images of the same category; the classification adversarial loss function includes a classification adversarial loss function of the image and a classification adversarial loss function of the text, the classification adversarial loss function of the image is used to maintain consistency between the global hash code and the local hash code generated by images of the same category, and the classification adversarial loss function of the text is used to maintain consistency between the hash codes generated by texts of the same category; The overall objective function of the neural network model is: J=J tg +J gl +J vc +J tc ; or J = w1J tg +w2J gl +w3J vc +w3J tc ; in, J tg represents the global consistency loss function within the text modality, J gl represents the local and global consistency loss function between modalities, J vc represents the classification adversarial loss function of the image, J tc Denotes the classification adversarial loss function of the text, H t1 , H t2 and H t Represents the text hash code of different texts, B t1 and B t2 They represent the K-bit binary matrices generated after inputting two text pairs, H vg and H vl Represent the global hash code of the text corresponding to the image and the local hash code of the text corresponding to the image, B vg and B vl Represent the global binary code and local binary code of the image respectively, L vc Representatives use H vgl The result of classification, L represents the true category of the image, L tc Representatives use H t The classification result, H vgl represents the global and local hash codes of the image, and w1, w2, and w3 represent the weights of the loss function respectively.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images