Cross-modal hash retrieval method based on mixed correlation learning

Abstract

The invention provides a cross-modal hash retrieval method based on mixed correlation learning. The method comprises the following steps: acquiring to-be-retrieved data; inputting the to-be-retrieved data into the trained retrieval model to obtain a hash code which is output by the retrieval model and corresponds to the to-be-retrieved data; according to the hash code, determining a retrieval result corresponding to the to-be-retrieved data from a preset database; wherein the retrieval model carries out supervised learning which takes label features as semantic guidance and unsupervised learning which focuses on sample features, so that sample features of different view angles can be obtained, and after the sample features of the two view angles are fused, the features are mapped into hash codes so as to accelerate the retrieval speed. Through the steps, the retrieval precision of cross-modal hash retrieval can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data retrieval, and in particular to a cross-modal hash retrieval method based on hybrid correlation learning. Background Art

[0002] With the rapid development of the Internet, a large amount of multimedia data in different modalities is generated every day, such as images, texts, videos, and audios. This makes cross-modal hash retrieval widely used in various search engines.

[0003] Cross-modal hash retrieval refers to using data from one modality as a query item, using hash codes to search a database consisting of data from another modality, and returning similar data. For example, when a user enters text data in a search engine, the search engine returns image data similar to the text data through cross-modal hash retrieval for the user to use.

[0004] However, the cross-modal hash retrieval method in the related art generally has the problem of low retrieval accuracy, resulting in poor similarity between the data returned by the search engine and the data entered by the user. Summary of the invention

[0005] An embodiment of the present invention provides a cross-modal hash retrieval method based on hybrid correlation learning to improve the retrieval accuracy of cross-modal hash retrieval.

[0006] In a first aspect, an embodiment of the present invention provides a cross-modal hash retrieval method based on hybrid correlation learning, comprising:

[0007] Get the data to be retrieved;

[0008] Inputting the data to be retrieved into a trained retrieval model to obtain a hash code output by the retrieval model corresponding to the data to be retrieved;

[0009] According to the hash code, determining a search result corresponding to the data to be searched from a preset database;

[0010] The retrieval model is obtained by respectively conducting supervised learning and unsupervised learning training based on sample data of different modalities and word embedding vectors of different labels.

[0011] In a second aspect, an embodiment of the present invention provides a cross-modal hash retrieval device based on hybrid correlation learning, comprising:

[0012] An acquisition module is used to acquire data to be retrieved;

[0013] A processing module, used for inputting the data to be retrieved into a trained retrieval model to obtain a hash code output by the retrieval model corresponding to the data to be retrieved;

[0014] A retrieval module, used to determine a retrieval result corresponding to the data to be retrieved from a preset database according to the hash code;

[0015] The retrieval model is obtained by respectively conducting supervised learning and unsupervised learning training based on sample data of different modalities and word embedding vectors of different labels.

[0016] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the processor implements the steps of the method described in the first aspect or any possible implementation manner of the first aspect.

[0017] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the first aspect or any possible implementation method of the first aspect are implemented.

[0018] The embodiment of the present invention provides a cross-modal hash retrieval method and electronic device based on hybrid correlation learning, by inputting the data to be retrieved into a trained retrieval model, obtaining a hash code corresponding to the data to be retrieved, and determining the retrieval result corresponding to the data to be retrieved from a preset database according to the hash code. The retrieval model is obtained by supervised learning and unsupervised learning training according to sample data of different modalities and word embedding vectors of different labels. The embodiment of the present invention takes into account that supervised learning can discover the semantic relationship between samples, and unsupervised learning is conducive to discovering the inherent laws of data. Therefore, the retrieval model is controlled to perform supervised learning and unsupervised learning respectively, and the semantic relationship and inherent laws of sample data of different modalities can be learned at the same time, so that the correlation between the hash code output by the retrieval model and the input data of the retrieval model is higher, and finally the correlation between the data to be retrieved and the retrieval results is improved, and the effect of improving the accuracy of cross-modal hash retrieval is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0020] Figure 1 is a flow chart for implementing a cross-modal hash retrieval method based on hybrid correlation learning provided by an embodiment of the present invention;

[0021] Figure 2is a flowchart of implementing training of a retrieval model provided by an embodiment of the present invention;

[0022] Figure 3 It is a schematic diagram of data flow in the retrieval model training process provided by an embodiment of the present invention;

[0023] Figure 4 It is a flow chart of implementing supervised learning and unsupervised learning by a teacher network provided by an embodiment of the present invention;

[0024] Figure 5 It is a training flow chart of a retrieval model when performing cross-modal hash retrieval of text images provided by an embodiment of the present invention;

[0025] Figure 6 It is a structural schematic diagram of a cross-modal hash retrieval device based on hybrid correlation learning provided by an embodiment of the present invention;

[0026] Figure 7 is a schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0028] Modality can be understood as the form in which data exists. For example, images, text, video, and audio all belong to different modalities. Cross-modal hash retrieval refers to using data of one modality as a query item, in the form of a hash code, to search on a database consisting of data of another modality, and return similar data. Cross-modal hash retrieval has been widely used in various search engines. For example, a user can enter text data in a search engine, and the search engine returns image data similar to the text data after cross-modal hash retrieval. However, the cross-modal hash retrieval method in the related art usually adopts a single supervised method or an unsupervised method, which cannot accurately measure the correlation between sample data, resulting in low retrieval accuracy. For example, a user enters the text "an eagle is catching rabbits on the grassland" in a search engine to retrieve related images, but the search engine may not accurately return the image corresponding to the above text, but returns other images, which cannot meet the user's needs. In other words, the correlation between the search results retrieved by the search engine and the data to be retrieved entered by the user is poor, and the retrieval accuracy is low.

[0029] In order to improve the correlation between the retrieval results and the data to be retrieved, and to improve the accuracy of cross-modal hash retrieval, in the implementation of the present application, the retrieval model is used to determine the hash code corresponding to the data to be retrieved, and the retrieval result is determined according to the hash code, thereby completing the cross-modal hash retrieval. Among them, the retrieval model can be obtained by supervised learning and unsupervised learning training according to sample data of different modalities and word embedding vectors of different labels. Here, in the model training process, the retrieval model can be supervised and unsupervised according to sample data of different modalities and word embedding vectors of different labels, wherein supervised learning is conducive to discovering the semantic relationship between samples, and unsupervised learning is conducive to discovering the inherent laws of data. By making full use of the characteristics of the two learning methods, the retrieval model can simultaneously learn the semantic relationship and inherent laws between sample data of different modalities, thereby improving the retrieval accuracy of the retrieval model, and then improving the correlation between the data to be retrieved and the retrieval results.

[0030] Taking the example of a user searching for an image through text, the user inputs the text data to be retrieved, such as "an eagle is catching rabbits on the grassland" into the search engine. The processor corresponding to the search engine inputs the text data to be retrieved "an eagle is catching rabbits on the grassland" into the retrieval model, and the retrieval model outputs a hash code corresponding to the above text data to be retrieved. The processor determines one or more images corresponding to the hash code from a preset image database based on the hash code, thereby completing cross-modal hash retrieval.

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clear, specific embodiments will be described below in conjunction with the accompanying drawings.

[0032] Figure 1 The implementation flow chart of the cross-modal hash retrieval method for hybrid correlation learning provided by the embodiment of the present invention is described in detail as follows:

[0033] Step 101, obtaining data to be retrieved.

[0034] The data to be retrieved here can be data of different modalities, such as images, texts, videos, and audios.

[0035] Step 102: input the data to be retrieved into the trained retrieval model to obtain a hash code corresponding to the data to be retrieved output by the retrieval model.

[0036] Among them, the retrieval model is obtained by supervised learning and unsupervised learning training respectively according to sample data of different modalities and word embedding vectors of different labels. Here, sample data of different modalities may include a cross-modal data group of multiple samples. Among them, the cross-modal data group of each sample includes data of different modalities corresponding to the sample. The embodiment of the present invention takes into account that cross-modal hash retrieval usually involves mutual retrieval between two different modal data. Accordingly, the cross-modal data group used to train the retrieval model may include data of two different modalities corresponding to the same sample.

[0037] Taking text image cross-modal hash retrieval as an example, each cross-modal data set can be an image-text pair. It contains image data and text data corresponding to the same sample. Here, the MIRFLICKR-25K dataset and NUS-WIDE dataset can be used to obtain image-text pairs corresponding to different samples. The image-text pairs corresponding to different samples can be used Indicates. Where O represents the image-text pairs corresponding to different samples, N represents the number of samples, and o i represents the i-th sample, represents the image data corresponding to the i-th sample, Represents the text data corresponding to the i-th sample.

[0038] In addition, to facilitate subsequent supervised learning, the embodiment of the present invention also sets labels for each sample. It should be noted that, for each sample, the label corresponding to the sample is the label of the data of different modalities corresponding to the sample. Exemplarily, when "an eagle is catching rabbits on the grassland" is used as a sample, the cross-modal data group corresponding to the sample may include: text data, image data, and video data corresponding to "an eagle is catching rabbits on the grassland". The label corresponding to the sample can be "eagle, grassland, rabbit". Correspondingly, the labels corresponding to the above text data, image data, and video data are also "eagle, grassland, rabbit".

[0039] In the embodiment of the present invention, a label vector may be used to represent the label of a sample. i It can represent the label vector corresponding to the i-th sample, that is, the label corresponding to the i-th sample. Here, y i ={y i1 ,y i2 ,...,y ic} Among them, y i1 ,y i2 ,y ic Both represent the label identifier of the sample. When the i-th sample belongs to the first label, y i1 = 1, when the i-th sample does not belong to the first label, yi1 = 0. Similarly, when the i-th sample belongs to the second label, y i2 =1, when the i-th sample does not belong to the second label, y i2 =0; when the i-th sample belongs to the c-th label, y ic = 1, when the i-th sample does not belong to the c-th label, y ic =0, c represents the total number of tags.

[0040] For example, assuming that all labels include: eagle, river, grassland, rabbit and dog in sequence, the label vector corresponding to the sample "an eagle is catching rabbits on the grassland" can be expressed as {1, 0, 1, 1, 0}.

[0041] The present invention can obtain word embedding vectors of different labels while obtaining sample data of different modalities. The word embedding vector can map words into vector space, thereby realizing vectorized representation of words. By obtaining word embedding vectors of different labels, the vectorized representations corresponding to different labels can be determined, so that subsequent supervised learning can be performed according to the vectorized representations corresponding to different labels.

[0042] The embodiment of the present invention can train the retrieval model based on the acquisition of sample data of different modalities and word embedding vectors of different labels. During the training process of the retrieval model, the retrieval model can perform supervised learning and unsupervised learning respectively according to the sample data of different modalities and the word embedding vectors of different labels. Among them, supervised learning can discover the semantic relationship between samples, and unsupervised learning is conducive to discovering the inherent laws of data. The retrieval model performs supervised learning and unsupervised learning respectively, and can simultaneously learn the semantic relationship and inherent laws of sample data of different modalities, so that the correlation between the hash code output by the retrieval model and the input data of the retrieval model is higher.

[0043] Step 103: Determine the search result corresponding to the data to be searched from a preset database according to the hash code.

[0044] The database contains different search results and corresponding hash codes. Among them, the modality of the search results in the database can be determined according to the actual situation. Exemplarily, if a text-image cross-modal search is performed, and the data to be searched is text data, the modality of the search results in the preset database is image. If a text-image cross-modal search is performed, and the data to be searched is image data, the modality of the search results in the preset database is text.

[0045] According to the hash code corresponding to the data to be retrieved, a search result corresponding to the hash code can be determined from a preset database, and the search result can be determined as the search result corresponding to the data to be retrieved.

[0046] The embodiment of the present invention inputs the data to be retrieved into a trained retrieval model to obtain a hash code corresponding to the data to be retrieved; according to the hash code, the retrieval result corresponding to the data to be retrieved is determined from a preset database. Among them, the retrieval model is obtained by performing supervised learning and unsupervised learning training according to sample data of different modalities and word embedding vectors of different labels. Considering that supervised learning can discover the semantic relationship between samples and unsupervised learning is conducive to discovering the inherent laws of data, the retrieval model performs supervised learning and unsupervised learning respectively, and can simultaneously learn the semantic relationship and inherent laws of sample data of different modalities, so that the correlation between the hash code output by the retrieval model and the input data of the retrieval model is higher, and finally the correlation between the data to be retrieved and the retrieval results is improved, and the effect of improving the accuracy of cross-modal hash retrieval is achieved.

[0047] The following is an introduction to the training process of the retrieval model.

[0048] In some embodiments, see Figure 2 When training the retrieval model, the following steps 201-203 may be performed:

[0049] Step 201, respectively obtain sample data of different modalities and word embedding vectors of different labels.

[0050] Step 202, based on the sample data of different modalities and the word embedding vectors of different labels, control the retrieval model to perform supervised learning and unsupervised learning respectively to determine the supervised sample features and unsupervised sample features corresponding to the sample data of different modalities, and determine the common representation and hash code corresponding to the sample data of different modalities based on the supervised sample features and unsupervised sample features.

[0051] Here, the sample data of different modalities may include a cross-modal data group of multiple samples, wherein each cross-modal data group of a sample includes data of different modalities corresponding to the sample.

[0052] Here, the retrieval model can use a cross-modal data set of multiple samples and word embedding vectors of different labels to perform supervised learning to determine the supervised sample features of different modalities corresponding to each sample. In addition, the cross-modal data set of multiple samples can also be used to perform unsupervised learning to determine the unsupervised sample features of different modalities corresponding to each sample.

[0053] The embodiment of the present invention can determine the common representation of different modalities corresponding to each sample by merging the supervised sample features and unsupervised sample features of different modalities corresponding to each sample and mapping them to a common representation space, and then the hash codes of different modalities can be determined correspondingly through the common representation of different modalities.

[0054] In some embodiments, see Figure 3,The retrieval model includes: teacher network, student network and hash conversion network.

[0055] In an embodiment of the present invention, sample data of different modalities and word embedding vectors of different labels can be first input into a teacher network to obtain a teacher public representation corresponding to the sample data of different modalities output by the teacher network, and then the teacher public representation is input into a hash conversion network to obtain a teacher hash code; in addition, sample data of different modalities are also input into a student network to obtain a student public representation corresponding to the sample data of different modalities output by the student network, and then the student public representation is input into a hash conversion network to obtain a student hash code.

[0056] Here, the teacher network can perform supervised learning and unsupervised learning respectively according to the input sample data of different modalities and word embedding vectors of different labels to determine the supervised sample features and unsupervised sample features corresponding to the sample data of different modalities; and determine the teacher common representation corresponding to the sample data of different modalities based on the supervised sample features and unsupervised sample features.

[0057] The hash conversion network performs hash conversion based on the public representation of the input and obtains the hash code corresponding to the public representation. The conversion process of the teacher hash code can be expressed as:

[0058] The conversion process of student hash code can be expressed as:

[0059] Among them, B t represents the teacher hash code of different modalities, H(·) represents the mapping of the hash layer, and F tea represents the common representation of teachers of different modalities, F stu Represents the common representation of students of different modalities, represents a real number matrix with 2N rows and k columns, where k represents the length of the hash code.

[0060] Taking text-image cross-modal hash retrieval as an example, the teacher hash codes of different modalities can include: the teacher hash code under the text modality, and the teacher hash code under the image modality, that is, Among them, B tv represents the teacher hash code in image mode, B tt Represents the teacher hash code in text mode.

[0061] Similarly, student hash codes of different modalities may include: student hash codes in text modality, student hash codes in image modality, that is, Among them, B sv represents the student hash code in image mode, B st Represents the student hash code in text mode.

[0062] The teacher public representations in different modes may include: the teacher public representation in text mode, the teacher public representation in image mode, that is, Among them, V tea represents the teacher’s common representation in the image modality, T tea Represents the teacher's public representation in text mode.

[0063] The student common representations of different modalities may include: the student common representations in the text modality and the student common representations in the image modality, that is, Among them, V stu represents the student common representation in image mode, T stu Represents the public representation of students in text mode.

[0064] In addition, the embodiments of the present invention take into account that the sign(·) activation function is usually used in traditional hash conversion networks, but the sign(·) activation function cannot perform gradient direction propagation. In the embodiments of the present invention, during the training stage of the retrieval model, tanh(·) is used as the activation function of the hash layer to perform model training, and after the training is completed, the sign(·) activation function is reused to generate discrete hash codes.

[0065] In the embodiment of the present invention, the student network can perform feature extraction and mapping processing based on the input sample data of different modalities to obtain the student common representation corresponding to the sample data of different modalities.

[0066] The student network includes multiple feature extraction models and a mapping model connected to the output ends of the multiple feature extraction models. The number of feature extraction models corresponds to the number of modalities. Each feature extraction model extracts features from the data of the corresponding modality to obtain sample features of the modality. The mapping model connected to the multiple feature extraction models includes two fully connected layers connected in sequence, which are used to map sample features of different modalities to a common representation space, and generate student common representations of different modalities.

[0067] Taking text-image cross-modal hash retrieval as an example, the student network can include the VGG-19 model and the bag-of-words model. The VGG-19 model is used to extract features from the sample's image data to obtain image features. The bag-of-words model is used to extract features from the sample's text data to obtain text features. The image features and text features are mapped to the common representation space through two fully connected layers, respectively, to generate the student common representation in the image modality and the student common representation in the text modality.

[0068] Step 203, determine the loss function corresponding to the retrieval model according to the public representation and the hash code, and when the value of the loss function is greater than or equal to the preset value, adjust the model parameters of the retrieval model until the value of the loss function is less than the preset value, thereby obtaining a trained retrieval model.

[0069] It should be noted that the teacher network and the student network can be simply understood as different feature extraction networks. Compared with the student network, the feature extraction process of the teacher network is more sophisticated. Accordingly, the teacher public representation output by it can more accurately reflect the semantics and internal laws of the sample data, and the teacher hash code determined by the teacher public representation will be more closely related to the sample data.

[0070] The feature extraction process of the teacher network is more refined, and accordingly, the feature extraction efficiency of the teacher network will be reduced accordingly. Compared with the teacher network, the feature extraction efficiency of the student network is higher, but the feature extraction accuracy will be reduced accordingly.

[0071] The embodiment of the present invention determines the value of the loss function through the teacher's public representation, the student's public representation, the teacher's hash code and the student's hash code in different modes, and can map the difference between the teacher's public representation and the student's public representation, as well as the difference between the teacher's hash code and the student's hash code, to the value of the loss function. According to the value of the loss function, the network parameters of the teacher network, the student network and the hash conversion network are adjusted until the value of the loss function is lower than the preset value, that is, the difference between the teacher's public representation and the student's public representation, as well as the difference between the teacher's hash code and the student's hash code, are within the error range, and a trained retrieval model is obtained.

[0072] In essence, the training process of the retrieval model is equivalent to the process of the teacher network guiding the student network to extract features. Compared with the student public representation and the student hash code, the teacher public representation and the teacher hash code have a higher correlation with the sample data. The process of determining the loss function based on the teacher public representation, the student public representation, the teacher hash code and the student hash code, and then adjusting the network parameters is equivalent to the process of adjusting the student public representation and the student hash code based on the teacher public representation and the teacher hash code.

[0073] The embodiment of the present invention uses the teacher network to guide the student network to perform model training, so that the trained student network can retain the advantage of high feature extraction efficiency and improve feature extraction accuracy. When searching for the data to be searched input by the user, the student network can be directly used for feature extraction to determine the student public representation, and the student hash code can be determined based on the student public representation, and then the search result can be determined based on the student hash code, and finally accurate and efficient cross-modal hash search can be achieved.

[0074] Here, when the data to be retrieved is input into the trained retrieval model and the hash code corresponding to the data to be retrieved is obtained, the data to be retrieved can be input into the student network in the trained retrieval model, and the student network outputs the student public representation corresponding to the data to be retrieved. The student network inputs the student public representation into the hash conversion network in the trained retrieval model, and the hash conversion network outputs the hash code corresponding to the data to be retrieved.

[0075] Taking text-image cross-modal hash retrieval as an example, the text data of "an eagle is catching rabbits on the grassland" can be input into the student network in the trained retrieval model, and the student network outputs the student public representation corresponding to the above text data. The hash conversion network converts the student public representation corresponding to the above text data into a student hash code corresponding to the above text data. According to the student hash code corresponding to the above text data, the embodiment of the present invention determines a hash code similar to the student hash code corresponding to the above text data from a preset image database, and determines the image data corresponding to the above similar hash code as the final retrieval result.

[0076] The above is the specific training process of the retrieval model. As can be seen from the above, the teacher network can significantly affect the training accuracy of the entire retrieval model. The following is an introduction to the teacher network. Figure 4 , the teacher network is mainly used to perform the following steps 401-406:

[0077] Step 401, receiving a cross-modal data set of multiple samples and word embedding vectors of different labels.

[0078] Taking cross-modal hash retrieval of text and images as an example, this embodiment can input image-text pairs corresponding to multiple samples and word embedding vectors of different labels into the teacher network for supervised learning and unsupervised learning.

[0079] Step 402 , extract features from word embedding vectors of different labels respectively, and determine corresponding different label features.

[0080] Here, the teacher network has a built-in label feature extraction subnetwork, which is used to extract label features from word embedding vectors of different labels to obtain different label features.

[0081] The label feature extraction subnetwork includes: a graph convolutional network, and a fully connected layer connected to the output end of the graph convolutional network.

[0082] The general mathematical expression of graph convolutional network can be expressed as:

[0083] Among them, H (g+1)represents the output result of the g+1th layer in the graph convolutional network, h(·) represents the nonlinear activation function corresponding to the graph convolutional network, represents the normalized matrix of the adjacency matrix A, H (g) Represents the output result of the g-th layer in the graph convolutional network, W (g) Represents the weight matrix of the g-th layer in the graph convolutional network.

[0084] When the graph convolutional network is used for label feature extraction, the mathematical expression of the graph convolutional network can be expressed as:

[0085] Among them, H L(g+1) It represents the output result of the g+1th layer in the graph convolutional network when performing label feature extraction. h(·) represents the nonlinear activation function corresponding to the graph convolutional network. Represents the adjacency matrix A when performing label feature extraction L The normalized matrix, H L(g) It represents the output result of the gth layer in the graph convolutional network when performing label feature extraction, W L(g) Represents the weight matrix of the g-th layer in the graph convolutional network when performing label feature extraction.

[0086] Here, when using graph convolutional networks for label feature extraction, the input data of the graph convolutional network, that is, the word embedding vectors corresponding to different labels, can be expressed as H L(0) The word embedding vectors corresponding to different labels can be represented as c rows d 0 A real matrix of columns, that is, Among them, d 0 Represents the dimension of the word embedding vector.

[0087] For example, a two-layer network structure can be set in the graph convolutional network. When used for label feature extraction, the final output result of the graph convolutional network can be expressed as H L(2) .

[0088] Among them, the adjacency matrix corresponding to the label can be expressed as a real number matrix with c rows and c columns, that is, The element in row a and column b of the adjacency matrix corresponding to the label can be expressed as:

[0089]

[0090] represents the element in the ath row and bth column of the adjacency matrix corresponding to the label, ReLU(·) represents the rectified linear unit activation function, P ab represents the element in the ath row and bth column of the conditional probability matrix P, P ak represents the element in the ath row and the kth column of the conditional probability matrix P, and γ represents the adaptive matrix D LThe weight parameter, Denotes the adaptive matrix D L The element at row a and column b in .

[0091] Here, the conditional probability matrix can be expressed as a real number matrix with c rows and c columns, that is, The value of each element in the conditional probability matrix can be calculated by the co-occurrence probability between labels:

[0092]

[0093] Among them, P(L a ,L b ) indicates label L a and L b The probability of co-occurrence in the data set, P(L b ) indicates label L b The probability of appearing in the dataset.

[0094] The embodiment of the present invention maps the final output result of the graph convolutional network through a fully connected layer to obtain different label features. Different label features can be expressed in the form of a real number matrix with c rows and d columns:

[0095] Among them, L represents the real number matrix formed by different label features, l a represents the ath label feature, and d represents the dimension of the common representation space.

[0096] Step 403 , extract features from the cross-modal data set of each sample respectively, and obtain initial sample features of different modalities corresponding to each sample.

[0097] Here, different feature extraction models can be used to extract features from data of different modalities in the cross-modal data group, respectively, to obtain initial sample features of different modalities corresponding to each sample.

[0098] Taking text image cross-modal hash retrieval as an example, the cross-modal data group of each sample includes the image data corresponding to each sample and the text data corresponding to each sample. The initial sample features of different modalities corresponding to each sample include the initial image features corresponding to each sample and the initial text features corresponding to each sample.

[0099] For the image data corresponding to each sample, the VGG-19 model can be used to extract the image vector corresponding to each sample. Subsequently, the image vector corresponding to each sample is mapped through two fully connected layers in turn to obtain the initial image features corresponding to each sample.

[0100] For the text data corresponding to each sample, the bag-of-words model can be used to extract the text vector corresponding to each sample. Subsequently, the image vector corresponding to each sample is mapped through two fully connected layers in turn to obtain the initial text features corresponding to each sample.

[0101] Step 404: construct a sample label graph based on the initial sample features of different modalities and different label features corresponding to each sample, and determine the supervised sample features of different modalities corresponding to each sample based on the sample label graph. The sample label graph is used to represent the structural connection relationship between the initial sample features of different modalities and different label features corresponding to each sample.

[0102] Graph is a data structure composed of vertices and edges, used to describe the relationship between things. The mathematical expression of Graph is: se =(V se ,E se ). Among them, G se Represents Graph, V se represents the vertex set, E se Represents an edge set, which can be represented by the adjacency matrix corresponding to the Graph.

[0103] The embodiment of the present invention describes the relationship between the initial sample features of different modalities and different label features by constructing a sample label graph, wherein the vertices in the sample label graph are used to represent the initial sample features or label features of different modalities, and the edges in the sample label graph are used to represent the connection relationship between the vertices.

[0104] In the embodiment of the present invention, on the basis of the sample label Graph, a supervised sample feature can be generated through a graph convolutional network. Here, the sample label Graph can be input into the graph convolutional network to obtain the supervised sample feature output by the graph convolutional network. The sample label Graph contains the initial sample features of different modalities, different label features, and the adjacency matrix corresponding to the sample label Graph.

[0105] When used to generate supervised sample features, the graph convolutional network can be expressed as:

[0106]

[0107] Among them, H se(g+1) It represents the output result of the g+1th layer in the graph convolutional network when used to generate supervised sample features. h(·) represents the nonlinear activation function corresponding to the graph convolutional network. Represents the adjacency matrix A corresponding to the sample label Graph se The normalized matrix, H se(g)Represents the output result of the g-th layer in the graph convolutional network when used to generate supervised sample features, W se(g) Represents the weight matrix of the g-th layer in the graph convolutional network when generating supervised sample features.

[0108] Here, when using the graph convolutional network to generate supervised sample features, the input data H of the graph convolutional network is se(0) It can be expressed as

[0109] Among them, F represents the initial sample features of different modalities corresponding to each sample, and L represents different label features.

[0110] Take text image cross-modal hash retrieval as an example. in, Indicates the initial image features corresponding to each sample, v i Represents the initial image features corresponding to the i-th sample. Represents the initial text features corresponding to each sample, represents a real matrix with 2N rows and d columns, t i Represents the initial text feature corresponding to the i-th sample.

[0111] Alternatively, it can be expressed as F = {f 1 ,…,f N ,f N+1 ,...,f 2N}, where {f 1 ,...,f N}={v 1 ,...,v N}, {f N+1 ,...,f 2N}={t 1 ,...,t N That is to say, the initial image features corresponding to each sample and the initial text features corresponding to each sample are merged and spliced ​​to form the initial sample features F of different modalities corresponding to each sample.

[0112] The graph convolutional network in the embodiment of the present invention can be configured with a two-layer network structure. When used to generate supervised sample features, the final output result H of the graph convolutional network is se(2) It can be expressed as

[0113] Among them, F se Indicates the supervised sample features of different modalities corresponding to each sample, L seRepresents different supervised label features. It should be noted that the output results of the graph convolutional network contain not only supervised sample features, but also supervised label features. In other words, the teacher network can determine not only supervised sample features but also supervised label features during supervised learning.

[0114] Taking text-image cross-modal hash retrieval as an example, the supervised sample features of different modalities corresponding to each sample include the supervised image features corresponding to each sample and the supervised text features corresponding to each sample. Among them, V se Indicates that each sample corresponds to a supervised image feature, T se Represents the supervised text features corresponding to each sample.

[0115] The sample label Graph contains the initial sample features and different label features under different modalities. On the one hand, the initial sample features under different modalities can be fused with each other's features through the graph convolutional network to reduce the heterogeneity within and between modalities. On the other hand, the label features and sample features can also be fused with each other's features through the graph convolutional network based on the correlation. In addition, the label features in the sample label Graph can get rid of the locality generated by the batch during training, and can have a global consistent structural relationship, which can better guide the update of sample features.

[0116] Step 405, construct a sample graph according to the initial sample features of different modalities corresponding to each sample, and based on the sample graph, determine the unsupervised sample features of different modalities corresponding to each sample; the sample graph is used to represent the structural connection relationship between the initial sample features of different modalities corresponding to each sample.

[0117] The embodiment of the present invention describes the relationship between the initial sample features of different modalities by constructing a sample graph, wherein the vertices in the sample graph are used to represent the initial sample features of different modalities, and the edges in the sample graph are used to represent the connection relationship between the vertices.

[0118] In the embodiment of the present invention, based on the sample Graph, an unsupervised sample feature can be generated through a graph convolutional network. Here, the sample Graph can be input into the graph convolutional network to obtain the unsupervised sample feature output by the graph convolutional network. The sample Graph contains the initial sample features of different modalities and the adjacency matrix corresponding to the sample Graph.

[0119] When used to generate unsupervised sample features, the graph convolutional network can be expressed as:

[0120]

[0121] Among them, H ue(g+1) It represents the output result of the g+1th layer in the graph convolutional network when used to generate unsupervised sample features. h(·) represents the nonlinear activation function corresponding to the graph convolutional network. Represents the adjacency matrix A corresponding to the sample Graph ue The normalized matrix, H ue(g) Represents the output result of the g-th layer in the graph convolutional network when used to generate unsupervised sample features, W ue(g) Represents the weight matrix of the g-th layer in the graph convolutional network when generating unsupervised sample features.

[0122] Here, when using the graph convolutional network to generate unsupervised sample features, the input data H of the graph convolutional network is ue(0) It can be expressed as

[0123] The graph convolutional network in the embodiment of the present invention can be configured with a two-layer network structure. When used to generate unsupervised sample features, the final output result H of the graph convolutional network is ue(2) It can be expressed as H ue(2) =F ue .

[0124] Among them, F ue Represents the unsupervised sample features of different modalities corresponding to each sample.

[0125] Taking text-image cross-modal hash retrieval as an example, the unsupervised sample features of different modalities corresponding to each sample include: the unsupervised image features corresponding to each sample, and the unsupervised text features corresponding to each sample, that is, Among them, V ue Indicates the unsupervised image features corresponding to each sample, T ue Represents the unsupervised text features corresponding to each sample.

[0126] Step 406 , respectively fuse and map the supervised sample features and unsupervised sample features of different modalities corresponding to each sample, and obtain the teacher common representation of different modalities corresponding to each sample.

[0127] In an embodiment of the present invention, the supervised sample features and unsupervised sample features of different modalities corresponding to each sample can be merged respectively, and the merged sample features can be input into the corresponding fully connected layer to be mapped to the common representation space to obtain the teacher common representation of different modalities corresponding to each sample.

[0128] Taking text-image cross-modal hash retrieval as an example, the teacher public representations of different modalities can include the teacher public representation under the image modality and the teacher public representation under the text modality. The teacher public representation under the image modality can be expressed as:

[0129] The teacher's public representation in text mode can be expressed as:

[0130] Among them, V tea represents the teacher’s common representation in the image modality, f v represents the fully connected layer that maps the merged image features, T tea represents the teacher's public representation in text mode, f t represents the fully connected layer that maps the merged text features. Represents a real matrix with N rows and d columns.

[0131] In addition, the teacher common representation in image modality can also be expressed as: in, represents the teacher’s common representation corresponding to the i-th sample in the image modality.

[0132] The teacher's public representation in text mode can also be expressed as: in, represents the teacher’s common representation corresponding to the i-th sample in the text modality.

[0133] The teacher network in the embodiment of the present invention can realize supervised learning by constructing a sample label Graph and obtain supervised sample features. At the same time, the teacher network also realizes unsupervised learning by constructing a sample Graph and obtains unsupervised sample features. By fusing supervised features and unsupervised features and then mapping to obtain a teacher public representation, the teacher public representation can retain the semantic information and internal rules in the sample at the same time, thereby effectively improving the correlation between the teacher public representation and the sample, and then improving the retrieval accuracy of the retrieval model.

[0134] The following is a detailed introduction to the construction methods of sample label Graph and sample Graph:

[0135] In some embodiments, when constructing a sample label Graph, the label vectors corresponding to sample data of different modalities can be obtained respectively, and the sample adjacency matrix and the sample label adjacency matrix can be constructed respectively according to the label vectors corresponding to the sample data of different modalities; then, the adjacency matrix corresponding to the sample label Graph is constructed according to the sample adjacency matrix and the sample label adjacency matrix; finally, the sample label Graph is determined according to the adjacency matrix corresponding to the sample label Graph, the initial sample features and the label features.

[0136] The embodiments of the present invention can be based on Construct a sample adjacency matrix.

[0137] in, Represents the sample adjacency matrix A s The element at row v and column w in Represents the label vector corresponding to the vth data, represents the label vector corresponding to the w-th data, and τ represents the semantic sensitivity. Here, That is, A s is a real matrix with 2N rows and 2N columns.

[0138] It should be noted that each sample corresponds to two different modal data. When the number of samples is N, the number of data is 2N. For each data, the label vector corresponding to the data is the label vector corresponding to the sample corresponding to the data.

[0139] The embodiments of the present invention can be based on Construct sample label adjacency matrix A sL .

[0140] in, Represents the element in the vth row and ath column of the sample label adjacency matrix, Represents the label identifier of the data. When the vth data belongs to the ath label, otherwise, here, That is, A sL is a real matrix with 2N rows and c columns.

[0141] The embodiments of the present invention can be based on Construct the adjacency matrix corresponding to the sample label Graph.

[0142] Among them, A se Represents the adjacency matrix corresponding to the sample label Graph, A s represents the sample adjacency matrix, A sL represents the sample label adjacency matrix, T represents the transposition factor, Represents the identity matrix, which is a real matrix with c rows and c columns.

[0143] According to the adjacency matrix corresponding to the sample label graph, different initial sample features and label features, the structural connection relationship between different initial sample features and label features can be determined to obtain the sample label graph. The sample label graph can reflect the semantic relevance between sample data of different modalities. Supervised learning is performed based on the sample label graph to fully explore the semantic information of sample data of different modalities.

[0144] In some embodiments, when constructing a sample Graph, the feature correlation matrix between different initial sample features can be determined based on the initial sample features of different modalities corresponding to each sample; then, the adjacency matrix corresponding to the sample Graph is constructed based on the feature correlation matrix; finally, the sample Graph is determined based on the adjacency matrix corresponding to the sample Graph and the initial sample features.

[0145] Specifically, the value of each element in the feature correlation matrix C can be determined by calculating the cosine similarity between different initial sample features.

[0146] The specific calculation formula can be expressed as: C vw =cos(f v ,f w ).

[0147] Among them, C vw represents the element value of the vth row and wth column in the feature correlation matrix, f v represents the vth initial sample feature, f w Represents the wth initial sample feature.

[0148] according to Calculate the adjacency matrix A corresponding to the sample Graph ue The value of each element in . Where η represents the threshold parameter.

[0149] It can be understood that when the number of samples is N, a total of 2N different modal data are generated, and 2N different modal initial sample features are generated accordingly. Here, the initial sample features correspond to the data of different modalities one by one. Accordingly, the feature correlation matrix and the adjacency matrix corresponding to the sample Graph can both be expressed as a real number matrix with 2N rows and 2N columns, that is,

[0150] According to the adjacency matrix corresponding to the sample graph and different initial sample features, the structural connection relationship between different initial sample features can be determined to obtain the sample graph. The sample graph can reflect the feature correlation between sample data of different modalities. Unsupervised learning based on the sample graph can fully explore the inherent laws of sample data of different modalities.

[0151] In the training process of the retrieval model, the teacher network is used to determine the teacher's common representation under different modalities based on the sample data of different modalities and different label features. The student network is used to determine the student's common representation under different modalities based on the sample data of different modalities. The hash conversion network is used to convert the teacher's common representation and the student's common representation into the teacher's hash code and the student's hash code.

[0152] The embodiment of the present invention can determine the value of the loss function through the above-mentioned teacher public representation, student public representation, teacher hash code and student hash code, and then adjust the network parameters of the teacher network, student network and hash conversion network according to the value of the loss function.

[0153] The following is a detailed introduction to the calculation process of the loss function:

[0154] The embodiment of the present invention can first determine the semantic relevance loss function according to the teacher's public representation of different modalities corresponding to each sample, and then determine the teacher-student feature alignment loss function according to the teacher's public representation and the student's public representation of different modalities corresponding to each sample, and then. Obtain supervised label features, and determine a multi-label feature matrix based on the supervised label features; determine the teacher-student semantic difference preservation loss function based on the multi-label feature matrix, the teacher's public representation and the student's public representation; then determine the hash loss function based on the teacher's hash code and the student's hash code; finally, perform weighted summation on the semantic relevance loss function, the teacher-student feature alignment loss function, the teacher-student semantic difference preservation loss function and the hash loss function to determine the loss function corresponding to the retrieval model.

[0155] Here, the semantic relevance loss function is used to measure the relevance between samples. The teacher-student feature alignment loss function is used to constrain the output of the student network using the output of the teacher network, so as to achieve the purpose of the teacher network guiding the student network. The teacher-student semantic difference preservation loss function is used to characterize the difference between the teacher's public representation and the student's public representation, and the intrinsic connection between the semantic difference of multi-label features, so that the difference of samples in the public representation space is consistent with their difference in the semantic space. The hash loss function is used to characterize the difference between the hash code and the corresponding public representation. By setting the hash loss function, the hash code in the Hamming space can learn the properties of the public representation of the original semantic space.

[0156] When determining the semantic relevance loss function, the embodiment of the present invention can first determine the semantic relevance loss functions under different modalities and the semantic relevance loss functions between different modalities, and then, by accumulating the semantic relevance loss functions under different modalities and the semantic relevance loss functions between different modalities, a final semantic relevance loss function is obtained.

[0157] Taking text-image cross-modal hash retrieval as an example, we can determine the semantic relevance loss function under the text modality, the semantic relevance loss function under the image modality, and the semantic relevance loss function between the text and image modalities, and then determine the final semantic relevance loss function.

[0158] Specifically, the semantic relevance loss function under the image modality is determined according to the following formula:

[0159]

[0160] in, represents the semantic relevance loss function in image mode, I(·) represents the indicator function, and y j represents the label vector corresponding to the jth sample, represents the teacher's common representation corresponding to the jth sample in the image modality, Used to indicate semantic weight.

[0161] Similarly, the semantic relevance loss function under text mode is determined according to the following formula:

[0162]

[0163] in, represents the semantic relevance loss function in text mode, represents the teacher’s common representation corresponding to the j-th sample in the text modality.

[0164] The semantic relevance loss function between text and image modalities is determined according to the following formula:

[0165]

[0166] in, Represents the semantic relevance loss function between text and image modalities.

[0167] On the basis of the above, according to Determine the final semantic relevance loss function. Among them, represents the final semantic relevance loss function.

[0168] The embodiment of the present invention determines the positive and negative sample pairs of samples based on whether there are the same label vectors between the samples. If two samples have the same label vectors, they are regarded as positive sample pairs, shortening the distance between the samples. Otherwise, they are regarded as negative sample pairs, increasing the distance between the samples. At the same time, by introducing semantic weights, it can be used to adjust the similarity relationship between positive sample pairs. Among them, the more labels shared between two positive samples, the greater the semantic weight.

[0169] The formula of the teacher-student feature alignment loss function in the embodiment of the present invention can be expressed as:

[0170] in, represents the teacher-student feature alignment loss function, V stu and V tea Represent the student public representation and teacher public representation under the image modality, T stu and T teaRepresent the student public representation and teacher public representation in text mode, respectively, ||·|| F represents the Frobenius norm.

[0171] The teacher-student feature alignment function can utilize the teacher common representation output by the teacher network to constrain the student common representation output by the student network, which is beneficial for the teacher network to guide the student network.

[0172] The embodiment of the present invention uses the teacher network to guide the student network during the model training process, and can seamlessly transfer the semantic knowledge learned from the teacher network to the lightweight student network. In the process of model application, the student network is used to determine the corresponding student public representation, and then the hash code is determined, which can effectively reduce the complexity and resource requirements of the model. In addition, the output accuracy of the student network can be effectively guaranteed without building a graph.

[0173] In some embodiments, M=Y(L se ⊙W m ) Determine the multi-label feature matrix;

[0174] Among them, M represents the multi-label feature matrix, m i Represents the multi-label feature corresponding to the i-th sample, N b represents the number of samples, y i Represents the label vector corresponding to the i-th sample, L se represents the supervised label feature, W m represents the weight matrix.

[0175] On this basis, according to

[0176]

[0177] Determine the loss function that preserves the semantic difference between teacher and student;

[0178] in, represents the loss function that preserves the semantic difference between teacher and student, and They represent the teacher’s public representation corresponding to the i-th sample in different modes, and They represent the student common representation corresponding to the jth sample under different modalities, m i Represents the multi-label feature corresponding to the i-th sample, m j Represents the multi-label feature corresponding to the jth sample.

[0179] Take text image cross-modal hash retrieval as an example. Indicates the teacher's public representation corresponding to the i-th sample in the image mode, They represent the teacher’s public representation corresponding to the i-th sample in the text mode, represents the student common representation corresponding to the jth sample in the image modality, represents the student common representation corresponding to the j-th sample in the text modality.

[0180] Here, by defining the loss function of the semantic difference between teachers and students, the difference between the teacher's public representation and the student's public representation can be made consistent with the semantic difference between the corresponding multi-label features. That is to say, when performing model training, the embodiment of the present invention not only considers the correlation between different samples, but also considers the difference between different samples, and improves the training accuracy of the model by comprehensively considering the correlation and difference.

[0181] Different from the related art which only considers the correlation between different samples when constructing the loss function, the embodiment of the present invention also considers the differences between different samples when constructing the loss function, thereby improving the training accuracy of the retrieval model and further improving the retrieval accuracy of cross-modal retrieval.

[0182] The hash loss function in the embodiment of the present invention mainly includes: a teacher-hash similarity loss function, a student-hash similarity loss function, and a hash code alignment loss function.

[0183] The mathematical expression of the teacher-hash similarity loss function is as follows:

[0184]

[0185] Among them, F tea Represents the common representation of teachers of different modalities, B t Teacher hash codes representing different modalities.

[0186] The mathematical expression of the student-hash similarity loss function is as follows:

[0187]

[0188] Among them, F stu Represents the common representation of students of different modalities, B s Hash codes representing students of different modalities.

[0189] The mathematical expression of the hash code alignment loss function is as follows:

[0190]

[0191] Among them, B tv and B tt Represents the teacher hash code of image modality and text modality respectively, B sv and B stRepresent the student hash codes for image modality and text modality respectively.

[0192] On this basis, the loss function corresponding to the retrieval model can be expressed as:

[0193]

[0194] in, represents the loss function corresponding to the retrieval model, and α, β, and δ represent weight parameters respectively.

[0195] When training the model, we can set the batch size to 128, the epoch to 100, and use the Adam optimizer to optimize the loss function corresponding to the retrieval model. The initial learning rate is 0.0001, and α, β, and δ are set to 0.2, 0.5, and 0.1 respectively through experiments. The weight parameter γ of the adaptive matrix is ​​set to 0.5, the semantic sensitivity τ is set to 0.3, and the threshold parameter η is set to 0.2.

[0196] After the model training is completed, the trained retrieval model can be used for cross-modal retrieval, and the average precision score is used to evaluate the performance of cross-modal retrieval. The calculation formula of the average precision score is as follows:

[0197]

[0198] Where q represents the number of samples in the preset database that are similar to the data to be retrieved, and P(r) refers to the accuracy of the first r retrieval results. If the rth retrieval result is similar to the data to be retrieved, then δ(r) = 1, otherwise δ(r) = 0. R represents the total amount of data in the preset database.

[0199] The following takes text image cross-modal hash retrieval as an example, combined with Figure 5 This paper introduces the training process of the retrieval model.

[0200] Get a dataset and word embedding vectors of different labels; the dataset contains image sample pairs corresponding to multiple samples.

[0201] The teacher network extracts image features from the image data corresponding to each sample, and extracts text features from the text data corresponding to each sample, and obtains initial image features and initial text features corresponding to each sample.

[0202] The teacher network also extracts features from the word embedding vectors of different labels, determines different label features, and constructs a sample label graph based on the above initial image features, initial text features, and label features. Here, different label features are obtained by extracting features from the word embedding vectors corresponding to different labels.

[0203] The teacher network determines the supervised sample features and supervised label features under different modalities corresponding to different samples based on the sample label Graph. In addition, the teacher network determines the unsupervised sample features under different modalities corresponding to different samples based on the sample Graph.

[0204] The teacher network merges and maps the supervised and unsupervised features under different modalities corresponding to different samples, and determines the teacher's common representation under different modalities corresponding to different samples.

[0205] The student network extracts image features from image data corresponding to different samples, and extracts text features from text data corresponding to different samples, and obtains initial image features and initial text features corresponding to different samples.

[0206] The hash conversion network performs hash conversion on the teacher's common representation and the student's common representation of different samples in different modes to obtain the teacher's hash code and the student's hash code of different samples in different modes. The embodiment of the present invention determines the value of the loss function corresponding to the retrieval model based on the above-mentioned teacher's common representation, student's common representation, teacher's hash code, student's hash code and supervised label features, and adjusts the network parameters of the teacher network, the student network and the hash conversion network according to the value of the loss function until the loss function is lower than the preset value, thereby obtaining a trained retrieval model.

[0207] The following is an embodiment of the device of the present invention. For details not described in detail therein, reference may be made to the corresponding method embodiment described above.

[0208] Figure 6 A schematic diagram of the structure of a cross-modal hash retrieval device based on hybrid correlation learning provided by an embodiment of the present invention is shown. For ease of explanation, only the part related to the embodiment of the present invention is shown, which is described in detail as follows:

[0209] like Figure 6 As shown, the cross-modal hash retrieval device 6 based on hybrid correlation learning includes: an acquisition module 61, a processing module 62 and a retrieval module 63.

[0210] The acquisition module 61 is used to obtain the data to be retrieved. The processing module 62 is used to input the data to be retrieved into the trained retrieval model to obtain the hash code corresponding to the data to be retrieved output by the retrieval model. The retrieval module 63 is used to determine the retrieval result corresponding to the data to be retrieved from a preset database according to the hash code. The retrieval model is obtained by supervised learning and unsupervised learning training based on sample data of different modalities and word embedding vectors of different labels.

[0211] The cross-modal hash retrieval method device based on hybrid correlation learning provided by the embodiment of the present invention can be used to implement the cross-modal hash retrieval method based on hybrid correlation learning. The technical principle and implementation effect of this device embodiment are the same as those of the above method embodiment, and will not be repeated here.

[0212] Figure 7 Schematic diagram of an electronic device provided by an embodiment of the present invention. Figure 7 As shown, the electronic device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor 70 executes the computer program 72, the functions of the modules in the above-mentioned device embodiments are implemented.

[0213] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the technical solutions from the spirit and scope of the technical solutions of the aforementioned embodiments, and should all be included in the protection scope of the present invention.

Claims

1. A cross-modal hash retrieval method based on hybrid correlation learning, characterized in that: include: Get the data to be retrieved; Inputting the data to be retrieved into a trained retrieval model to obtain a hash code output by the retrieval model corresponding to the data to be retrieved; According to the hash code, determining a search result corresponding to the data to be searched from a preset database; The retrieval model is obtained by respectively conducting supervised learning and unsupervised learning training based on sample data of different modalities and word embedding vectors of different labels.

2. According to claim 1, the cross-modal hash retrieval method based on hybrid correlation learning is characterized in that: Before inputting the data to be retrieved into the trained retrieval model to obtain the hash code output by the retrieval model and corresponding to the data to be retrieved, the method further includes: Obtain sample data of different modalities and word embedding vectors of different labels respectively; According to the sample data of different modalities and the word embedding vectors of different labels, the retrieval model is controlled to perform supervised learning and unsupervised learning respectively to determine the supervised sample features and unsupervised sample features corresponding to the sample data of different modalities, and according to the supervised sample features and the unsupervised sample features, the common representation and hash code corresponding to the sample data of different modalities are determined; According to the common representation and the hash code, the loss function corresponding to the retrieval model is determined, and when the value of the loss function is greater than or equal to a preset value, the model parameters of the retrieval model are adjusted until the value of the loss function is less than the preset value, thereby obtaining a trained retrieval model.

3. The cross-modal hash retrieval method based on hybrid correlation learning according to claim 2 is characterized in that: The retrieval model includes: a teacher network, a student network and a hash conversion network; According to the sample data of different modalities and the word embedding vectors of different labels, the control retrieval model performs supervised learning and unsupervised learning respectively, including: Input sample data of different modalities and word embedding vectors of different labels into the teacher network to obtain the teacher common representation corresponding to the sample data of different modalities output by the teacher network; the teacher network performs supervised learning and unsupervised learning respectively according to the input sample data of different modalities and word embedding vectors of different labels to determine the supervised sample features and unsupervised sample features corresponding to the sample data of different modalities; determine the teacher common representation corresponding to the sample data of different modalities according to the supervised sample features and the unsupervised sample features; The teacher's public representation is input into the hash conversion network to obtain the teacher's hash code; the hash conversion network performs hash conversion according to the input public representation to obtain a hash code corresponding to the public representation; Inputting sample data of different modalities into the student network, obtaining a student common representation corresponding to the sample data of different modalities output by the student network; the student network performs feature extraction and mapping processing according to the input sample data of different modalities, obtaining a student common representation corresponding to the sample data of different modalities; The student public representation is input into the hash conversion network to obtain the student hash code.

4. The cross-modal hash retrieval method based on hybrid correlation learning according to claim 3 is characterized in that: The sample data of different modalities includes a plurality of cross-modal data groups of samples, wherein the cross-modal data group of each sample includes data of different modalities corresponding to the sample; The teacher network is used to: Receive a cross-modal data set of multiple samples and word embedding vectors of different labels; Feature extraction is performed on word embedding vectors of different labels respectively, and different label features are determined accordingly; Perform feature extraction on the cross-modal data group of each sample respectively, and obtain the initial sample features of different modalities corresponding to each sample; According to the initial sample features of different modalities and different label features corresponding to each sample, a sample label graph is constructed, and based on the sample label graph, the supervised sample features of different modalities corresponding to each sample are determined; the sample label graph is used to represent the structural connection relationship between the initial sample features of different modalities and different label features corresponding to each sample; According to the initial sample features of different modalities corresponding to each sample, a sample graph is constructed, and based on the sample graph, unsupervised sample features of different modalities corresponding to each sample are determined; the sample graph is used to represent the structural connection relationship between the initial sample features of different modalities corresponding to each sample; The supervised sample features and unsupervised sample features of different modalities corresponding to each sample are fused and mapped respectively to obtain the teacher common representation of different modalities corresponding to each sample.

5. The cross-modal hash retrieval method based on hybrid correlation learning according to claim 4 is characterized in that: The sample label Graph is constructed according to the initial sample features of different modalities and different label features corresponding to each sample, including: Obtain label vectors corresponding to sample data of different modalities respectively, and construct sample adjacency matrices and sample label adjacency matrices respectively according to the label vectors corresponding to sample data of different modalities; Constructing an adjacency matrix corresponding to the sample label Graph according to the sample adjacency matrix and the sample label adjacency matrix; The sample label Graph is determined according to the adjacency matrix corresponding to the sample label Graph, the initial sample features, and the label features.

6. The cross-modal hash retrieval method based on hybrid correlation learning according to claim 4 is characterized in that: The sample Graph is constructed according to the initial sample features of different modes corresponding to each sample, including: According to the initial sample features of different modes corresponding to each sample, a feature correlation matrix between different initial sample features is determined; Constructing an adjacency matrix corresponding to the sample Graph according to the feature correlation matrix; The sample Graph is determined according to the adjacency matrix corresponding to the sample Graph and the initial sample features.

7. The cross-modal hash retrieval method based on hybrid correlation learning according to any one of claims 3 to 6, characterized in that: The teacher network is also used to perform supervised learning based on sample data of different modalities and word embedding vectors of different labels to determine supervised label features; Determining a loss function corresponding to the retrieval model according to the common representation and the hash code includes: Determine the semantic relevance loss function based on the teacher’s common representation of different modalities corresponding to each sample; According to the teacher's common representation and the student's common representation of different modalities corresponding to each sample, determine the teacher-student feature alignment loss function; Obtaining the supervised label features, and determining a multi-label feature matrix according to the supervised label features; Determine a loss function for preserving teacher-student semantic differences according to the multi-label feature matrix, the teacher common representation, and the student common representation; Determine a hash loss function according to the teacher hash code and the student hash code; The semantic relevance loss function, the teacher-student feature alignment loss function, the teacher-student semantic difference preservation loss function and the hash loss function are weightedly summed to determine the loss function corresponding to the retrieval model.

8. The cross-modal hash retrieval method based on hybrid correlation learning according to claim 7 is characterized in that: According to the supervised label features, a multi-label feature matrix is ​​determined, including: According to M=Y(L se ⊙W m ) Determine the multi-label feature matrix; Wherein, M represents the multi-label feature matrix, m i represents the multi-label feature corresponding to the i-th sample, N represents the number of samples, y i Represents the label vector corresponding to the i-th sample, L se represents the supervised label feature, W m represents the weight matrix.

9. The cross-modal hash retrieval method based on hybrid correlation learning according to claim 7, characterized in that: The step of determining a loss function for preserving the semantic difference between a teacher and a student according to the multi-label feature matrix, the teacher's common representation, and the student's common representation comprises: according to Determine the loss function that preserves the semantic difference between teacher and student; in, represents the loss function for preserving the semantic difference between teacher and students, and They represent the teacher’s public representation corresponding to the i-th sample in different modes, and They represent the student common representation corresponding to the jth sample under different modalities, m i Represents the multi-label feature corresponding to the i-th sample, m j Represents the multi-label feature corresponding to the jth sample.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the cross-modal hash retrieval method based on hybrid correlation learning as described in any one of claims 1 to 9 are implemented.

Images