A cross-modal hash retrieval method, device and equipment based on relational network

Abstract

The present application relates to a cross-modal hash retrieval method, device and equipment based on a relational network. The method first obtains the image feature vector and text feature vector of a sample pair, and obtains a fused feature vector; then, the association relationship between objects of different modalities is mined from the image feature vector, text feature vector and fused feature vector through a relational network, and the core commonality of the paired relationship between object samples is captured; then, the first similarity between the first hash code and the second hash code of each sample pair is calculated according to the hash network, a loss function is constructed according to the second similarity, and finally, the hash network of the hash retrieval model is reversely optimized based on the loss function. The method introduces a relational network into cross-modal hash retrieval, solves the problem that when the feature vectors of multi-modal data are similar, the similarity of modal semantic features is lost, resulting in insufficient accuracy of cross-modal data retrieval, and realizes high-precision cross-modal retrieval of multi-modal data.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of hash retrieval, and in particular to a cross-modal hash retrieval method, apparatus, and device based on a relational network. Background Art

[0002] In the era of big data, the production, storage and circulation of data have reached an unprecedented scale and speed. In this context, data of various modalities, including text, images, etc., have shown explosive growth, and with it a significant increase in data heterogeneity. Since different modalities contain heterogeneous information about the same object, this heterogeneous information not only increases the computational burden during the retrieval process, but also affects the efficiency and accuracy of the retrieval. How to find the key semantics of the retained data and the correlation between the features of data of different modalities has become the core research area of ​​cross-modal retrieval.

[0003] The commonly used cross-modal hash retrieval methods currently rely on learning a common representation of all modalities by assuming that there is an equal amount of information in different modalities. However, most existing cross-modal hashing methods assume that in the process of multimodal representation learning, an equal amount of information can be obtained from different modalities, and try to project multimodal information into a potential single shared space, where the different modalities are intuitively contrasted. Then a widely used distance metric is used to calculate the cross-modal retrieval of different modal hash codes. However, since the amount of information between cross-modal samples is unbalanced and unequal, it is inappropriate to directly match the modality-specific representations of different modes in the common space. When mapping cross-modal features to low-dimensional space, these methods often sacrifice a large number of semantic features, reducing the accuracy of cross-modal hash retrieval.

[0004] Therefore, how to extract high-dimensional data into low-dimensional hash codes without losing too much modal semantic features is a major challenge for current cross-modal hash retrieval methods. Summary of the invention

[0005] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0006] The main purpose of the embodiments of the present disclosure is to propose a cross-modal hash retrieval method, apparatus and device based on a relational network, which can generate compact and strongly correlated hash codes to achieve more accurate cross-modal hash retrieval.

[0007] A first aspect of an embodiment of the present application proposes a cross-modal hash retrieval method based on a relational network, the method comprising:

[0008] Get the target text and target image;

[0009] The target text and the target image are input into a hash retrieval model to obtain a hash retrieval result between a first target hash code corresponding to the target text and a second target hash code corresponding to the target image output by the hash retrieval model, wherein the training process of the hash retrieval model includes:

[0010] Acquire a text feature vector and an image feature vector of each sample pair according to the feature extraction network of the hash retrieval model; the sample pair is composed of any text word segmentation sample of the text to be trained and any image point sample of the image to be trained;

[0011] fusing the text feature vector and the image feature vector of each sample pair according to the fusion network of the hash retrieval model to obtain a fused feature vector of each sample pair;

[0012] Converting the text feature vector of each sample pair into a first hash code and the image feature vector of each sample pair into a second hash code according to the hash network of the hash retrieval model, and calculating a first similarity between the first hash code and the second hash code of each sample pair;

[0013] Calculating the second similarity of the vector group of each sample pair according to the relational network of the hash retrieval model; the vector group of each sample pair is composed of at least two of the text feature vector, the image feature vector and the fusion feature vector of each sample pair;

[0014] According to the second similarity of the vector group of each sample pair, a loss function is constructed to constrain the first similarity between similar hash codes in all sample pairs to be smaller than the first similarity between dissimilar hash codes, and the hash network of the hash retrieval model is reversely optimized based on the loss function until the trained hash retrieval model is obtained.

[0015] This application proposes a cross-modal hash retrieval method based on a relational network, which has the following beneficial effects:

[0016] This method first uses a feature extraction network to extract features from heterogeneous modal data such as images and texts, obtains the image feature vector and text feature vector of each sample pair, and fuses the image feature vector and text feature vector of each sample pair to obtain a fused feature vector; then, through a relational network, the association relationship between objects of different modalities is mined from the image feature vector, the text feature vector and the fused feature vector, the core commonality of the pairwise relationship between object samples is captured, and the second similarity between vector groups of objects of different modalities is calculated to represent a certain relationship representation between vector groups of objects of different modalities; then, the first similarity between the first hash code of the text feature vector and the second hash code of the image feature vector of each sample pair is calculated according to the hash network, and a loss function is constructed according to the second similarity to constrain the first similarity between similar hash codes in all sample pairs to be less than the first similarity between dissimilar hash codes; finally, the hash network of the hash retrieval model is reversely optimized based on the loss function until the hash retrieval model after training is obtained. This embodiment introduces relational networks into cross-modal hash retrieval to solve the problem of insufficient accuracy in cross-modal data retrieval due to loss of modal semantic feature similarity when obtaining similarity for feature vectors of multimodal data. It establishes deep relationships between data pairs between multimodal data, extracts and encodes deep semantic features to achieve high-precision cross-modal retrieval of multimodal data.

[0017] In some embodiments of the present application, the calculating the second similarity of the vector group of each sample pair includes:

[0018] ;

[0019] ;

[0020] in, The feature vector and the eigenvector The concatenation vector of the vector group, the feature vector and the eigenvector is one of the text feature vector, the image feature vector and the fusion feature vector, and the feature vector With the feature vector Not eigenvectors of the same mode; is the first bias term, is the first weight matrix, is a nonlinear activation function, is the second bias term, is the second weight matrix, for The transposed matrix of is the Sigmoid activation function, The feature vector and the eigenvector The second similarity of the group of vectors.

[0021] In some embodiments of the present application, before constructing a loss function for constraining the first similarity between similar hash codes in all sample pairs to be less than the first similarity between dissimilar hash codes according to the second similarity of the vector group of each sample pair, the method further includes:

[0022] Calculate a first second similarity of a vector group consisting of a text feature vector and an image feature vector of each sample pair, a second second similarity of a vector group consisting of an image feature vector and a fused feature vector of each sample pair, and a third second similarity of a vector group consisting of a text feature vector and a fused feature vector of each sample pair;

[0023] Calculate an average value among the first second similarity, the second second similarity and the third second similarity of each sample pair;

[0024] The step of constructing, according to the second similarity of the vector group of each sample pair, a loss function for constraining the first similarity between similar hash codes in all sample pairs to be smaller than the first similarity between dissimilar hash codes, comprises:

[0025] According to the average value of each sample pair, a loss function is constructed for constraining the first similarities between similar hash codes in all sample pairs to be smaller than the first similarities between dissimilar hash codes.

[0026] In some embodiments of the present application, the loss function includes:

[0027] ;

[0028] in, is a sample pair, is a set of similar sample pairs, is a set of dissimilar sample pairs, For sample pairs The average value of For sample pairs The first similarity of is the loss function.

[0029] In some embodiments of the present application, the loss function includes:

[0030] ;

[0031] ;

[0032] ;

[0033] in, is a sample pair, is a set of similar sample pairs, is a set of dissimilar sample pairs, For sample pairs The average value of For sample pairs The first similarity of is the loss function, is the preset boundary value, The maximum value function, is the first intermediate loss function, is the second intermediate loss function, and are the preset weight values ​​respectively.

[0034] In some embodiments of the present application, the fusing of the text feature vector and the image feature vector of each sample pair includes:

[0035] ;

[0036] in, , The sample pairs The text feature vector and image feature vector of For sample pairs The fusion feature vector of and are weight values ​​respectively, and the weight values and The calculation process includes:

[0037] Construct two initial weight values;

[0038] Pair the samples Normalizing the text feature vector and the image feature vector to obtain a normalized text feature vector and an image feature vector;

[0039] According to the normalized text feature vector and image feature vector, the two initial weight values ​​are optimized by using the gradient descent method until the weight value is obtained. and .

[0040] In some embodiments of the present application, the first similarity of the sample pair is the Hamming distance between the first hash code and the second hash code of each sample pair.

[0041] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present invention provides a cross-modal hash retrieval device based on a relational network, the device comprising:

[0042] An image acquisition module, used to acquire target text and target image;

[0043] The result retrieval module is used to input the target text and the target image into a hash retrieval model to obtain a hash retrieval result between a first target hash code corresponding to the target text and a second target hash code corresponding to the target image output by the hash retrieval model, wherein the training process of the hash retrieval model includes:

[0044] Acquire a text feature vector and an image feature vector of each sample pair according to the feature extraction network of the hash retrieval model; the sample pair is composed of one text segmentation sample of the text to be trained and one image point sample of the image to be trained;

[0045] fusing the text feature vector and the image feature vector of each sample pair according to the fusion network of the hash retrieval model to obtain a fused feature vector of each sample pair;

[0046] Converting the text feature vector of each sample pair into a first hash code and the image feature vector of each sample pair into a second hash code according to the hash network of the hash retrieval model, and calculating a first similarity between the first hash code and the second hash code of each sample pair;

[0047] Calculating the second similarity of the vector group of each sample pair according to the relational network of the hash retrieval model; the vector group of each sample pair is composed of the text feature vector and the image feature vector of each sample pair, or is composed of the image feature vector and the fused feature vector of each sample pair, or is composed of the text feature vector and the fused feature vector of each sample pair;

[0048] According to the second similarity of the vector group of each sample pair, a loss function is constructed to constrain the first similarity between similar hash codes in all sample pairs to be smaller than the first similarity between dissimilar hash codes, and the hash network of the hash retrieval model is reversely optimized based on the loss function until the hash retrieval model after training is obtained.

[0049] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present invention provides an electronic device, comprising: at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the above-mentioned cross-modal hash retrieval method based on the relational network.

[0050] To achieve the above-mentioned purpose, a fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned cross-modal hash retrieval method based on a relational network.

[0051] It can be understood that the beneficial effects of the second to fourth aspects compared with the related art are the same as the beneficial effects of the first aspect compared with the related art. Please refer to the relevant description in the first aspect, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 is a flow chart of a cross-modal hash retrieval method based on a relational network provided in an embodiment of the present application;

[0054] Figure 2 It is a training flow chart of the hash retrieval model provided in the embodiment of the present application;

[0055] Figure 3 is a flowchart of a cross-modal hash retrieval method based on a relational network provided in another embodiment of the present application;

[0056] Figure 4 It is a structural diagram of a cross-modal hash retrieval device based on a relational network provided in an embodiment of the present application;

[0057] Figure 5 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0061] like Figure 1 and Figure 2 , an embodiment of the present application provides a cross-modal hash retrieval method based on a relational network, the method comprising the following steps:

[0062] Step S110, obtaining target text and target image;

[0063] Step S120, inputting the target text and the target image into a hash retrieval model, and obtaining a hash retrieval result between a first target hash code corresponding to the target text and a second target hash code corresponding to the target image output by the hash retrieval model, wherein the training process of the hash retrieval model includes:

[0064] Step S210, obtaining a text feature vector and an image feature vector of each sample pair according to a feature extraction network of a hash retrieval model; a sample pair is composed of any text segmentation sample of the text to be trained and any image point sample of the image to be trained;

[0065] Step S220, fusing the text feature vector and the image feature vector of each sample pair according to the fusion network of the hash retrieval model to obtain a fused feature vector of each sample pair;

[0066] Step S230, converting the text feature vector of each sample pair into a first hash code and the image feature vector of each sample pair into a second hash code according to the hash network of the hash retrieval model, and calculating a first similarity between the first hash code and the second hash code of each sample pair;

[0067] Step S240, calculating the second similarity of the vector group of each sample pair according to the relationship network of the hash retrieval model; the vector group of each sample pair is composed of at least two of the text feature vector, the image feature vector and the fusion feature vector of each sample pair;

[0068] Step S250, based on the second similarity of the vector group of each sample pair, construct a loss function for constraining the first similarity between similar hash codes in all sample pairs to be less than the first similarity between dissimilar hash codes, and reversely optimize the hash network of the hash retrieval model based on the loss function until a trained hash retrieval model is obtained.

[0069] In step S110, the target text is text data that needs to be processed by hash retrieval, and the target image is image data that needs to be processed by hash retrieval; in step S120, the hash retrieval model can extract the corresponding first target hash code from the target text, extract the corresponding second target hash code from the target image, and then extract the hash retrieval result from the first target hash code and the second target hash code.

[0070] In this embodiment, hash retrieval, also known as hash search, is an efficient method for data retrieval using a hash function. Its core principle is to map data (such as an image) into a hash code of a fixed length through a specific hash function, and then achieve fast retrieval based on the hash code.

[0071] It should be noted that step S110 and step S120 are the process of using the hash retrieval model, and step S210 to step S260 are the process of training the hash retrieval model.

[0072] In this embodiment, the hash retrieval model includes: a feature extraction network, a fusion network, a hash network and a relationship network. The following is a detailed description of the training process of the hash retrieval model:

[0073] In step S210, the text to be trained refers to the text data used for training, and the image to be trained refers to the image data used for training. A single image to be trained is composed of multiple image points, so there are multiple image point samples, and the text to be trained is composed of multiple text segmentation words, so there are multiple text segmentation word samples. Any image point sample and text segmentation word sample can be used as a sample pair. For example, if there are 10 samples of the image to be trained and 5 samples of the text to be trained, 50 sample pairs can be formed.

[0074] In this embodiment, a dedicated image feature extraction network is first used to extract feature vectors of all image points, and then a dedicated text feature extraction network is used to extract feature vectors of all text word segments, and then the text feature vector and image feature vector of each sample pair are obtained.

[0075] In step S220, the text feature vector and the image feature vector of each sample pair are fused according to the fusion network to obtain a fused feature vector of each sample pair.

[0076] Ways to fuse feature vectors include, but are not limited to: concatenation and weighted fusion.

[0077] This embodiment fuses the text feature vector and image feature vector of each sample pair to obtain the corresponding fused feature vector. After obtaining the fused feature, the association relationship between the text feature vector, image feature vector and fused feature vector of a sample pair can be searched through a subsequent relationship network.

[0078] In step S230, the text feature vector of each sample pair is converted into a first hash code according to the hash network, and the image feature vector of each sample pair is converted into a second hash code, and a first similarity between the first hash code and the second hash code of each sample pair is calculated.

[0079] For the two heterogeneous modal data, image and text, the hash learning method is used here to perform hash encoding on the image and text feature vectors through the hash network. The binary hash code is used instead of the original image and text to be trained for comparison.

[0080] In this embodiment, the first similarity between the first hash code and the second hash code of each sample pair can be calculated using methods such as Euclidean distance and Hamming distance. One purpose of this embodiment is to make the first similarity between hash codes with greater similarity smaller to generate hash codes that meet the constraints, that is, to constrain the hash network through the loss function so that it generates more accurate and strongly correlated hash codes.

[0081] In step S240 , the second similarity of the vector group of each sample pair is calculated according to the relationship network.

[0082] In this embodiment, according to the above steps S210 and S220, three feature vectors of each sample point are obtained, namely, a text feature vector, an image feature vector and a fusion feature vector, and a vector group is composed of at least two of them. The relationship network can calculate the second similarity of the vector group corresponding to each sample pair, and the second similarity represents the potential semantic association degree between the vector groups (for example, the text and image feature vectors of the same sample pair, or the text and fusion feature vectors of the same sample pair, or the image and fusion feature vectors of the same sample pair).

[0083] In step S250, based on the second similarity of the vector group of each sample pair, a loss function is constructed to constrain the first similarity between similar hash codes in all sample pairs to be less than the first similarity between dissimilar hash codes, and the hash network of the hash retrieval model is reversely optimized based on the loss function until the hash retrieval model after training is obtained. This embodiment uses a loss function to constrain the hash code generated in step S220, and introduces the second similarity of the vector group of each sample pair obtained by the relationship network into the loss function to strengthen the connection between hash codes with strong semantic association. The loss function is used to make the first similarity between hash codes with greater similarity smaller to generate hash codes that meet the constraints. The optimization process is repeated until the loss function converges, so that the model can obtain compact and strongly correlated hash codes.

[0084] The cross-modal hash retrieval method based on the relationship network provided in this embodiment has the following technical effects:

[0085] The method first uses a feature extraction network to extract features from heterogeneous modal data such as images and texts, obtains image feature vectors and text feature vectors for each sample pair, and fuses the image feature vectors and text feature vectors for each sample pair to obtain a fused feature vector; then, the association relationship between objects of different modalities is mined from the image feature vectors, text feature vectors and fused feature vectors through a relational network, the core commonality of the pairwise relationship between object samples is captured, and the second similarity between vector groups of objects of different modalities is calculated to represent a certain relationship representation between vector groups of objects of different modalities; then, the first similarity between the first hash code of the text feature vector and the second hash code of the image feature vector of each sample pair is calculated according to the hash network, and a loss function is constructed according to the second similarity to constrain the first similarity between similar hash codes in all sample pairs to be less than the first similarity between dissimilar hash codes; finally, the hash network of the hash retrieval model is reversely optimized based on the loss function until a hash retrieval model after training is obtained, and the hash retrieval model after training can generate compact and strongly correlated hash codes to achieve more accurate cross-modal hash retrieval.

[0086] In some embodiments of the present application, the process of calculating the second similarity of the vector group in the relational network includes:

[0087] ;

[0088] ;

[0089] in, The feature vector and the eigenvector The concatenation vector of the vector group, the feature vector and the eigenvector is one of the text feature vector, image feature vector and fusion feature vector, and the feature vector With the eigenvector are not eigenvectors of the same mode. is the first bias term, is the first weight matrix, is a nonlinear activation function, is the second bias term, is the second weight matrix, for The transposed matrix of is the Sigmoid activation function, The feature vector and the eigenvector The second similarity of the group of vectors.

[0090] In some embodiments of the present application, before constructing a loss function for constraining the first similarity between similar hash codes in all sample pairs to be less than the first similarity between dissimilar hash codes in step S250 according to the second similarity of the vector group of each sample pair, the method further includes the following steps:

[0091] Step S310, calculating a first second similarity of a vector group consisting of a text feature vector and an image feature vector of each sample pair, a second second similarity of a vector group consisting of an image feature vector and a fused feature vector of each sample pair, and a third second similarity of a vector group consisting of a text feature vector and a fused feature vector of each sample pair;

[0092] Step S320, calculating the average of the first second similarity, the second second similarity and the third second similarity of each sample pair;

[0093] The step S250 constructs a loss function for constraining the first similarity between similar hash codes in all sample pairs to be smaller than the first similarity between dissimilar hash codes according to the second similarity of the vector group of each sample pair, including:

[0094] According to the average value of each sample pair, a loss function is constructed for constraining the first similarities between similar hash codes in all sample pairs to be smaller than the first similarities between dissimilar hash codes.

[0095] This embodiment directly uses the average value to construct the loss function, by calculating the second similarity of the vector group composed of the text feature vector and the image feature vector of each sample pair, the second similarity of the vector group composed of the image feature vector and the fused feature vector of each sample pair, and the second similarity of the vector group composed of the text feature vector and the fused feature vector of each sample pair, and the average value of these three second similarities is obtained by averaging the values ​​of the three second similarities. The average value can reflect the central tendency of the three second similarities, balance the values ​​of the three second similarities, and can construct a more accurate loss function.

[0096] In some embodiments of the present application, the loss function includes:

[0097] ;

[0098] in, is a sample pair, is a set of similar sample pairs, is a set of dissimilar sample pairs, For sample pairs The average value of For sample pairs The first similarity of is the loss function.

[0099] In this embodiment, the first similarity is the Hamming distance. It is expressed as follows:

[0100] ;

[0101] for A set of hash codes, is the second hash code, is the first hash code, is the norm.

[0102] In some embodiments of the present application, the loss function includes:

[0103] ;

[0104] ;

[0105] ;

[0106] in, is a sample pair, is a set of similar sample pairs, is a set of dissimilar sample pairs, For sample pairs The average value of For sample pairs The first similarity of is the loss function, is the preset boundary value, The maximum value function, is the first intermediate loss function, is the second intermediate loss function, and are the preset weight values ​​respectively.

[0107] The loss function provided in the above embodiment The difference is that the loss function of this embodiment is Increased margin ,By setting a suitable margin value, the distribution of the hash codes can be better controlled so that ,dissimilar samples can be effectively separated while keeping similar ,samples compact.

[0108] In some embodiments of the present application, the text feature vector and the image feature vector of each sample pair are fused, including:

[0109] ;

[0110] in, , The sample pairs The text feature vector and image feature vector of For sample pairs The fusion feature vector of and are weight values ​​respectively, and the weight and The calculation process includes:

[0111] Step S410, constructing two initial weight values;

[0112] Step S420: Normalizing the text feature vector and the image feature vector to obtain a normalized text feature vector and an image feature vector;

[0113] Step S430, according to the normalized text feature vector and image feature vector, the two initial weight values ​​are optimized by using the gradient descent method until the weights are obtained. and .

[0114] In this embodiment, through the above step S410 to step S430, a more accurate weight value can be assigned by using the gradient descent method to improve the effect of fusing the feature vector.

[0115] This embodiment can be applied to the following scenarios:

[0116] For large-scale cross-modal retrieval (searching for text with images, searching for images with text), the trained model can perform cross-modal retrieval between different modalities. Storing massive data in the form of low-dimensional hash codes greatly reduces storage space while ensuring retrieval accuracy, which is suitable for cross-modal retrieval projects with large data sets.

[0117] like Figure 3 For ease of understanding, an embodiment of a cross-modal hash retrieval method based on a relational network is provided, and the method comprises the following steps:

[0118] Step S910, extracting features of images and texts through a feature extraction network of a hash retrieval model.

[0119] The hash retrieval model is set up with two feature extraction networks, one for extracting image features from images and one for extracting text features from text. It should be noted that the structures of the two feature extraction networks are shown in the subsequent process.

[0120] The multimodal data involved in this embodiment mainly includes image data and text data.

[0121] For image data (let the image be 224*224 pixels), use the VGG (Visual Geometry Group Network) convolutional neural network pre-trained on ImageNet (visual database) to extract 512*7*7 image features from the 224*224 pixel image, and use the VGG convolutional neural network to extract high-level semantic features from the image features, and flatten the three-dimensional array into a one-dimensional image feature vector.

[0122] The image data is represented as:

[0123] ;

[0124] Image data The image feature vector is expressed as:

[0125] ;

[0126] in, represents the total number of image samples (i.e., image points), The first samples, is the image label, represents the image feature vector, Represents the VGG convolutional neural network.

[0127] For text data, the text is segmented and then the Bag-of-Words model is used to obtain text features, and the high-level semantic features (i.e., text feature vectors) in the text features are extracted through the text convolutional neural network.

[0128] The text data is represented as:

[0129] ;

[0130] Text Data The text feature vector is represented as:

[0131] ;

[0132] in, Represents the total number of text segmentation samples, The text samples, For text labels, represents the text feature vector, Represents the text convolutional neural network subnetwork.

[0133] Any image point sample and text word segmentation sample constitute a sample pair.

[0134] Step S920, performing semantic hash coding on the text feature vector and the image feature vector through a hash network.

[0135] In view of the problem that it is difficult to directly perform similarity comparison at this stage due to different modal data dimensions and file formats, this embodiment adopts a semantic hash learning method to perform hash encoding on the feature vectors corresponding to the image and text, and uses binary hash codes instead of the original image and text data for comparison.

[0136] The hash network consists of three fully connected layers, including input layer, hidden layer, and hash layer.

[0137] The input layer receives the image feature vector and the text feature vector.

[0138] The hidden layer uses the nonlinear activation function Sigmoid to increase the network's expressiveness and reduce the dimensionality of the high-dimensional feature vector.

[0139] The output of the hidden layer to the image feature vector is:

[0140] ;

[0141] The output of the hidden layer to the text feature vector is:

[0142] ;

[0143] in, is the output of the hidden layer of the hash network, and is the weight value of the fully connected layer, and is the bias of the fully connected layer.

[0144] The hash layer uses the Sign function to map continuous low-dimensional feature vectors into discrete binary hash codes to ensure that the output value is {1, -1} or {1, 0}. An image point sample or a text word segmentation sample corresponds to a string of binary hash codes.

[0145] The binary hash code output by the hash network is:

[0146] ;

[0147] The binary hash code output by the hash network is:

[0148] ;

[0149] in, , is the hash code output of the hash layer of the hash network, is a sign function, when > 0 returns 1, otherwise returns -1. (When completed, the hash code -1 is mapped to 0, and the final hash code consists of 0 and 1).

[0150] Subsequently, the similarity between data of different modalities is measured by comparing the distances between hash codes in the Hamming space to achieve cross-modal retrieval.

[0151] Step S930, weighted fusion of image and text feature vectors through a fusion network.

[0152] The fused feature vector is expressed as:

[0153] ;

[0154] in, is the fusion feature vector, is the weight of the image feature vector, is the weight of the text feature vector.

[0155] The dimension of the image feature vector is controlled at 512, while the dimension of the text feature vector varies according to the number of words in the bag of words. This leads to a significant difference in the length of the image feature vector and the text feature vector. The use of a simple concatenation and fusion mechanism may cause the weight of the fusion feature to shift. For example, a 512-dimensional image feature vector is concatenated with a 10,000-dimensional text feature vector to obtain a 10,512-dimensional fusion feature vector, in which most of the features come from the text feature vector.

[0156] In order to solve the problem of possible weight shift, this example uses a feature weighted fusion mechanism to assign a weight to each feature vector. First, the image and text feature vectors are normalized to ensure that the feature vectors have the same scale. Then the weight parameters are optimized through the gradient descent algorithm. Each normalized feature vector is multiplied by its corresponding weight, and the results are added to generate a fused feature vector.

[0157] The purpose of feature vector normalization is to ensure that the numerical range of each feature vector is comparable when merging or comparing different feature vectors, and to scale the numerical range of the feature vector to [0, 1] or [-1, 1]. This can prevent some features from having a disproportionate impact on model training due to their large numerical range.

[0158] Normalization:

[0159] ;

[0160] in, is the nth normalized element in a certain eigenvector, is the nth element in a certain eigenvector, is the smallest element in a certain eigenvector, The largest element in a particular eigenvector.

[0161] Gradient descent is an optimization algorithm used to minimize the loss function. First, initialize and , then for each weight parameter, calculate the gradient of the loss function with respect to the parameter, and finally update each weight parameter according to the gradient and learning rate.

[0162] Gradient calculation:

[0163] ;

[0164] in, for The weight, express about The gradient is a vector whose components are the partial derivatives of the function at that point along the corresponding coordinate axis.

[0165] Parameter update:

[0166] ;

[0167] in, is the learning rate, which controls the step size of parameter updates, To update the weight parameters, is the old weight parameter.

[0168] Step S940: obtaining an average value corresponding to the second similarity according to the relationship network.

[0169] The relational network specifically includes an input layer, an embedding layer, a relational subnetwork, an aggregation layer, and an output layer; wherein the input layer accepts the embedding of two or more entities as input, and the embodiment uses normalized images, texts, and fused feature vectors as input entities. The embedding layer maps the input entities to a shared embedding space to enable mutual comparison between entities. The relational subnetwork is the core of the relational network, which can calculate the potential semantic associations between entity pairs and output the relational similarity scores through a neural network. The aggregation layer and the output layer aggregate the relational information to generate the final similarity score matrix.

[0170] The simple form of the relationship network is a composite function:

[0171] ;

[0172] in, is a set of input samples, It is samples, , are two learning parameters , function.

[0173] According to the relationship network form, it is designed as follows:

[0174] ;

[0175] in, Represents a vector group consisting of any two vectors in the image, text, and fusion feature vector corresponding to any sample pair. For example, it is the image feature vector of the sample pair and text feature vector A vector group, or a text feature vector and fusion feature vector A vector group. ReLU is a nonlinear activation function. represents the concatenation of vectors, is the weight matrix, is the bias term.

[0176] This embodiment calculates the similarity score for each vector group:

[0177] (1) A vector group consisting of image and text feature vectors:

[0178] ;

[0179] (2) A vector group consisting of an image and a fused feature vector:

[0180] ;

[0181] (3) A vector group consisting of text and fused feature vectors:

[0182] ;

[0183] Get the vector group of each sample pair After the relationship, the fully connected layer The relationship is converted into a second similarity.

[0184] Convert to the second similarity:

[0185] ;

[0186] in, is the second similarity, represents any sample pair, is the weight Transpose the matrix, is the bias term.

[0187] The second similarity range is [0,1], and all the calculated second similarity scores are Put it into a matrix to form the score matrix of the second similarity , the score matrix of the second similarity The dimension is the number of graph feature vectors + the number of text feature vectors + the square of the number of fusion feature vectors. The three second similarities corresponding to each sample pair are: , , , take the average of the three to generate the average matrix of the second similarity , the mean value matrix The dimension is the number of graph feature vectors × the number of text feature vectors.

[0188] The score matrix of the second similarity:

[0189] ;

[0190] The average matrix of the second similarity is:

[0191] ;

[0192] in, The score matrix representing the second similarity, Represents the average value matrix of the second similarity, N, M, L represent the number of images, texts and fused feature vectors.

[0193] Step S950: loss function constrains hash code.

[0194] Here, a loss function is used to constrain the hash code generated in step S920.

[0195] The relationship network obtained The average value in The loss function is introduced to strengthen the connection between hash codes with strong semantic association. The distance between feature vectors is calculated using the Hamming space, and the loss function is used to make the Hamming distance between hash codes with greater similarity smaller to generate hash codes that meet the constraints. The optimization process is repeated until the loss function converges to obtain the trained hash retrieval model. The hash retrieval model can be used to obtain compact and strongly related hash codes using the hash network.

[0196] Hamming space distance calculation:

[0197] ;

[0198] in, , is the first hash code and the second hash code of a set of sample pairs, Represents the Hamming distance between hash codes.

[0199] The loss function is used to constrain the Hamming distance between similar hash codes to be smaller than the Hamming distance between dissimilar hash codes, expressed as:

[0200] ;

[0201] ;

[0202] in, is a set of similar sample pairs, is a set of dissimilar sample pairs, is the mean similarity score. To further improve the compactness of the hash code, a marginal , expressed as:

[0203] ;

[0204] in is the marginal value, which can effectively separate dissimilar samples.

[0205] The hash code of the hash network in step S920 is as the loss function for training the hash retrieval model.

[0206] Step S960, retrieval of the target image and target text is performed based on the trained hash retrieval model.

[0207] For example, the target text and the target image are input into the hash retrieval model, and a hash retrieval result between a first target hash code corresponding to the target text and a second target hash code corresponding to the target image output by the hash retrieval model is obtained.

[0208] This embodiment has at least the following beneficial effects:

[0209] The method first uses a feature extraction network to extract features from heterogeneous modal data such as images and texts, obtains image feature vectors and text feature vectors for each sample pair, and fuses the image feature vectors and text feature vectors for each sample pair to obtain a fused feature vector; then, the association relationship between different modal objects is mined from the image feature vector, the text feature vector and the fused feature vector through a relational network, the core commonality of the pairwise relationship between object samples is captured, and the second similarity between vector groups of different modal objects is calculated to represent a certain relationship representation between vector groups of different modal objects; then, the first similarity between the first hash code of the text feature vector and the second hash code of the image feature vector of each sample pair is calculated according to the hash network, and a loss function is constructed according to the second similarity to constrain the first similarity between similar hash codes in all sample pairs to be less than the first similarity between dissimilar hash codes. Finally, the hash network of the hash retrieval model is reversely optimized based on the loss function until the hash retrieval model after training is obtained. The present invention proposes to combine a relational network with a cross-modal hash retrieval method so that different modal data features form a relational similarity matrix to achieve more accurate cross-modal hash retrieval.

[0210] like Figure 4 As shown, an embodiment of the present application provides a cross-modal hash retrieval device based on a relational network, the device comprising:

[0211] The image acquisition module 1100 is used to acquire target text and target image;

[0212] The result retrieval module 1200 is used to input the target text and the target image into the hash retrieval model, and obtain the hash retrieval result between the first target hash code corresponding to the target text and the second target hash code corresponding to the target image output by the hash retrieval model, wherein the training process of the hash retrieval model includes:

[0213] The text feature vector and image feature vector of each sample pair are obtained according to the feature extraction network of the hash retrieval model; the sample pair is composed of any text word segmentation sample of the text to be trained and any image point sample of the image to be trained;

[0214] The text feature vector and the image feature vector of each sample pair are fused according to the fusion network of the hash retrieval model to obtain a fused feature vector of each sample pair;

[0215] Converting the text feature vector of each sample pair into a first hash code and the image feature vector of each sample pair into a second hash code according to the hash network of the hash retrieval model, and calculating a first similarity between the first hash code and the second hash code of each sample pair;

[0216] The second similarity of the vector group of each sample pair is calculated according to the relational network of the hash retrieval model; the vector group of each sample pair is composed of at least two of the text feature vector, the image feature vector and the fusion feature vector of each sample pair;

[0217] According to the second similarity of the vector group of each sample pair, a loss function is constructed to constrain the first similarity between similar hash codes in all sample pairs to be less than the first similarity between dissimilar hash codes, and the hash network of the hash retrieval model is reversely optimized based on the loss function until a trained hash retrieval model is obtained.

[0218] It should be noted that the cross-modal hash retrieval device based on relational network provided in this embodiment and the above-mentioned cross-modal hash retrieval method based on relational network are based on the same inventive concept, so the relevant content of the above-mentioned cross-modal hash retrieval method based on relational network is also applicable to the content of the cross-modal hash retrieval device based on relational network, and therefore, it will not be repeated here.

[0219] like Figure 5 The embodiment of the present application further provides an electronic device, the electronic device comprising:

[0220] at least one memory;

[0221] at least one processor;

[0222] at least one program;

[0223] The program is stored in the memory, and the processor executes at least one program to implement the cross-modal hash retrieval method based on the relational network described above.

[0224] The electronic device may be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a vehicle-mounted computer, etc.

[0225] The electronic device according to the embodiment of the present application is described in detail below.

[0226] The processor 1600 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0227] The memory 1700 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1700 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1700, and the processor 1600 calls and executes the cross-modal hash retrieval method based on a relational network in the embodiment of the present invention.

[0228] Input / output interface 1800, used to implement information input and output;

[0229] Communication interface 1900, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0230] Bus 2000 , which transmits information between various components of the device (e.g., processor 1600 , memory 1700 , input / output interface 1800 , and communication interface 1900 );

[0231] The processor 1600 , the memory 1700 , the input / output interface 1800 , and the communication interface 1900 are connected to each other in communication within the device via the bus 2000 .

[0232] An embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the above-mentioned cross-modal hash retrieval method based on a relational network.

[0233] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0234] The embodiments described in the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0235] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0236] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0237] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0238] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0239] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0240] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0241] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0242] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0243] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable an electronic device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0244] The above is a specific description of the preferred implementation of the embodiments of the present application, but the embodiments of the present application are not limited to the above-mentioned implementation methods. Technical personnel familiar with the field can also make various equivalent modifications or substitutions without violating the spirit of the embodiments of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the embodiments of the present application.

Claims

1. A cross-modal hash retrieval method based on a relational network, characterized in that: The method comprises: Get the target text and target image; The target text and the target image are input into a hash retrieval model to obtain a hash retrieval result between a first target hash code corresponding to the target text and a second target hash code corresponding to the target image output by the hash retrieval model, wherein the training process of the hash retrieval model includes: Acquire a text feature vector and an image feature vector of each sample pair according to the feature extraction network of the hash retrieval model; the sample pair is composed of any text word segmentation sample of the text to be trained and any image point sample of the image to be trained; fusing the text feature vector and the image feature vector of each sample pair according to the fusion network of the hash retrieval model to obtain a fused feature vector of each sample pair; Converting the text feature vector of each sample pair into a first hash code and the image feature vector of each sample pair into a second hash code according to the hash network of the hash retrieval model, and calculating a first similarity between the first hash code and the second hash code of each sample pair; Calculating the second similarity of the vector group of each sample pair according to the relational network of the hash retrieval model; the vector group of each sample pair is composed of at least two of the text feature vector, the image feature vector and the fusion feature vector of each sample pair; According to the second similarity of the vector group of each sample pair, construct a loss function for constraining the first similarity between similar hash codes in all sample pairs to be smaller than the first similarity between dissimilar hash codes, and reversely optimize the hash network of the hash retrieval model based on the loss function until the trained hash retrieval model is obtained; The calculating the second similarity of the vector group of each sample pair includes: ； ； in, The feature vector and the eigenvector The concatenation vector of the vector group, the feature vector and the eigenvector is one of the text feature vector, the image feature vector and the fusion feature vector, and the feature vector With the feature vector are not eigenvectors of the same mode; is the first bias term, is the first weight matrix, is a nonlinear activation function, is the second bias term, is the second weight matrix, for The transposed matrix of is the Sigmoid activation function, The feature vector and the eigenvector The second similarity of the composed vector group; The loss function includes: ； ； ； in, is a sample pair, is a set of similar sample pairs, is a set of dissimilar sample pairs, For sample pairs The average value of For sample pairs The first similarity of is the loss function, is the preset boundary value, The maximum value function, is the first intermediate loss function, is the second intermediate loss function, and are the preset weight values ​​respectively.

2. The cross-modal hash retrieval method based on relational network according to claim 1 is characterized in that: Before constructing a loss function for constraining the first similarity between similar hash codes in all sample pairs to be smaller than the first similarity between dissimilar hash codes according to the second similarity of the vector group of each sample pair, the method further includes: Calculate a first second similarity of a vector group consisting of a text feature vector and an image feature vector of each sample pair, a second second similarity of a vector group consisting of an image feature vector and a fused feature vector of each sample pair, and a third second similarity of a vector group consisting of a text feature vector and a fused feature vector of each sample pair; Calculate an average value among the first second similarity, the second second similarity and the third second similarity of each sample pair; The step of constructing, according to the second similarity of the vector group of each sample pair, a loss function for constraining the first similarity between similar hash codes in all sample pairs to be smaller than the first similarity between dissimilar hash codes, comprises: According to the average value of each sample pair, a loss function is constructed for constraining the first similarities between similar hash codes in all sample pairs to be smaller than the first similarities between dissimilar hash codes.

3. The cross-modal hash retrieval method based on relational network according to claim 2 is characterized in that: The loss function includes: ； in, is a sample pair, is a set of similar sample pairs, is a set of dissimilar sample pairs, For sample pairs The average value of For sample pairs The first similarity of is the loss function.

4. The cross-modal hash retrieval method based on relational network according to claim 1 is characterized in that: The fusing of the text feature vector and the image feature vector of each sample pair includes: ； in, , The sample pairs The text feature vector and image feature vector of For sample pairs The fusion feature vector of and are weight values ​​respectively, and the weight values and The calculation process includes: Construct two initial weight values; Pair the samples Normalizing the text feature vector and the image feature vector to obtain a normalized text feature vector and an image feature vector; According to the normalized text feature vector and image feature vector, the two initial weight values ​​are optimized by using the gradient descent method until the weight value is obtained. and .

5. The cross-modal hash retrieval method based on relational network according to claim 1 is characterized in that: The first similarity of the sample pairs is the Hamming distance between the first hash code and the second hash code of each sample pair.

6. A cross-modal hash retrieval device based on a relational network, characterized in that: The device comprises: An image acquisition module, used to acquire target text and target image; The result retrieval module is used to input the target text and the target image into a hash retrieval model to obtain a hash retrieval result between a first target hash code corresponding to the target text and a second target hash code corresponding to the target image output by the hash retrieval model, wherein the training process of the hash retrieval model includes: Acquire a text feature vector and an image feature vector of each sample pair according to the feature extraction network of the hash retrieval model; the sample pair is composed of any text word segmentation sample of the text to be trained and any image point sample of the image to be trained; fusing the text feature vector and the image feature vector of each sample pair according to the fusion network of the hash retrieval model to obtain a fused feature vector of each sample pair; Converting the text feature vector of each sample pair into a first hash code and the image feature vector of each sample pair into a second hash code according to the hash network of the hash retrieval model, and calculating a first similarity between the first hash code and the second hash code of each sample pair; Calculating the second similarity of the vector group of each sample pair according to the relational network of the hash retrieval model; the vector group of each sample pair is composed of at least two of the text feature vector, the image feature vector and the fusion feature vector of each sample pair; According to the second similarity of the vector group of each sample pair, construct a loss function for constraining the first similarity between similar hash codes in all sample pairs to be smaller than the first similarity between dissimilar hash codes, and reversely optimize the hash network of the hash retrieval model based on the loss function until the trained hash retrieval model is obtained; The calculating the second similarity of the vector group of each sample pair includes: ； ； in, The feature vector and the eigenvector The concatenation vector of the vector group, the feature vector and the eigenvector is one of the text feature vector, the image feature vector and the fusion feature vector, and the feature vector With the feature vector are not eigenvectors of the same mode; is the first bias term, is the first weight matrix, is a nonlinear activation function, is the second bias term, is the second weight matrix, for The transposed matrix of is the Sigmoid activation function, The feature vector and the eigenvector The second similarity of the composed vector group; The loss function includes: ； ； ； in, is a sample pair, is a set of similar sample pairs, is a set of dissimilar sample pairs, For sample pairs The average value of For sample pairs The first similarity of is the loss function, is the preset boundary value, The maximum value function, is the first intermediate loss function, is the second intermediate loss function, and are the preset weight values ​​respectively.

7. An electronic device, characterized in that: It includes at least one control processor and a memory for communicating with the at least one control processor; the memory stores instructions that can be executed by the at least one control processor, and the instructions are executed by the at least one control processor so that the at least one control processor can execute the cross-modal hash retrieval method based on the relational network as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the cross-modal hash retrieval method based on a relational network as described in any one of claims 1 to 5.

Images