Cross-modal hash retrieval method based on graph convolutional network fusion

Abstract

The invention relates to a cross-modal hash retrieval method based on graph convolutional network fusion, and belongs to the technical field of multimedia retrieval. According to the method, a modal fusion module based on a graph convolutional neural network is constructed, modal sharing expressions are learned, the sharing expressions combine pairwise losses and intra-modal and inter-modal identification losses to reduce inter-modal heterogeneity, the similarity relation between modals is fully learned, and feature alignment between different modal data is enhanced; according to the method, an effective learning target, namely a loss function, is constructed by adopting pairwise loss, triple loss and quantization loss functions, the high-order semantic similarity of the method among different modes is captured, the learning of the method is guided, and the learning ability of the method for the relationship between data is further enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimedia retrieval, and in particular relates to a cross-modal hash retrieval method based on graph convolutional network fusion. Background Art

[0002] With the increase in the amount of multimedia data, multimedia retrieval becomes more and more important. How to effectively retrieve the required information from the massive multimedia data based on the user input information is a key task. Multimedia retrieval covers two main forms: unimodal retrieval and cross-modal retrieval. The task of cross-modal retrieval is to return another modal data (such as text, image) semantically related to the query data of one modality (such as image, text). The cross-modal retrieval method is to map multimodal data (such as image, text, etc.) into a common space and retain the similarity relationship between the original data in the common space. Since the similarity relationship of the original samples is well preserved in the subspace, the original data corresponding to the data with high similarity in the subspace also has high similarity. Therefore, in the process of cross-modal retrieval, it is only necessary to retrieve similar samples according to the feature similarity in the subspace. Among them, the cross-modal retrieval method based on hashing technology can achieve faster query speed and lower storage cost when facing large-scale graphic and text data through XOR operation and 01 sequence storage, which has attracted the attention of many researchers.

[0003] Hashing technology is a commonly used nearest neighbor search technology. It uses the learned hash function to map high-dimensional raw data into binary hash codes and retain the proximity relationship between the raw data as much as possible, thereby achieving the purpose of data compression and fast search. The cross-modal retrieval method based on hashing technology projects the data from different modalities into a low-dimensional common Hamming space (limited to a binary space of {0, 1} or {-1, +1}) through the encoder and hash layer of each modality, and uses the hash code of the original data as a new feature for retrieval. Since hash codes can be directly stored using memory addresses and similarity can be calculated using XOR operations, cross-modal hash retrieval is suitable for fast retrieval of large-scale multimodal data.

[0004] Depending on whether the sample data has artificially labeled labels, cross-modal hash retrieval methods are divided into supervised cross-modal hashing and unsupervised cross-modal hashing. Supervised cross-modal hashing guides the generation of hash codes by mining the semantic information in the labels. Unsupervised cross-modal hashing learns hash functions to generate hash codes by mining the distribution of raw data. The present invention is mainly aimed at existing supervised cross-modal hashing retrieval methods.

[0005] Supervised methods use the label information of sample data to maintain the original semantic similarity of the data and improve the accuracy of hash retrieval. According to the different technologies adopted, supervised cross-modal hash retrieval methods can be divided into the following four types: semantic relevance maximization, class label similarity matrix, asymmetric hash learning, and joint semantic embedding hashing. The semantic relevance maximization method is committed to learning hash functions to maximize the semantic relevance between cross-modal data. One of the representative methods is Deep Visual Semantic Hashing (DVSH). Deep Visual Semantic Hashing integrates multimodal embedding and cross-modal hash learning processes by introducing a structured maximum objective. DVSH can learn the intrinsic cross-modal correlation between images and texts and generate compact hash codes. The class label similarity matrix method uses the class label similarity matrix as supervised information to learn the hash function. The innovation of Deep Cross-Modal Hashing (DCMH) is that it optimizes feature extraction and hash code generation at the same time, thereby improving the quality of hash codes and the performance of cross-modal retrieval. Asymmetric hash learning methods take into account the differences between different modal data and process data of different modalities in an asymmetric way. For example, Deep Asymmetric Supervised Hashing (DASH) is a representative work. DASH solves the differences between different modal data and the distinguishability of hash codes through asymmetric processing and discrete optimization strategies, and achieves more accurate and efficient supervised cross-modal retrieval. Supervised cross-modal hash retrieval based on joint semantic embedding hashing improves the performance of hash retrieval by jointly learning semantic embedding and hash functions. Deep Joint Semantic Embedding Hash (DJSEH) is an example, which makes full use of the powerful feature representation learning ability of deep neural networks. The core idea is to jointly learn semantic associations and hash codes through an end-to-end network structure, thereby ensuring that data of different modalities in the hash space can maintain their original semantic information. In addition to the previously mentioned methods such as DCMH, DASH and DJSEH, there are some typical deep cross-modal hashing methods that deserve attention. Such as: Triple Fusion Network Hashing (TFNH), Sort Based Deep CrossModal Hashing (SBDCMH), etc.

[0006] As mentioned above, in recent years, deep learning technology has become a key driving force for major breakthroughs in many fields. Its powerful feature extraction capabilities make complex data processing tasks more efficient and accurate. In this context, cross-modal hash retrieval methods have also been influenced by deep learning and have shown great potential and prospects. Supervised cross-modal hashing methods have become one of the current hot topics in cross-modal hashing retrieval research because they can use label information to learn more compact hash codes in the common Hamming space. However, existing supervised methods still face several major problems: (1) Multimodal features naturally have a large semantic gap. Existing methods do not fully exploit the supervised information provided by labels to achieve interaction between modalities, and fail to effectively reduce the modal gap to achieve high-quality modal alignment; (2) Existing methods often lack effective learning objectives, resulting in insufficient learning of existing methods and the retrieval performance of generated hash codes failing to meet expectations. Summary of the invention

[0007] 1. Technical issues to be resolved

[0008] The technical problem to be solved by the present invention is to provide a cross-modal hash retrieval method for mining the supervision information provided by labels to achieve interaction between modalities and high-quality modality alignment, thereby improving the cross-modal hash retrieval performance in supervised scenarios.

[0009] (II) Technical solution

[0010] In order to solve the above technical problems, the present invention provides a cross-modal hash retrieval method based on graph convolutional network fusion, comprising the following steps:

[0011] (1) Extracting the original semantic features of the image modality; for the image modality, the pre-trained model ResNet50 is used to extract the original semantic features of the image from the original image, namely, the image modality features. The image modality features extracted by ResNet50 are defined as F v ;

[0012] (2) Extracting the original semantic features of the text modality; for the text modality, first convert the text sample into a bag-of-words vector; then, input the text information represented by the bag-of-words vector into a multi-layer perceptron to extract the original semantic features in the text sample, namely, the text modality features. The extracted text modality features are defined as F t ;

[0013] (3) Obtain unified embedding features of image modality and text modality; use three fully connected layers of the multi-layer perceptron to extract the image modality features F v and text modality feature F t Perform further feature mapping to obtain F v and F tThe corresponding unified embedding feature representation Z v and Z t ;

[0014] (4) Based on Z v and Z t , generate binary hash codes for the image and text modalities in the training samples, respectively, and generate binary hash codes for the image and text modalities in the test samples, respectively;

[0015] (5) A fusion module based on graph convolutional neural network is introduced to learn the shared representation of image and text modalities. The specific fusion process is as follows: First, the adjacency matrix A is used to guide the graph convolutional neural network to aggregate the information of each node and learn more discriminative feature representations; then, the image modality feature F v and text modality feature F t Perform splicing to obtain the modal shared representation R = [F v ; F t ]; Finally, construct the graph structure (R, A), and obtain the total similarity matrix U, the label-based similarity matrix S, and the output Z of the graph convolutional neural network S ;

[0016] (6) constructing a loss function based on step (5), wherein three losses are used to guide the learning process, namely, triplet loss, pairwise loss and quantization loss;

[0017] (7) Based on step (6), the total loss function of the entire process of training the graph convolutional neural network is calculated, and the graph convolutional neural network is trained based on the loss function. During the training process, back propagation is used to iteratively update the parameters of the graph convolutional neural network, that is, the parameters of the loss function, until the graph convolutional neural network converges, and then the next step is executed;

[0018] (8) Based on the binary hash codes of the training samples and test samples generated in step (4), the Hamming distance between the retrieved samples in the text or image modality and the retrieved samples in the image or text modality is calculated, and the samples are sorted according to the size of the Hamming distance. Then, the first k samples corresponding to the minimum distance in the retrieval samples of the text or image modality are taken as the retrieval results, so that semantically similar retrieval results can be obtained between different modalities.

[0019] (III) Beneficial effects

[0020] (1) In the present invention, a modality fusion module based on graph convolutional neural network is constructed to learn modality shared representations. These shared representations will combine pairwise loss and intra-modality and inter-modality discrimination losses to reduce the heterogeneity between modalities, fully learn the similarity relationship between modalities, and enhance the feature alignment between different modality data;

[0021] (2) Pairwise loss, triplet loss and quantization loss functions are used to construct an effective learning objective, i.e., a loss function, to capture the high-order semantic similarity between different modalities of the method, guide the learning of the hash method of the present invention, and further enhance the learning ability of the method of the present invention for the relationship between data. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The figure is a schematic diagram of the overall implementation principle of the method of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0024] The present invention proposes a cross-modal hash retrieval method based on graph convolutional network fusion, which is used to mine the supervision information provided by labels to realize the interaction between modalities and high-quality modality alignment, thereby improving the cross-modal hash retrieval performance in supervised scenarios.

[0025] The core idea of ​​this method is to construct a graph structure containing multiple modal information, where nodes represent data samples of each modality and edges reflect the similarity relationship between samples; the introduction of graph convolutional networks makes full use of the information in the graph structure to achieve information transmission and fusion between different modal data; at the same time, pairwise loss, triplet loss and quantization loss are used to construct loss functions to capture high-order semantic similarities between different modalities and guide the generation of hash codes. The main contents of this invention are as follows:

[0026] (1) A modality fusion method based on graph convolutional neural networks is proposed to learn modality shared representations. These shared representations combine pairwise loss and intra-modality and inter-modality discrimination losses to reduce the heterogeneity between modalities, fully learn the similarity relationship between modalities, and enhance the feature alignment between different modal data.

[0027] (2) A new loss function construction method is proposed, which uses pairwise loss, triplet loss and quantization loss functions to construct an effective learning objective, namely the loss function, to capture the high-order semantic similarities between different modalities and further enhance the learning ability of the method of the present invention for the relationship between data.

[0028] The present invention mainly includes three parts: a deep feature extraction module, a GCN-based fusion module and a loss function module. The deep feature extraction module uses image and text encoders to extract original semantic features; the GCN-based fusion module uses graph convolutional neural networks to learn modality sharing representations, thereby reducing heterogeneity between modalities. The method uses pairwise loss, quantization loss and triple loss to construct a loss function and optimize the network method of the present invention.

[0029] Figure 1 This is the workflow diagram of the unsupervised cross-modal hash retrieval method based on the attention fusion mechanism proposed in the present invention. The present invention uses deep learning and hashing methods and other related theoretical knowledge as basic principles to make a specific analysis of image and text data modalities. In the training phase, the image modality and text modality data are respectively extracted through image-text encoders to obtain high-dimensional semantic feature vectors of the original sample data. The image modality uses an encoder based on a deep neural network, while the text modality uses the bag-of-words method of the present invention and a multi-layer perceptron. Definition F v and F t are the output high-dimensional features of the image and text modalities, respectively. Then, the multi-modal features are connected to obtain the modal shared representation R, and the adjacency matrix between batches of data is calculated through R. The graph structure with the adjacency matrix as the edge and the shared representation as the node will be sent to the graph convolutional neural network to learn the structural similarity information between data and perform feature fusion between modalities. At the same time, the high-dimensional features will also be mapped into hash codes B by the hash layers of each modality. v and B t , corresponding to image hash code and text hash code respectively. Finally, the fusion features of the hash code and the output of the graph neural network will be used to calculate the loss function to optimize the parameters of the method of the present invention. The method of the present invention sets a learning goal based on pairwise loss, triplet loss and quantization loss to optimize the method of the present invention, so that the obtained hash code has sufficient distinguishability and more accurately reflects the content semantic information of the original data.

[0030] In the test phase, the graph convolutional network fusion process and the loss function optimization process will be discarded, and the method of the present invention only needs to pass through the encoder and hash layer to directly output high-quality hash codes. In the test phase, the method of the present invention does not need to be updated and learned, because the hash code generation efficiency in the test phase is sufficient to meet the high efficiency requirements of cross-modal retrieval.

[0031] The present invention provides a cross-modal hash retrieval method based on graph convolutional network fusion, which specifically includes the following steps:

[0032] First, the derivation process of the Hamming distance calculation formula is given: given n training instances containing two modalities: image and text o i =(V i ,T i ), where V i represents the image modality data of the ith instance, T i represents the text modal data of the i-th instance; the label matrix is Among them l i represents the label vector of the i-th instance, and c represents the number of labels. The goal of this method is to learn hash codes for both image and text modalities, and define the hash code for the image modality as Bv ={-1,+1} K , the hash code of the text mode is B t ={-1,+1} K , where K represents the length of the hash code, and the number of mini-batches divided into each training is m; the Hamming distance is used to measure the distance between binary hash codes, which is the number of different characters in strings of equal length; given a binary hash code h i and h j , the Hamming distance calculation formula is as follows:

[0033]

[0034] Where b is the hash code length, ⊙ is h i and h j Cross-modal hashing maps data of different modalities into the Hamming space, retaining the semantic similarity of the original data, improving retrieval speed and reducing storage consumption.

[0035] Then follow these steps:

[0036] (1) Extracting the original semantic features of the image modality; for the image modality, the pre-trained model ResNet50 is used to extract the original semantic features of the image from the original image, namely, the image modality features. The image modality features extracted by ResNet50 are defined as F v ;

[0037] (2) Extracting the original semantic features of the text modality; for the text modality, first convert the text sample into a bag-of-words vector; then, input the text information represented by the bag-of-words vector into a multi-layer perceptron to extract the original semantic features in the text sample, namely, the text modality features. The extracted text modality features are defined as F t ;

[0038] (3) Obtain unified embedding features of image modality and text modality; use three fully connected layers (FC layers) of the multi-layer perceptron to embed the extracted image modality features F v and text modality feature F t Perform further feature mapping to obtain F v and F t The corresponding unified embedding feature representation Z v and Z t ;

[0039] (4) Based on Z v and Z t, generate binary hash codes (i.e., binary hash codes) for the image and text modalities in the training sample (retrieval sample), and generate binary hash codes (i.e., binary hash codes) for the image and text modalities in the test sample (retrieved sample);

[0040] (5) A fusion module based on graph convolutional neural network is introduced to learn the shared representation of image and text modalities. The specific fusion process is as follows: First, the adjacency matrix A is used to guide the graph convolutional neural network to aggregate the information of each node and learn more discriminative feature representations; then, the image modality feature F v and text modality feature F t Perform splicing to obtain the modal shared representation R = [F v ; F t ]; Finally, construct the graph structure (R, A), and obtain the total similarity matrix U, the label-based similarity matrix S, and the output Z of the graph convolutional neural network S ;

[0041] (6) Based on step (5), a loss function of the method of the present invention is constructed to capture high-order semantic similarities between different modalities, and further enhance the learning ability of the present invention for the relationship between data; the present invention uses three losses to guide the learning process of the method of the present invention, namely, triple loss, pair loss and quantization loss;

[0042] (7) Based on step (6), the total loss function of the entire process of training the graph convolutional neural network is calculated. Based on the loss function, the graph convolutional neural network is trained, and during the training process, back propagation is used to iteratively update the parameters of the graph convolutional neural network (the parameters of the above loss function) until the graph convolutional neural network converges, and then the next step is executed; when the entire graph convolutional neural network tends to converge, the hash code reconstruction phase for the image modality and text modality retrieval samples and the retrieved samples will also end;

[0043] (8) According to formula (1), the Hamming distance from the retrieved sample in the text (image) modality to the retrieval sample in the image (text) modality is calculated based on the binary hash codes of the training samples and test samples generated in step (4), and the samples are sorted according to the size of the Hamming distance. Then, the first k samples corresponding to the minimum distance in the text (or image) modality retrieval samples are taken as the retrieval results, and semantically similar retrieval results can be obtained between different modalities.

[0044] Among them, in step (1), for the image modality, the present invention uses a pre-trained ResNet50 to extract image modality features. ResNet50 gradually extracts deep feature information from the original image through a series of convolution and pooling operations to obtain a 2048-dimensional image feature vector. The extracted image feature vector is input into a fully connected layer containing 512 nodes to obtain a 512-dimensional image feature vector, i.e., the image modality feature. This 512-dimensional image feature vector serves as a high-dimensional feature representation of the image, capturing the key information in the image sample, and is used for subsequent hash function learning.

[0045] In step (2), for text modal data, the text sample is usually converted into a bag-of-words vector. Then, the text information represented by the bag-of-words vector is input into a multilayer perceptron containing three fully connected layers (FC layers) for processing; the multilayer perceptron is a feedforward neural network that performs nonlinear transformations on the input data through multiple fully connected layers (also called dense layers or linear layers); during the training process, the number of nodes in the last fully connected layer of the multilayer perceptron is set to 512, which means that after being processed by the multilayer perceptron, the original bag-of-words vector is converted into a 512-dimensional text feature vector; this 512-dimensional text feature vector, as a high-dimensional feature representation of the text, captures the key information in the text sample and is used for subsequent hash function learning;

[0046] We use V and T to represent the training samples of images and texts, respectively, and θ v ,θ t Indicates the parameters of image and text feature encoding, Encv and Enct are used to represent the image encoder and text encoder respectively. are the output high-dimensional features of the image modality and text modality, respectively, where d v and d t Represent the image and text feature dimensions respectively; the image modality feature and text modality feature are defined as shown in formula (2):

[0047] F v =Encv(V,θ v ); F t =Enct(T,θ t ) (2).

[0048] In step (3), three fully connected layers (fc layers) are used to further map the extracted image modality features and text modality features to obtain a unified embedded feature representation Z v and Z t ,Right now F * 2 =FC * 2 (F *1 ), Z * =FC * 3 (F * 2 ), F * 1 Represents the feature map obtained after the image and text modality features pass through the first fully connected layer FC; FC * 1 represents the first fully connected layer; F * 2 It indicates that the feature map obtained after the image and text modality features pass through the first fully connected layer FC is input into the feature map obtained after the second fully connected layer; FC * 2 represents the second fully connected layer; F * 3 It indicates that the feature map obtained after the image and text modality features pass through the second fully connected layer FC is input into the feature map obtained after the third fully connected layer; FC * 3 represents the third fully connected layer, *∈{v,t}.

[0049] In step (4), Z is converted by the sign function sign(x) (when x ≥ 0, sign(x) = 1, otherwise sign(x) = -1) v and Z t Binarization can be performed to obtain the binary image and text hash code B v and B t :

[0050] B * =sign(Z * ),*∈{v,t} (3)

[0051] In order to fully explore the semantic structure information of the data and fuse the features from different modalities to achieve semantic alignment in step (5), the present invention adopts a fusion module based on a graph convolutional neural network. In this fusion module, a graph convolutional neural network is used to learn modality sharing representations. Specifically, the adjacency matrix A is used to guide the graph convolutional neural network to aggregate the information of each node and learn more discriminative feature representations. Therefore, how to construct the adjacency matrix A is very important. The construction method is as follows: First, the cosine distance is used to quantify the similarity between instances, that is, J ij =(R i ) T R j , J ij (indicates cosine distance) The larger it is, the more characteristic R i With R jThe greater the similarity between them, the more similar they are. At the same time, the Euclidean distance is introduced to calculate the distance between features and construct the similarity matrix M. ij :

[0052]

[0053] Among them, ρ is the quantization parameter, which is set to 4; the two similarity matrices J are combined ij and M ij Construct the total similarity matrix U:

[0054] U=J ij *M ij (5)

[0055] Finally, we get the adjacency matrix A:

[0056] A ij =U ij *S ij (6)

[0057] Among them, A ij represents the element in the i-th row and j-th column of matrix A, S ij represents the element in the i-th row and j-th column of the matrix S, S ij =1 indicates feature R i With R j There is at least one identical tag and the semantics are similar, otherwise S ij =0,U ij Represents the element in the i-th row and j-th column of the total similarity matrix U.

[0058] In summary, the shared feature R represents the node information in the graph structure, and the adjacency matrix A obtained from the total similarity matrix U and the label-based similarity matrix S represents the mutual connection between nodes. The constructed graph structure (R, A) is obtained, and the total similarity matrix U and the label-based similarity matrix S are obtained. After obtaining the adjacency matrix A, the output of the graph convolutional neural network can be calculated. The definition of each layer of the graph convolutional neural network is as shown in formula (7):

[0059]

[0060] D is the degree matrix of the adjacency matrix A D ii are the elements on the diagonal of matrix D. All elements except the diagonal in D are 0. I H represents the trainable parameters of layer I, I = 1, 2, 3; 0 Represents the input of the graph convolutional neural network, H 2 Represents the output Z of the graph convolutional neural network S =H 2 .

[0061] In order to fully optimize the method and capture the accurate relationship between data in step (6), the present invention uses three losses to guide the learning process of the method of the present invention, namely triple loss, pair loss and quantization loss. In the triple loss function, each triple is composed of a positive sample, a negative sample and an anchor sample, which can be expressed as in Indicates that v i There is the same semantic information, which is the jth positive sample, v i represents the i-th anchor point sample, Indicates that v i There is different semantic information, which is the kth negative sample, i, j, and k are not equal. The triple loss function is defined as follows:

[0062]

[0063] Among them, D[f(x i ),f(x j )]=||f(x i )-f(x j )|| 2 ; f(λ) is the feature vector representation of the sample; α is a hyperparameter set artificially; [·] + It means that when the internal value is greater than zero, the result is the original value, and when the internal value is less than zero, the result is zero. According to the definition of triple loss function, the intra-modal loss function L is constructed. inner :

[0064]

[0065] Among them, v represents the image sample features; t represents the text sample features; S represents the sample features after fusion of the graph convolutional neural network; It represents the unified embedded feature representation obtained after the i-th anchor image sample passes through three layers of fully connected layer feature mapping; It represents the unified embedded feature representation of the j-th negative image sample after being mapped by three layers of fully connected layers; It represents the unified embedded feature representation of the kth positive image sample after three layers of fully connected layer feature mapping; It represents the unified embedded feature representation obtained after the i-th anchor text sample passes through three layers of fully connected layer feature mapping; It represents the unified embedded feature representation obtained after the j-th negative example text sample passes through three layers of fully connected layer feature mapping;

[0066] It represents the unified embedded feature representation of the kth positive text sample after three layers of fully connected layer feature mapping; Represents the output of the i-th anchor image and text fusion sample features after passing through the graph convolutional neural network; Represents the output of j negative example images and text fusion sample features after passing through the graph convolutional neural network; Represents the output of the sample features after the fusion of the k-th positive image and text passes through the graph convolutional neural network.

[0067] In order to focus on the changes between modalities, the inter-modal triplet loss is constructed at the same time. The inter-modal loss function L is constructed according to the definition of the triplet loss function. intra :

[0068]

[0069] In addition to the above two triplet loss functions describing intra-modality and inter-modality, the present invention defines the following pairwise loss and quantization loss. Pairwise loss is used to reduce Z v With Z S , Z t With Z S The difference between:

[0070]

[0071] Since there will be quantization errors when generating binary codes, the following quantization loss function is defined to reduce the quantization loss between hash codes and real values:

[0072]

[0073] Step (7) During the training process, back propagation is used to iteratively update the parameters used in the entire method process. When the entire network converges, the hash reconstruction phase will also end. The total loss function during the entire training process is:

[0074] L TOTAL =L inner +αL intra +βL pair +λL qu (13)

[0075] Among them, the hyperparameters α, β, and λ are weight parameters responsible for balancing the importance of each loss to prevent the method of the present invention from focusing on only part of the optimization objectives. In the iterative process, minimizing the loss function can make data with semantic similarity generate more consistent hash codes, while the Hamming distance of the hash codes of dissimilar data is larger.

[0076] In step (8), the Hamming distance from the retrieved sample in the image modality to the retrieved sample in the text modality is calculated based on the generated binary hash code (Formula (1)); the Hamming distance from the retrieved sample in the text modality to the retrieved sample in the image modality is calculated using the same steps. Next, the calculated Hamming distances are sorted in ascending order, and then the first k samples with the minimum distance in the retrieved sample in the text (or image) modality are taken and returned as the retrieval results.

[0077] The relevant experiments implemented for the present invention are mainly verified by the MIRFlickr25k cross-modal image and text dataset. MIRFlickr25k contains 25,000 image-text pairs collected from the Flickr website. Each image-text pair has multiple text tags associated with it. These tags describe the content of the image, and these tags belong to 20 different categories. The present invention uses 20,015 pairs of image-text pairs for experiments.

[0078] The present invention uses the Mean of Average Precision (MAP) to evaluate the method of the present invention. The Mean of Average Precision (MAP) is an important indicator to measure the global performance of the retrieval method. The higher the MAP value, the better the overall performance of the retrieval method. Because a high MAP value indicates that the retrieval method can maintain a high accuracy in most queries, thereby providing users with more accurate and reliable retrieval results. The hash code of the query instance is compared and sorted with the hash codes of all samples in the database, and the top R retrieval result instances with the minimum Hamming distance are returned. The definition of AP is as follows:

[0079]

[0080] Specifically, L represents the number of samples in the search set that are related to the query sample; P(r) represents the accuracy of the first r samples in the search results; δ(r) refers to an indicator. If a specific case is related to the searched case, then δ(r) is equal to 1, otherwise δ(r) is equal to 0. Then, the AP is averaged to get MAP:

[0081]

[0082] Where Q represents the number of query samples. The larger the MAP value, the better the retrieval performance of the method of the present invention.

[0083] The proposed method sets the banch-size to 32 and uses the Adam optimizer with momentum set to 0.8, weight decay set to 5e-4, and learning rate set to 0.01.

[58] To continuously update the network parameters of images and texts to optimize the method. Set α = 10, β = 10, λ = 0.01 on the MIRFlickr25k dataset:

[0084] The comparison methods of the present invention are: inter-media hashing IMH, cross-view hashing CVH, set matrix factorization hashing CMFH, deep cross-modal hashing DCMH, self-supervised adversarial hashing SSAH, and attentive deep adversarial hashing ADAH. Table 1 shows the MAP values ​​of the present invention and the comparison methods when performing cross-modal hash retrieval on the MIRFlickr25k dataset.

[0085] It can be concluded from Table 1 that no matter what the length of the hash code is, the deep cross-modal hashing method is generally superior to the traditional cross-modal hashing retrieval method. Compared with CMFH, which performs best in the traditional cross-modal hashing retrieval method, the MAP value of the present invention for 16, 32 and 64 bit code lengths in the image query text task is at least 15.7% higher; in the text query image task, the MAP value of the present invention is at least 14.9% higher. This is because compared with the traditional cross-modal hashing method, the present method uses a deep neural network to extract more semantic information of the original data. When compared with ADAH, which has the best performance among the deep cross-modal hashing methods, the MAP value of the present invention on both retrieval tasks is at least 2.07% higher. Even though the deep neural network is used to extract features like ADAH, the graph convolution fusion module used in the present method aggregates the neighborhood information and utilizes more deep semantic information than ADAH. In summary, the method of the present invention has achieved a better MAP value on the MIRFlickr25k dataset, whether compared with the traditional cross-modal hashing method or the deep cross-modal hashing method. The superiority of this method in handling cross-modal retrieval tasks is demonstrated.

[0086] Table 1 shows the comparison of MAP values ​​of various methods on the MIRFlickr25k dataset.

[0087] Table 1 Comparison of MAP values ​​of MIRFlickr25k dataset

[0088]

[0089] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A cross-modal hash retrieval method based on graph convolutional network fusion, characterized in that: The following steps are involved: (1) Extracting the original semantic features of the image modality; for the image modality, the pre-trained model ResNet50 is used to extract the original semantic features of the image from the original image, namely, the image modality features. The image modality features extracted by ResNet50 are defined as F v ; (2) Extracting the original semantic features of the text modality; for the text modality, first convert the text sample into a bag-of-words vector; then, input the text information represented by the bag-of-words vector into a multi-layer perceptron to extract the original semantic features in the text sample, namely, the text modality features. The extracted text modality features are defined as F t ; (3) Obtain unified embedding features of image modality and text modality; use three fully connected layers of the multi-layer perceptron to extract the image modality features F v and text modality feature F t Perform further feature mapping to obtain F v and F t The corresponding unified embedding feature representation Z v and Z t ; (4) Based on Z v and Z t , generate binary hash codes for the image and text modalities in the training samples, and generate binary hash codes for the image and text modalities in the test samples; (5) A fusion module based on graph convolutional neural network is introduced to learn the shared representation of image and text modalities. The specific fusion process is as follows: First, the adjacency matrix A is used to guide the graph convolutional neural network to aggregate the information of each node and learn more discriminative feature representations; then, the image modality feature F v and text modality feature F t Perform splicing to obtain the modal shared representation R = [F v ; F t ]; Finally, construct the graph structure (R, A), and obtain the total similarity matrix U, the label-based similarity matrix S, and the output Z of the graph convolutional neural network S ; (6) constructing a loss function based on step (5), wherein three losses are used to guide the learning process, namely, triplet loss, pairwise loss and quantization loss; (7) Based on step (6), the total loss function of the entire process of training the graph convolutional neural network is calculated, and the graph convolutional neural network is trained based on the loss function. During the training process, back propagation is used to iteratively update the parameters of the graph convolutional neural network, that is, the parameters of the loss function, until the graph convolutional neural network converges, and then the next step is executed; (8) Based on the binary hash codes of the training samples and test samples generated in step (4), the Hamming distance between the retrieved samples in the text or image modality and the retrieved samples in the image or text modality is calculated, and the samples are sorted according to the size of the Hamming distance. Then, the first k samples corresponding to the minimum distance in the retrieval samples of the text or image modality are taken as the retrieval results, so that semantically similar retrieval results can be obtained between different modalities.

2. The method according to claim 1, characterized in that In step (1), ResNet50 gradually extracts deep feature information from the original image through a series of convolution and pooling operations to obtain a 2048-dimensional image feature vector. The extracted image feature vector is input into a fully connected layer containing 512 nodes to obtain a 512-dimensional image feature vector, i.e., image modal feature. This 512-dimensional image feature vector is used as a high-dimensional feature representation of the image for subsequent learning of the hash function.

3. The method according to claim 2, characterized in that In step (2), the text information represented by the bag-of-words vector is input into a multilayer perceptron containing three fully connected layers for processing; during the training process, the number of nodes in the last fully connected layer of the multilayer perceptron is set to 512, which means that after being processed by the multilayer perceptron, the original bag-of-words vector is converted into a 512-dimensional text feature vector; this 512-dimensional text feature vector is used as a high-dimensional feature representation of the text and is used for subsequent hash function learning; We use V and T to represent the training samples of images and texts, respectively, and θ v ,θ t Indicates the parameters of image and text feature encoding, Encv and Enct are used to represent the image encoder and text encoder respectively. are the output high-dimensional features of the image modality and text modality, respectively, where d v and d t Represent the image and text feature dimensions respectively; the image modality feature and text modality feature are defined as shown in formula (2): F v =Encv(V,θ v );F t =Enct(T,θ t ) (2).

4. The method according to claim 3, characterized in that In step (3), F * 1 Represents the feature map obtained after the image and text modality features pass through the first fully connected layer FC; FC * 1 represents the first fully connected layer; F * 2 It indicates that the feature map obtained after the image and text modality features pass through the first fully connected layer FC is input into the feature map obtained after the second fully connected layer; FC * 2 represents the second fully connected layer; F * 3 It indicates that the feature map obtained after the image and text modality features pass through the second fully connected layer FC is input into the feature map obtained after the third fully connected layer; FC * 3 represents the third fully connected layer, *∈{v,t}.

5. The method according to claim 4, characterized in that In step (4), Z is transformed by the sign function sign(x) v and Z t Binarization can be performed to obtain the binary image and text hash code B v and B t : B * =sign(Z * ),*∈{v,t} (3) When x≥0, sign(x)=1, otherwise sign(x)=-1.

6. The method according to claim 5, characterized in that In step (5), the method of constructing the adjacency matrix A is as follows: First, the cosine distance is used to quantify the similarity between instances, that is, J ij =(R i ) T R j , J ij The larger the value, the more characteristic R i With R j The greater the similarity between them, the more similar they are. At the same time, the Euclidean distance is introduced to calculate the distance between features and construct the similarity matrix M. ij : Among them, ρ is the quantization parameter, which is set to 4; the two similarity matrices J are combined ij and M ij Construct the total similarity matrix U: U=J ij *M ij (5) Finally, we get the adjacency matrix A: A ij =U ij *S ij (6) Among them, A ij represents the element in the i-th row and j-th column of matrix A, S ij represents the element in the i-th row and j-th column of the matrix S, S ij =1 indicates feature R i With R j There is at least one identical tag and the semantics are similar, otherwise S ij =0,U ij represents the element in the i-th row and j-th column of the total similarity matrix U; The shared feature R represents the node information in the graph structure, and the adjacency matrix A obtained by the total similarity matrix U and the label-based similarity matrix S represents the mutual connection between the nodes; the graph structure (R, A) is constructed, and the total similarity matrix U and the label-based similarity matrix S are obtained; after obtaining the adjacency matrix A, the output of the graph convolutional neural network is calculated; each layer of the graph convolutional neural network is defined as formula (7): D is the degree matrix of the adjacency matrix A D ii are the elements on the diagonal of matrix D. All elements except the diagonal in D are 0. I represents the trainable parameters of layer I, I = 1, 2, 3; H0 represents the input of the graph convolutional neural network, and H2 represents the output Z of the graph convolutional neural network S =H2.

7. The method according to claim 6, characterized in that In step (6), the triplet loss function is formed by triplet loss, where each triplet consists of a positive sample, a negative sample, and an anchor sample, expressed as in Indicates that v i There is the same semantic information, which is the jth positive sample, v i represents the i-th anchor point sample, Indicates that v i There is different semantic information, which is the kth negative sample, i, j, and k are not equal; the triple loss function is defined as follows: Among them, D[f(x i ),f(x j )]=||f(x i )-f(x j )||2; f(λ) is the feature vector representation of the sample; α is a hyperparameter set artificially; [·] + It means that when the internal value is greater than zero, the result is the original value, and when the internal value is less than zero, the result is zero; according to the definition of the triple loss function, the intra-modal loss function L is constructed. inner : Among them, v represents the image sample features; t represents the text sample features; S represents the sample features after fusion of the graph convolutional neural network; It represents the unified embedded feature representation obtained after the i-th anchor image sample passes through three layers of fully connected layer feature mapping; It represents the unified embedded feature representation of the j-th negative image sample after being mapped by three layers of fully connected layers; It represents the unified embedded feature representation of the k-th positive image sample after being mapped by three layers of fully connected layers; It represents the unified embedded feature representation obtained after the i-th anchor text sample passes through three layers of fully connected layer feature mapping; It represents the unified embedded feature representation obtained after the j-th negative example text sample passes through three layers of fully connected layer feature mapping; It represents the unified embedded feature representation of the kth positive text sample after three layers of fully connected layer feature mapping; Represents the output of the i-th anchor image and text fusion sample features after passing through the graph convolutional neural network; Represents the output of j negative example images and text fusion sample features after passing through the graph convolutional neural network; It represents the output of the sample features after the fusion of the k-th positive image and text passes through the graph convolutional neural network; Construct the inter-modal triplet loss and construct the inter-modal loss function L according to the definition of the triplet loss function intra : In addition to the above two triplet loss functions describing intra-modal and inter-modal loss, the following pairwise loss and quantization loss are also defined. The pairwise loss is used to reduce Z v With Z S , Z t With Z S The difference between: Since there is a quantization error when generating binary codes, the following quantization loss function is defined:

8. The method according to claim 7, characterized in that In step (7), the total loss function during the entire training process is calculated according to the following formula: L TOTAL =L inner +αL intra +βL pair +λL qu (13) Among them, the hyperparameters α, β, and λ are weight parameters.

9. The method according to claim 8, characterized in that In step (8), the Hamming distance from the retrieved sample in the image modality to the retrieved sample in the text modality is calculated based on the binary hash code; the Hamming distance from the retrieved sample in the text modality to the retrieved sample in the image modality is calculated using the same steps.

10. The method according to claim 9, characterized in that In step (8), the Hamming distance is calculated using the following formula: The derivation process of the above formula is as follows: Given n training instances containing two modalities: image and text o i =(V i ,T i ), where V i represents the image modality data of the ith instance, T i represents the text modal data of the i-th instance; the label matrix is Among them l i represents the label vector of the i-th instance, c represents the number of labels; the hash code of the image modality is defined as B v ={-1,+1} K , the hash code of the text mode is B t ={-1,+1} K , where K represents the length of the hash code, and the number of mini-batches divided into each training is m; the Hamming distance is used to measure the distance between binary hash codes, which is the number of different characters in strings of equal length; given a binary hash code h i and h j , the Hamming distance calculation formula is as follows: Where b is the hash code length, ⊙ is h i and h j The dot product of .

Images