Cross-modal hash retrieval method based on attention mechanism

Abstract

The invention discloses a cross-modal hash retrieval method based on an attention mechanism. The method comprises the following steps: acquiring first multi-modal data; inputting the first multi-modal data into a retrieval and representation learning model to obtain a target fusion discrete Hash code, the target fusion discrete Hash code being used for retrieving the first multi-modal data; wherein the retrieval and representation learning model is obtained through the following steps that a plurality of training samples are obtained, and each training sample comprises second multi-modal data and a first fusion discrete hash code; based on the loss function, inputting the plurality of training samples into an initial retrieval and representation learning model for model iteration training until the loss function value reaches a preset convergence value, and obtaining a retrieval and representation learning model, the initial retrieval and representation learning model is constructed based on a convolutional neural network, a bag-of-words model, a Transform encoder and / or a cross-modal attention mechanism.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal data retrieval, and in particular to a cross-modal hash retrieval method based on an attention mechanism. Background Art

[0002] With the development of big data and deep learning, multimodal big data processing and analysis has become an important research topic. Multimodal data fusion aims to integrate information from different modalities (such as images, text, etc.) to improve the model's ability to understand and express complex information. However, due to the significant differences in feature structures and semantic expressions of different modalities, existing multimodal hashing methods have problems such as difficulty in aligning feature spaces due to the heterogeneity of multimodal big data and insufficient information complementarity between modalities. Summary of the invention

[0003] This application aims to propose a cross-modal hash retrieval method based on the attention mechanism, which can solve the problems of difficulty in feature space alignment and insufficient information complementarity between modalities in traditional multimodal data fusion technology.

[0004] The embodiment of the present application provides a cross-modal hash retrieval method based on an attention mechanism, comprising: acquiring first multimodal data; Inputting the first multimodal data into a retrieval and representation learning model to obtain a target fused discrete hash code, wherein the target fused discrete hash code is used to retrieve the first multimodal data; The retrieval and representation learning model is obtained through the following steps: Acquire a plurality of training samples, wherein the training samples include second multimodal data and a first fused discrete hash code; Based on the loss function, multiple training samples are input into the initial retrieval and representation learning model for iterative model training until the loss function value reaches a preset convergence value, thereby obtaining the retrieval and representation learning model, wherein the initial retrieval and representation learning model is constructed based on a convolutional neural network, a bag-of-words model, a Transformer encoder and / or a cross-modal attention mechanism.

[0005] According to some embodiments of the present application, the model iterative training includes: Extract features from the second multimodal data using the convolutional neural network and the bag-of-words model in the initial retrieval and representation learning model to obtain a text feature value matrix and an image feature value matrix; Input the text eigenvalue matrix and the image eigenvalue matrix into a Transformer encoder to generate a first text representation matrix and a first image representation matrix; Using the cross-modal attention mechanism in the initial retrieval and representation learning model, the first text representation matrix and the first image representation matrix are fused to obtain a second text representation matrix and a second image representation matrix; Inputting the text feature value matrix and the image feature value matrix into the similarity module in the initial retrieval and representation learning model to obtain a text self-similarity matrix and an image self-similarity matrix; Using the first fusion module in the initial retrieval and representation learning model, the text self-similarity matrix and the image self-similarity matrix are fused to obtain an overall similarity matrix; Using the second fusion module in the initial retrieval and representation learning model, the text self-similarity matrix, the image self-similarity matrix and the overall similarity matrix are fused respectively to obtain a text cross-modal transfer matrix and an image cross-modal transfer matrix; Inputting the text cross-modal transfer matrix and the image cross-modal transfer matrix into a third fusion module in the initial retrieval and representation learning model to obtain an overall cross-modal transfer matrix; Using the graph convolutional network in the initial retrieval and representation learning model, generating a third text representation matrix and a third image representation matrix according to the second text representation matrix, the second image representation matrix and the overall similarity matrix; Using a fourth fusion module in the initial retrieval and representation learning model, the first text representation matrix, the first image representation matrix and the overall cross-modal transfer matrix are fused to obtain a fused representation matrix; Mapping the fused representation matrix into a fused continuous-valued hash code using a linear layer in the initial retrieval and representation learning model; Inputting the fused continuous value hash code into the SIGN function in the initial retrieval and representation learning model to obtain a second fused discrete hash code; Performing a self-similar transformation on the fused continuous-value hash code to obtain a fused hash representation matrix; Using the fully connected layer in the initial retrieval and representation learning model, combining the third image representation matrix and the fusion representation matrix, a first single-path hash representation matrix is ​​obtained; Using the fully connected layer in the initial retrieval and representation learning model, combining the third text representation matrix and the fusion representation matrix, a second single-way hash representation matrix is ​​obtained; Combining the first single-way hash representation matrix and the second single-way hash representation matrix to obtain a multi-way joint hash representation matrix; Based on the loss function, a loss function value is obtained according to the first single-way hash representation matrix, the second single-way hash representation matrix, the overall similarity matrix, the fused hash representation matrix, the multi-way joint hash representation matrix and the first fused discrete hash code; Based on the loss function value, the initial retrieval and representation learning model is optimized.

[0006] According to some embodiments of the present application, the fourth fusion module in the initial retrieval and representation learning model is used to fuse the first text representation matrix, the first image representation matrix and the overall cross-modal transfer matrix to obtain a fused representation matrix, including: Using the splicing submodule of the fourth fusion module in the initial retrieval and representation learning model, splicing the first text representation matrix and the first image representation matrix to obtain an overall representation matrix; The overall representation matrix and the overall cross-modal transfer matrix are input into the point multiplication submodule of the fourth fusion module in the initial retrieval and representation learning model to obtain the fused representation matrix.

[0007] According to some embodiments of the present application, the step of inputting the text eigenvalue matrix and the image eigenvalue matrix into a similarity module in the initial retrieval and representation learning model to obtain a text self-similarity matrix and an image self-similarity matrix includes: Inputting the text feature value matrix and the image feature value matrix into a similarity module in the initial retrieval and representation learning model; Using the normalization submodule of the similarity module in the initial retrieval and representation learning model, the text feature value matrix and the image feature value matrix are normalized to obtain a text feature vector and an image feature vector; The first dot product submodule of the similarity module in the initial retrieval and representation learning model is used to perform dot product calculations on the text feature vector and the image feature vector, respectively, to obtain the text self-similarity matrix and the image self-similarity matrix.

[0008] According to some embodiments of the present application, the first fusion module in the initial retrieval and representation learning model is used to fuse the text self-similarity matrix and the image self-similarity matrix to obtain an overall similarity matrix, including: Using the first weighted fusion submodule of the first fusion module in the initial retrieval and representation learning model, the text self-similarity matrix and the image self-similarity matrix are linearly weightedly fused to obtain a first similarity matrix; Using the smoothing submodule of the first fusion module in the initial retrieval and representation learning model, the first similarity matrix is ​​smoothed to obtain a second similarity matrix; Using the second weighted fusion submodule of the first fusion module in the initial retrieval and representation learning model, the first similarity matrix and the second similarity matrix are linearly weightedly fused to obtain a third similarity matrix; Using the matrix calculation submodule of the first fusion module in the initial retrieval and representation learning model, a Laplace matrix is ​​calculated according to the third similarity matrix; The Laplacian matrix is ​​negated by using the negation submodule of the first fusion module in the initial retrieval and representation learning model to obtain the overall similarity matrix.

[0009] According to some embodiments of the present application, the cross-modal attention mechanism in the initial retrieval and representation learning model is used to fuse the first text representation matrix and the first image representation matrix to obtain a second text representation matrix and a second image representation matrix, including: Using the second dot product module of the cross-modal attention mechanism in the initial retrieval and representation learning model, taking the first text representation matrix as a first query matrix, taking the first image representation matrix as a first key matrix and a first key and value matrix, and calculating a first attention score; Using a first soft attention weighting module of the cross-modal attention mechanism in the initial retrieval and representation learning model, soft attention weighting is performed on the first key matrix and the first key and value matrix according to the first attention score to obtain the second text representation matrix; Using the third dot product module of the cross-modal attention mechanism in the initial retrieval and representation learning model, taking the first image representation matrix as the second query matrix, taking the first text representation matrix as the second key matrix and the second key and value matrix, a second attention score is calculated; The second soft attention weighting module of the cross-modal attention mechanism in the initial retrieval and representation learning model is utilized to perform soft attention weighting on the second key matrix and the second key and value matrix according to the second attention score to obtain the second image representation matrix.

[0010] According to some embodiments of the present application, the loss function includes: , , , , , , Where n represents the number of training samples. is the element in the i-th row and j-th column of the overall similarity matrix, is the element in the i-th row and j-th column of the fused hash representation matrix, represents the element of the i-th row and j-th column of the multi-way joint hash representation matrix, is a hyperparameter to control the weight, c represents the length of the hash code, and are the j-th bits of the two hash codes generated according to the first one-way hash representation matrix and the second one-way hash representation matrix in the i-th training sample, respectively. is the jth hash bit in the first fused discrete hash code of the i-th training sample.

[0011] In an embodiment of the present application, the first multimodal data is input into a retrieval and representation learning model, and features of the first multimodal data are extracted through a convolutional neural network and a bag-of-words model. The self-attention mechanism of the Transformer encoder is used to explore the relationship between features, thereby enhancing the expression of cross-modal features. Then, through the cross-modal attention mechanism, information interaction is performed between the image modality and the text modality, so that each modality can receive information from other modalities, enhance the complementarity of features, and alleviate the information gap caused by the differences between modalities. Finally, multiple target fusion discrete hash codes with high similarity and consistency corresponding to different modalities are obtained, which effectively solves the problems of difficulty in aligning feature space and insufficient information complementarity between modalities in multimodal learning, and improves the accuracy and robustness of target fusion discrete hash codes.

[0012] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The present application is further described below with reference to the accompanying drawings and embodiments, wherein: Figure 1 A flowchart of an embodiment of a cross-modal hash retrieval method based on an attention mechanism provided in this application; Figure 2 The overall framework diagram of the initial retrieval and representation learning model in the embodiment of the cross-modal hash retrieval method based on the attention mechanism provided in this application; Figure 3 A schematic diagram of an electronic device embodiment provided in the present application.

[0014] Reference numerals: Electronic device 100 , processor 110 , and memory 120 . DETAILED DESCRIPTION

[0015] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0016] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0017] In the description of this application, "a plurality" means more than two. If there is a description of "first" or "second", it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0018] In the description of this application, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in this application based on the specific content of the technical solution.

[0019] The present application embodiment provides a cross-modal hash retrieval method based on an attention mechanism, such as Figure 1 As shown, including: Step S100: Acquire first multimodal data; Step S200: inputting the first multimodal data into a retrieval and representation learning model to obtain a target fused discrete hash code, where the target fused discrete hash code is used to retrieve the first multimodal data; The retrieval and representation learning models are obtained through the following steps: Step S300: Acquire multiple training samples, where the training samples include second multimodal data and first fused discrete hash codes; Step S400: Input multiple training samples into the initial retrieval and representation learning model for model training to obtain the retrieval and representation learning model, wherein the initial retrieval and representation learning model is constructed based on a convolutional neural network, a bag-of-words model, a Transformer encoder and / or a cross-modal attention mechanism.

[0020] In an embodiment of the present application, the first multimodal data is input into a retrieval and representation learning model, and features of the first multimodal data are extracted through a convolutional neural network and a bag-of-words model. The self-attention mechanism of the Transformer encoder is used to explore the relationship between features, thereby enhancing the expression of cross-modal features. Then, through the cross-modal attention mechanism, information interaction is performed between the image modality and the text modality, so that each modality can receive information from other modalities, enhance the complementarity of features, and alleviate the information gap caused by the differences between modalities. Finally, multiple target fusion discrete hash codes with high similarity and consistency corresponding to different modalities are obtained, which effectively solves the problems of difficulty in aligning feature space and insufficient information complementarity between modalities in multimodal learning, and improves the accuracy and robustness of target fusion discrete hash codes.

[0021] In some embodiments of the present application, the initial retrieval and representation learning model includes: a convolutional neural network, a bag-of-words model, a Transformer encoder, a cross-modal attention mechanism, a similarity module, a first fusion module, a second fusion module, a third fusion module, a graph convolutional network, a fourth fusion module, a linear layer and a SIGN function.

[0022] In some embodiments of the present application, the first multimodal data may be one or more, and when multiple first multimodal data are input into the retrieval and representation learning model, multiple target fusion discrete hash codes can be obtained. Both the first multimodal data and the second multimodal data include text modal data and image modal data.

[0023] In some embodiments of the present application, Figure 2 As shown, the model iterative training process in the above steps includes: Step S201: Using the convolutional neural network and bag-of-words model in the initial retrieval and representation learning model to extract features from the second multimodal data, and obtain a text feature value matrix (such as Figure 2 In ) and the image eigenvalue matrix (such as Figure 2 In ); Step S202: Input the text eigenvalue matrix and the image eigenvalue matrix into the Transformer encoder to generate a first text representation matrix (such as Figure 2 In ) and the first image representation matrix (such as Figure 2 In ); Step S203: Using the cross-modal attention mechanism in the initial retrieval and representation learning model, the first text representation matrix and the first image representation matrix are fused to obtain a second text representation matrix (such as Figure 2 In ) and the second image representation matrix (such as Figure 2 In ); Step S204: Input the text feature value matrix and the image feature value matrix into the similarity module in the initial retrieval and representation learning model to obtain a text self-similarity matrix (such as Figure 2 In ) and the image self-similarity matrix (such as Figure 2 In ); Step S205: Using the first fusion module in the initial retrieval and representation learning model, fuse the text self-similarity matrix and the image self-similarity matrix to obtain an overall similarity matrix (such as Figure 2 In ); Step S206: Using the second fusion module in the initial retrieval and representation learning model, the text self-similarity matrix, the image self-similarity matrix and the overall similarity matrix are fused to obtain a text cross-modal transfer matrix (such as Figure 2 In ) and the image cross-modal transfer matrix (such as Figure 2 In ); Step S207: Input the text cross-modal transfer matrix and the image cross-modal transfer matrix into the third fusion module in the initial retrieval and representation learning model to obtain the overall cross-modal transfer matrix (such as Figure 2 In ); Step S208: Using the graph convolutional network in the initial retrieval and representation learning model, a third text representation matrix (such as Figure 2 In ) and the third image representation matrix (such as Figure 2 In ); Step S209: Using the fourth fusion module in the initial retrieval and representation learning model, the first text representation matrix, the first image representation matrix and the overall cross-modal transfer matrix are fused to obtain a fused representation matrix (such as Figure 2 In ); Step S210: Using the linear layer in the initial retrieval and representation learning model, the fused representation matrix is ​​mapped into a fused continuous value hash code (such as Figure 2 In ); Step S211: Input the fused continuous value hash code into the SIGN function in the initial retrieval and representation learning model to obtain a second fused discrete hash code (such as Figure 2 In ); Step S212: performing a self-similar transformation on the fused continuous value hash code to obtain a fused hash representation matrix; Step S213: using the fully connected layer in the initial retrieval and representation learning model, combining the third image representation matrix and the fusion representation matrix, to obtain a first single-path hash representation matrix; Step S214: using the fully connected layer in the initial retrieval and representation learning model, combining the third text representation matrix and the fusion representation matrix, to obtain a second single-way hash representation matrix; Step S215: combining the first single-way hash representation matrix and the second single-way hash representation matrix to obtain a multi-way joint hash representation matrix; Step S216: Based on the loss function, a loss function value is obtained according to the first single-way hash representation matrix, the second single-way hash representation matrix, the overall similarity matrix, the fused hash representation matrix, the multi-way joint hash representation matrix and the first fused discrete hash code; Step S217: Optimize the initial retrieval and representation learning model based on the loss function value.

[0024] In the above step S201, a convolutional neural network is used to perform feature extraction on the image data of the second multimodal data, and a bag-of-words model is used to perform feature extraction on the text data of the second multimodal data, to construct a preliminary multimodal feature representation, and obtain an image eigenvalue matrix and a text eigenvalue matrix.

[0025] In the above step S202, the image eigenvalue matrix and the text eigenvalue matrix are input into the Transformer encoder, and the image eigenvalue matrix and the text eigenvalue matrix are converted into high-dimensional embedding representations through embedding and serialization to obtain more fine-grained hidden information. Then the Transformer decoder is used to integrate and reconstruct the serialized high-dimensional embedding representation to obtain a cross-modal aligned feature space. Specifically, the image eigenvalue matrix and the text eigenvalue matrix are divided into sequence representations with multiple embedding dimensions, and the relationship between features is mined through the self-attention mechanism of the Transformer encoder, thereby enhancing the expression of cross-modal features. For example: for the image eigenvalue matrix, , where n is the number of second multimodal data and m is the feature vector dimension. m is split so that m=sl*ed. Where sl represents the sequence length and ed represents the dimension of the embedding vector. After the Transformer encoder is processed, the first image representation matrix is ​​restored to the same shape as the image eigenvalue matrix through vector concatenation and linear layers, that is, Similarly, for the first text representation matrix, we also have .

[0026] In the above step S203, the cross-modal attention mechanism in the initial retrieval and representation learning model is used to fuse the first text representation matrix and the first image representation matrix to obtain the second text representation matrix and the second image representation matrix, including: Step S2031: using the second dot product module of the cross-modal attention mechanism in the initial retrieval and representation learning model, taking the first text representation matrix as the first query matrix, taking the first image representation matrix as the first key matrix and the first key and value matrix, and calculating a first attention score; Step S2032: using the first soft attention weighting module of the cross-modal attention mechanism in the initial retrieval and representation learning model, soft attention weighting is performed on the first key matrix and the first key and value matrix according to the first attention score to obtain a second text representation matrix; Step S2033: using the third dot product module of the cross-modal attention mechanism in the initial retrieval and representation learning model, taking the first image representation matrix as the second query matrix, taking the first text representation matrix as the second key matrix and the second key and value matrix, and calculating a second attention score; Step S2034: Utilize the second soft attention weighting module of the cross-modal attention mechanism in the initial retrieval and representation learning model, and perform soft attention weighting on the second key matrix and the second key and value matrix according to the second attention score to obtain a second image representation matrix.

[0027] In this implementation, the data of the image modality is used as keys and values, thereby performing information interaction between image features and text features, so that the features of each modality can complement each other, alleviating the information gap caused by the differences between the modalities and enhancing the complementary learning ability of the model.

[0028] Specifically, taking the acquisition of image feature information by text features as an example, assuming that the first image representation matrix is ​​the first key matrix and the first key and value matrix, and the first text representation matrix is ​​the first query matrix, the calculation process of the cross-modal attention mechanism is as follows: First, the attention score is calculated by the dot product of the first query matrix and the first key matrix: , in, represents the first query matrix, The number of queries. denote the first key matrix and the first key and value matrix respectively, is the number of keys and values. The dimension of the key.

[0029] Soft attention is weighted on the relationship between keys and values ​​to interact information between queries in the text modality and values ​​in the image modality: , Among them, Z is the output result of the cross-modal attention mechanism, which represents the information associated with the text modality obtained from the image modality. This result can serve as a feature supplement to the image modality.

[0030] Specifically, the first image representation matrix is ​​fused with the first text representation matrix through a cross-modal attention mechanism to obtain the second image representation matrix: , in, is the second image representation matrix, , is the first image representation matrix, is the first text representation matrix.

[0031] In the above step S204, the text eigenvalue matrix and the image eigenvalue matrix are input into the similarity module in the initial retrieval and representation learning model to obtain the text self-similarity matrix and the image self-similarity matrix, including: Step S2041: inputting the text feature value matrix and the image feature value matrix into the similarity module in the initial retrieval and representation learning model; Step S2042: using the normalization submodule of the similarity module in the initial retrieval and representation learning model, normalizing the text feature value matrix and the image feature value matrix to obtain a text feature vector and an image feature vector; Step S2043: using the first dot product submodule of the similarity module in the initial retrieval and representation learning model, perform dot product calculations on the text feature vector and the image feature vector respectively to obtain a text self-similarity matrix and an image self-similarity matrix.

[0032] In this implementation, the image feature matrix and the text feature matrix are first normalized using a normalization submodule to ensure that the norm of each feature vector is 1: , in, is the image eigenvalue matrix, is the text eigenvalue matrix.

[0033] Using the first dot product submodule, the normalized eigenvectors are used to calculate the self-similarity matrix of the image modality and the text modality through dot product to obtain the text self-similarity matrix and the image self-similarity matrix: , in, is the image self-similarity matrix, is the text self-similarity matrix.

[0034] In the above step S205, the first fusion module in the initial retrieval and representation learning model is used to fuse the text self-similarity matrix and the image self-similarity matrix to obtain an overall similarity matrix, including: Step S2051: using the first weighted fusion submodule of the first fusion module in the initial retrieval and representation learning model, linear weighted fusion processing is performed on the text self-similarity matrix and the image self-similarity matrix to obtain a first similarity matrix; Step S2052: using the smoothing processing submodule of the first fusion module in the initial retrieval and representation learning model, smoothing the first similarity matrix to obtain a second similarity matrix; Step S2053: using the second weighted fusion submodule of the first fusion module in the initial retrieval and representation learning model, performing linear weighted fusion processing on the first similarity matrix and the second similarity matrix to obtain a third similarity matrix; Step S2054: using the matrix calculation submodule of the first fusion module in the initial retrieval and representation learning model, calculate and obtain the Laplace matrix according to the third similarity matrix; Step S2055: using the negation submodule of the first fusion module in the initial retrieval and representation learning model, negate the Laplacian matrix to obtain an overall similarity matrix.

[0035] In this implementation, the first weighted fusion submodule is used to perform linear weighted fusion on the image self-similarity matrix and the text self-similarity matrix to obtain a first similarity matrix: , in, is the first similarity matrix, is the first hyperparameter, which controls the weighted ratio of the similarity between the two modalities. Set to 0.6.

[0036] Among the massive training samples, the most useful ones are often those with high similarity, while those with moderate similarity become noise. Therefore, local Gaussian kernel enhancement is used to improve the local similarity of training samples. The similarity values ​​of the first similarity matrix are locally weighted using the Gaussian kernel function. First, the overall pairwise distance matrix of the image feature matrix and the text feature matrix is ​​calculated: , in, is the overall pairwise distance matrix, which is used to measure the overall distance of each pair of training samples. is the pairwise distance matrix of the image eigenvalue matrix, and the Euclidean distance between each pair of image features is calculated. is the pairwise distance matrix of the text feature value matrix. The overall pairwise distance matrix reflects the relative distance of the second multimodal data in the feature space. The larger the value, the lower the similarity between the two training samples.

[0037] Then, the mean of the overall pairwise distance matrix is ​​used as the local scale , define the Gaussian kernel matrix: , It represents the average distance of training samples in the entire feature space and is used as the range of the Gaussian kernel. The smaller the value, the more obvious the clustering effect of similar training samples. :Usually take , which is used to avoid zero division errors in subsequent calculations. The Gaussian kernel function G is used to convert the overall pairwise distance matrix into similarity values. This function has the characteristic of "the smaller the distance, the greater the similarity". By strengthening the influence of the Gaussian kernel, similar training samples are more likely to be clustered together.

[0038] The smoothing submodule is used to perform similarity smoothing on the first similarity matrix. The first similarity matrix is ​​smoothed by multiplying it with the square of the Gaussian kernel matrix to better separate the noise training samples with moderate similarity in the global domain and capture more obvious local similarities to obtain the second similarity matrix: , in is the second similarity matrix, is a weight factor used to adjust the local similarity, for example Set to 1.0. Next, the second similarity matrix is ​​input into the scaled Sigmoid activation function to better separate the noisy training samples with moderate similarity, and the values ​​are mapped to the (-1,1) interval by the final minus "1" operation. Add 1 on the diagonal to ensure that the similarity of the training samples themselves is high: , Among them, I is the identity matrix, which is used to add 1 to the diagonal of the second similarity matrix (to ensure that the similarity of the training samples themselves is high). Then, to ensure that the second similarity matrix is ​​symmetrical, the second similarity matrix is ​​averaged with its transpose to obtain the final second similarity matrix: , The first similarity matrix and the final second similarity matrix are linearly weighted fused to obtain the third similarity matrix: , in, is the third similarity matrix, is the second hyperparameter, controlling and The weighted proportion of .

[0039] The third similarity matrix is ​​used to calculate the Laplacian matrix. The Laplacian matrix is ​​often used to capture the relationship between nodes in the graph structure, which can help better cluster similar training samples, thereby enhancing the relevance of similar training samples.

[0040] The degree matrix D is a diagonal matrix, and each diagonal element represents the sum of the similarities between the corresponding training sample in the third similarity matrix and all training samples: ,Right now , Laplacian Matrix : , , Through the Laplacian matrix, the relationship between similar training samples can be enhanced, thereby encouraging them to be closer in the similarity matrix. When i=j, it means the cumulative similarity between training sample i and all training samples minus the similarity of training sample i itself; when i When j, it indicates the direct similarity between training sample i and training sample j, and the negative sign indicates the degree of separation between the two.

[0041] The Laplace matrix is ​​negated to obtain the overall similarity matrix, so that the final similarity value is more in line with the range requirements, that is, the overall similarity matrix and the Laplace matrix numerical value represent the opposite logic. Use negative Laplace matrix values Represents the clustering effect and obtains the overall similarity matrix: , in, is the overall similarity matrix.

[0042] By scaling the overall similarity matrix with the tanh function, we can ensure that the values ​​of all elements are between (-1, 1) and obtain the final overall similarity matrix:

[0043] Where n represents the number of training samples. It represents the similarity between the i-th training sample and the j-th training sample.

[0044] In the above step S206, by weightedly combining the overall similarity matrix with the text self-similarity matrix and the image self-similarity matrix, the information within and between modalities can be fully utilized in cross-modal feature alignment, and the features of each modality can be effectively fused to help the model achieve more accurate transfer learning between heterogeneous modalities. Specifically, the overall similarity matrix and the text self-similarity matrix are fused to obtain the image cross-modal transfer matrix. Similarly, the text cross-modal transfer matrix can be obtained.

[0045] , , in, is the image cross-modal transfer matrix, is the text self-similarity matrix, is the text cross-modal transfer matrix, Represents the weight of the overall similarity matrix, ensuring the stability of the eigenvalues ​​of the modality when learning using the cross-modal transfer matrix.

[0046] In the above step S207, the image cross-modal transfer matrix and the text cross-modal transfer matrix are concatenated to obtain the overall cross-modal transfer matrix , , which allows the learned new representation to take into account information from both modalities:

[0047] From the perspective of block matrix, The upper left corner is , the lower right corner is , the upper right and lower left corners are both unit matrices .

[0048] In the above step S208, the third image representation matrix after graph convolution is obtained by the effect of the overall similarity matrix on the second image representation matrix: , in, is the third image representation matrix, , is the overall similarity matrix, is the second image representation matrix.

[0049] Similarly, the third text representation matrix can be obtained.

[0050] In the above step S209, “using the fourth fusion module in the initial retrieval and representation learning model to fuse the first text representation matrix, the first image representation matrix and the overall cross-modal transfer matrix to obtain a fused representation matrix” includes: Step S2091: using the concatenation submodule of the fourth fusion module in the initial retrieval and representation learning model, concatenate the first text representation matrix and the first image representation matrix to obtain an overall representation matrix; Step S2092: input the overall representation matrix and the overall cross-modal transfer matrix into the point multiplication submodule of the fourth fusion module in the initial retrieval and representation learning model to obtain a fused representation matrix.

[0051] In this embodiment, the splicing submodule is used to splice the first image representation matrix and the first text representation matrix to obtain an overall representation matrix: , in, is the overall representation matrix, is the first image representation matrix, is the first text representation matrix.

[0052] By using the point multiplication submodule, a fully integrated fusion representation matrix can be obtained by performing the point multiplication of the overall cross-modal transfer matrix and the overall representation matrix: , , in, is the fusion representation matrix, .

[0053] In step S210, the fused representation matrix is ​​mapped using the linear layer in the initial retrieval and representation learning model to obtain a fused continuous value hash code , , where c is the length of the fused continuous value hash code.

[0054] In the above step S211, the fused continuous value hash code is input into the SIGN function to obtain the target fused discrete hash code.

[0055] In some embodiments of the present application, the step S300 of “inputting a plurality of training samples into the initial retrieval and representation learning model for model training to obtain the initial retrieval and representation learning model” includes: Step S310: Based on the loss function, multiple training samples are input into the initial retrieval and representation learning model for iterative training until the loss function value reaches a preset convergence value, thereby obtaining the initial retrieval and representation learning model.

[0056] In this embodiment, in some embodiments of the present application, during the training of the initial retrieval and representation learning model, the training sample is input into the initial retrieval and representation learning model to obtain the fused continuous value hash code, and then the fused continuous value hash code is self-similarly transformed to obtain the final fused hash representation matrix (such as Figure 2 In , ).

[0057] At the same time, the overall similarity matrix is ​​used to generate a separate semantic representation for each modality, and a single-path representation matrix is ​​constructed (such as Figure 2 In , ), thereby generating a cross-modal representation with consistent semantics for subsequent processing and retrieval applications. In order to ensure the stability of the subsequent single-channel representation matrix, the third image representation matrix and the fusion representation matrix are combined through the full connection layer of the initial retrieval and representation learning model to obtain the first single-channel hash representation matrix of the image modality (such as Figure 2 In , ). Similarly, the second one-way hash representation matrix of the text modality can be obtained (such as Figure 2 In ). Therefore, the first single-way hash characterization matrix and the second single-way hash characterization matrix are combined to obtain a multi-way joint hash characterization matrix:

[0058] , in, is a multi-way joint hash representation matrix, representing the cross-modal similarity between images and texts. It reflects the characteristic relationship between image sample i and text sample j in the hash space.

[0059] In some embodiments of the present application, the semantic expression ability of the model-generated hash code is improved by constructing a matrix similarity loss space. Specifically, the semantic similarity loss is calculated from three perspectives: "overall similarity matrix-fused hash representation matrix", "overall similarity matrix-multi-way joint hash representation matrix" and "fused hash representation matrix-multi-way joint hash representation matrix" to ensure that the hash code has a high degree of semantic consistency and accuracy between different modalities.

[0060] Specifically, the semantic similarity loss of the overall similarity matrix-fused hash representation matrix: Using the relationship between the overall similarity matrix and the fused hash representation matrix, the semantic similarity loss is calculated using the cross entropy loss function. Assume that the elements of the overall similarity matrix are , the elements of the fused hash representation matrix are , the first loss function is as follows: , Among them, n represents the number of training samples. The first loss function is the average of the squares of the element-by-element subtractions between the overall similarity matrix and the fused hash representation matrix. The first loss function is used to make the fused hash representation matrix closer to the overall similarity matrix, thereby ensuring that the similarity structure of the hash representation is more consistent with the real semantic relationship.

[0061] Overall similarity matrix - semantic similarity loss of multi-way joint hash representation matrix: Through the relationship between the overall similarity matrix and the multi-way joint hash representation matrix, the semantic similarity loss between modalities is further calculated. Assume that the elements of the multi-way joint hash representation matrix are , then the second loss function is defined as: , in, Represents the element in the i-th row and j-th column of the multi-way joint hash representation matrix.

[0062] Semantic similarity loss of fused hash representation matrix-multi-way joint hash representation matrix: The semantic similarity loss is achieved by comparing the fused hash representation matrix and the multi-way joint hash representation matrix. Specifically, the mean square error between the fused hash representation matrix and the multi-way joint hash representation matrix is ​​calculated as the similarity loss to ensure the semantic consistency of the two. The calculation formula of the third loss function is as follows: , By minimizing the difference between the fused hash representation matrix and the multi-way joint hash representation matrix through the mean square error, the consistency of the semantic structure of the fused hash representation matrix and the multi-way joint hash representation matrix is ​​ensured, thereby generating hash codes with high semantic consistency in different modalities, supporting more accurate cross-modal feature alignment and information retrieval.

[0063] In some embodiments of the present application, based on the matrix similarity loss space, the generated hash code is further optimized to construct a cross-modal hash coding optimization space to ensure that the generated hash code has stronger semantic consistency and expression ability in different modalities. Specifically, it includes the following aspects: Hash representation consistency loss: To ensure the consistency of the hash codes generated by the image modality and the text modality, the mean square error between the hash representations of the image modality and the text modality in the second multimodal data and the first fused discrete hash code is calculated while keeping the hash codes of the two modalities consistent. The calculation formula is as follows: , Where c represents the length of the hash code. , denote the jth bit of the continuous hash code of the i-th training sample generated by the first one-way hash representation and the second one-way hash representation, respectively, Represents the jth hash bit of the i-th training sample in the first fused discrete hash code.

[0064] Spatial mapping consistency preservation loss: By calculating the mean square error between the similarity matrix between the generated hash representations and the overall similarity matrix, it is ensured that the model can maintain the similarity structure within and between modalities after optimization, and maintain a reliable mapping relationship from feature space to hash space. Specifically, the loss covers the hash similarity matrix between image and text modalities (i.e., the multi-way joint hash representation matrix), the first hash similarity matrix within the image modality, and the second hash similarity matrix within the text modality, to ensure that the feature representations of different modalities can still maintain their semantic relationship after being mapped to the hash space. The specific calculation formula is as follows: , The total optimization objective loss function is: , in, is a hyperparameter that controls the weight, for example Set to 0.5 to adjust the proportion of similarity loss in the total loss. is the first hash similarity matrix, is the second hash similarity matrix.

[0065] In some embodiments of the present application, cyclic optimization is performed to ensure that the initial retrieval and representation learning model gradually converges to the optimal state. The total optimization target loss function is used as the loss function, and the Adam optimization algorithm is selected to perform model training on the initial retrieval and representation learning model. In each iteration, the model parameters of the initial retrieval and representation learning model are gradually updated through back propagation, so that the cross-modal hash coding is continuously improved in semantic consistency and modality alignment. The training process continues until the loss function value tends to stabilize and reaches the preset convergence standard, thereby obtaining the optimal retrieval and representation learning model.

[0066] In addition, the present application embodiment provides an electronic device 100, such as Figure 3 As shown, including: at least one processor 110; At least one memory 120, used to store at least one program; When at least one program is executed by at least one processor 110, the cross-modal hash retrieval method based on the attention mechanism as described above is implemented.

[0067] The electronic device 100 provided in the embodiment of the present application can implement each process implemented in the above method embodiment and achieve the same beneficial effects. To avoid repetition, it will not be described again here.

[0068] In addition, an embodiment of the present application provides a computer-readable storage medium, which stores a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to implement the cross-modal hash retrieval method based on the attention mechanism as described above.

[0069] The computer-readable storage medium provided in the embodiment of the present application can implement each process implemented in the above method embodiment and achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0070] It will be appreciated by those skilled in the art that all or some of the steps and systems in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or transient medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0071] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A cross-modal hash retrieval method based on attention mechanism, characterized in that: include: Acquiring first multimodal data; Inputting the first multimodal data into a retrieval and representation learning model to obtain a target fused discrete hash code, wherein the target fused discrete hash code is used to retrieve the first multimodal data; The retrieval and representation learning model is obtained through the following steps: Acquire a plurality of training samples, wherein the training samples include second multimodal data and a first fused discrete hash code; Based on the loss function, multiple training samples are input into the initial retrieval and representation learning model for iterative model training until the loss function value reaches a preset convergence value, thereby obtaining the retrieval and representation learning model, wherein the initial retrieval and representation learning model is constructed based on a convolutional neural network, a bag-of-words model, a Transformer encoder and / or a cross-modal attention mechanism.

2. The cross-modal hash retrieval method based on the attention mechanism according to claim 1 is characterized in that: The model iterative training includes: Extract features from the second multimodal data using the convolutional neural network and the bag-of-words model in the initial retrieval and representation learning model to obtain a text feature value matrix and an image feature value matrix; Input the text eigenvalue matrix and the image eigenvalue matrix into a Transformer encoder to generate a first text representation matrix and a first image representation matrix; Using the cross-modal attention mechanism in the initial retrieval and representation learning model, the first text representation matrix and the first image representation matrix are fused to obtain a second text representation matrix and a second image representation matrix; Inputting the text feature value matrix and the image feature value matrix into the similarity module in the initial retrieval and representation learning model to obtain a text self-similarity matrix and an image self-similarity matrix; Using the first fusion module in the initial retrieval and representation learning model, the text self-similarity matrix and the image self-similarity matrix are fused to obtain an overall similarity matrix; Using the second fusion module in the initial retrieval and representation learning model, the text self-similarity matrix, the image self-similarity matrix and the overall similarity matrix are fused respectively to obtain a text cross-modal transfer matrix and an image cross-modal transfer matrix; Inputting the text cross-modal transfer matrix and the image cross-modal transfer matrix into a third fusion module in the initial retrieval and representation learning model to obtain an overall cross-modal transfer matrix; Using the graph convolutional network in the initial retrieval and representation learning model, generating a third text representation matrix and a third image representation matrix according to the second text representation matrix, the second image representation matrix and the overall similarity matrix; Using a fourth fusion module in the initial retrieval and representation learning model, the first text representation matrix, the first image representation matrix and the overall cross-modal transfer matrix are fused to obtain a fused representation matrix; Mapping the fused representation matrix into a fused continuous-valued hash code using a linear layer in the initial retrieval and representation learning model; Inputting the fused continuous value hash code into the SIGN function in the initial retrieval and representation learning model to obtain a second fused discrete hash code; Performing a self-similar transformation on the fused continuous-value hash code to obtain a fused hash representation matrix; Using the fully connected layer in the initial retrieval and representation learning model, combining the third image representation matrix and the fusion representation matrix, a first single-path hash representation matrix is ​​obtained; Using the fully connected layer in the initial retrieval and representation learning model, combining the third text representation matrix and the fusion representation matrix, a second single-way hash representation matrix is ​​obtained; Combining the first single-way hash representation matrix and the second single-way hash representation matrix to obtain a multi-way joint hash representation matrix; Based on the loss function, a loss function value is obtained according to the first single-way hash representation matrix, the second single-way hash representation matrix, the overall similarity matrix, the fused hash representation matrix, the multi-way joint hash representation matrix and the first fused discrete hash code; Based on the loss function value, the initial retrieval and representation learning model is optimized.

3. The cross-modal hash retrieval method based on the attention mechanism according to claim 2 is characterized in that: The using of the fourth fusion module in the initial retrieval and representation learning model to fuse the first text representation matrix, the first image representation matrix and the overall cross-modal transfer matrix to obtain a fused representation matrix includes: Using the splicing submodule of the fourth fusion module in the initial retrieval and representation learning model, splicing the first text representation matrix and the first image representation matrix to obtain an overall representation matrix; The overall representation matrix and the overall cross-modal transfer matrix are input into the point multiplication submodule of the fourth fusion module in the initial retrieval and representation learning model to obtain the fused representation matrix.

4. The cross-modal hash retrieval method based on the attention mechanism according to claim 2 is characterized in that: The step of inputting the text feature value matrix and the image feature value matrix into the similarity module in the initial retrieval and representation learning model to obtain a text self-similarity matrix and an image self-similarity matrix comprises: Inputting the text feature value matrix and the image feature value matrix into a similarity module in the initial retrieval and representation learning model; Using the normalization submodule of the similarity module in the initial retrieval and representation learning model, the text feature value matrix and the image feature value matrix are normalized to obtain a text feature vector and an image feature vector; The first dot product submodule of the similarity module in the initial retrieval and representation learning model is used to perform dot product calculations on the text feature vector and the image feature vector, respectively, to obtain the text self-similarity matrix and the image self-similarity matrix.

5. The cross-modal hash retrieval method based on the attention mechanism according to claim 2 is characterized in that: The method of using the first fusion module in the initial retrieval and representation learning model to fuse the text self-similarity matrix and the image self-similarity matrix to obtain an overall similarity matrix includes: Using the first weighted fusion submodule of the first fusion module in the initial retrieval and representation learning model, the text self-similarity matrix and the image self-similarity matrix are linearly weightedly fused to obtain a first similarity matrix; Using the smoothing submodule of the first fusion module in the initial retrieval and representation learning model, the first similarity matrix is ​​smoothed to obtain a second similarity matrix; Using the second weighted fusion submodule of the first fusion module in the initial retrieval and representation learning model, the first similarity matrix and the second similarity matrix are linearly weightedly fused to obtain a third similarity matrix; Using the matrix calculation submodule of the first fusion module in the initial retrieval and representation learning model, a Laplace matrix is ​​calculated according to the third similarity matrix; The Laplacian matrix is ​​negated by using the negation submodule of the first fusion module in the initial retrieval and representation learning model to obtain the overall similarity matrix.

6. The cross-modal hash retrieval method based on the attention mechanism according to claim 2 is characterized in that: The method utilizes the cross-modal attention mechanism in the initial retrieval and representation learning model to fuse the first text representation matrix and the first image representation matrix to obtain a second text representation matrix and a second image representation matrix, including: Using the second dot product module of the cross-modal attention mechanism in the initial retrieval and representation learning model, taking the first text representation matrix as a first query matrix, taking the first image representation matrix as a first key matrix and a first key and value matrix, and calculating a first attention score; Using a first soft attention weighting module of the cross-modal attention mechanism in the initial retrieval and representation learning model, soft attention weighting is performed on the first key matrix and the first key and value matrix according to the first attention score to obtain the second text representation matrix; Using the third dot product module of the cross-modal attention mechanism in the initial retrieval and representation learning model, taking the first image representation matrix as the second query matrix, taking the first text representation matrix as the second key matrix and the second key and value matrix, a second attention score is calculated; The second soft attention weighting module of the cross-modal attention mechanism in the initial retrieval and representation learning model is utilized to perform soft attention weighting on the second key matrix and the second key and value matrix according to the second attention score to obtain the second image representation matrix.

7. The cross-modal hash retrieval method based on the attention mechanism according to claim 2 is characterized in that: The loss function includes: ， ， ， ， ， ， Where n represents the number of training samples. is the element in the i-th row and j-th column of the overall similarity matrix, is the element in the i-th row and j-th column of the fused hash representation matrix, represents the element of the i-th row and j-th column of the multi-way joint hash representation matrix, is a hyperparameter to control the weight, c represents the length of the hash code, and are the j-th bits of the two hash codes generated according to the first one-way hash representation matrix and the second one-way hash representation matrix in the i-th training sample, respectively. is the jth hash bit in the first fused discrete hash code of the i-th training sample.

Images