Cross-modal hash retrieval method based on fusion attention mechanism and DenseNet network

By introducing a fusion attention mechanism and DenseNet network in cross-modal hash retrieval, image feature learning is optimized, and the problem of not considering the internal relationship of modal data in the existing technology is solved, and the retrieval accuracy and accuracy are improved.

CN114817581BActive Publication Date: 2024-11-15CHONGQING UNIV OF POSTS & TELECOMM
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202210546822.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-05-18
Publication Date
2024-11-15
Estimated Expiration
2042-05-18

AI Technical Summary

Technical Problem

The existing cross-modal hash retrieval method does not consider the semantic relationships within the modal data, and the image feature learning network has optimization space, which leads to the extracted modal data features not being fine enough, affecting the retrieval accuracy.

Method used

Using the five-tuple cross-modal hash retrieval method based on the fusion attention mechanism and the DenseNet network, the image feature learning is optimized, more refined modal data features are extracted, and the loss is achieved. Loss terms within and between modals are added to the function, and a five-tuple structure is established to enhance semantic relationship measurements.

Benefits of technology

The search accuracy and accuracy of cross-modal hash retrieval is improved, and the quality of the search results is improved by taking into account both the semantic relationships between the modals and within the modals.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114817581B_ABST
    Figure CN114817581B_ABST
Patent Text Reader

Abstract

The present invention relates to a cross-modal hash retrieval method based on a fusion attention mechanism and a DenseNet network, and belongs to the field of cross-modal retrieval. The method comprises: S1: preprocessing images and texts in a data set; S2: extracting modal data features; S3: hash code mapping: respectively passing the image features and text features extracted in step S2 through two layers of fully connected layers to obtain low-dimensional features of the same dimension; S4: establishing a loss function based on a quintuple structure; S5: inputting quintuple data to train a retrieval model; S6: using the trained retrieval model to perform a cross-modal hash retrieval task. The present invention improves the precision of extracting modal data features and improves the model retrieval accuracy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of cross-modal retrieval and relates to a cross-modal hash retrieval method based on a fusion attention mechanism and a DenseNet network. Background Art

[0002] With the rapid development of the Internet and information technology, multimodal data in different media forms has exploded, including many media forms such as images, text, audio and video. Massive amounts of multimedia data are generated on the Internet every moment. Traditional single-modal retrieval is far from meeting the daily information retrieval needs of users. Efficient retrieval between multiple modal data describing the same thing has become a research hotspot in the field of information retrieval. Driven by this application background, cross-modal retrieval has developed rapidly. Cross-modal retrieval refers to retrieving and returning other modal data related to its semantics based on query modal data. Due to the characteristics of low storage requirements and fast calculation, hashing methods have received widespread attention and application in the field of cross-modal retrieval in recent years. Hash methods are an effective way to solve the semantic gap between cross-modal data. It embeds the low-dimensional features of the feature-extracted modal data into a fixed-length hash code, and then performs a bitwise XOR operation on the hash codes of the query modal data and the retrieval modal data with the same length. Through this operation, we can quickly obtain the distance between the query modality data and the retrieved modality data in the Hamming space, and then obtain the semantic similarity between the data.

[0003] Cross-modal hashing retrieval methods can be divided into three types: supervised cross-modal hashing, unsupervised cross-modal hashing, and semi-supervised cross-modal hashing, depending on whether there are supervised labels when training the model. Among them, unsupervised cross-modal hashing mainly uses the characteristics and data structure of different modal data for learning; supervised cross-modal hashing uses the manually annotated information in the data set for purposeful training, which can obtain better retrieval results than unsupervised cross-modal hashing; the semi-supervised cross-modal hashing training method is to first use part of the labeled data for supervised training, and then use the unlabeled data for unsupervised training. Since supervised training methods have better retrieval results than other training methods, supervised cross-modal hashing based on deep learning has attracted the attention of researchers.

[0004] At present, many supervised cross-modal hash retrieval methods based on deep learning have emerged. However, these methods generally have some defects. For example, the relationship between modal data is not taken into account, but the relationship between different modal data is measured only from the perspective of the outside of things. In fact, there are also rich semantic associations within the same modal data. The semantic information between these modal data of the same category can be used as the index basis for cross-modal hash retrieval. At present, many cross-modal retrieval methods still use image feature learning networks based on the VGG-Net structure in feature learning networks. However, with the rapid development of deep learning technology, deep learning models with excellent performance are constantly produced. Therefore, there are problems such as the modal data feature learning network needs to be optimized and the extracted modal data features are not fine enough, which affects the retrieval accuracy.

[0005] Therefore, a new cross-modal hash retrieval method is urgently needed to solve the above problems. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a five-tuple image-text cross-modal hash retrieval method based on a fusion attention mechanism and a DenseNet network to solve the following problems existing in the current triple-based methods: the semantic relationship between the modal data is not taken into account, the image feature learning network needs to be optimized, and the extracted modal data features are not refined enough to affect the accuracy of the retrieval task.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A cross-modal hash retrieval method based on the fusion of attention mechanism and DenseNet network introduces the DenseNet network into the image feature learning network, and uses its deep structure and connection on the feature channel to achieve feature reuse, strengthen the transmission of features in the deep network, and alleviate the problem of gradient disappearance. At the same time, the SENet attention mechanism is integrated into the feature learning network to improve the refinement of the extracted modal data features and improve the model retrieval accuracy. The method specifically includes the following steps:

[0009] S1: Preprocess the images and texts in the dataset;

[0010] S2: Extracting modal data features, specifically including: using an image feature learning network based on fusion channel domain attention and DenseNet network structure to extract image features, and using a text feature learning network based on the BoW model to extract text features;

[0011] S3: Hash code mapping: The image features and text features extracted in step S2 are respectively passed through two fully connected layers to obtain low-dimensional features of the same dimension;

[0012] S4: Establish a loss function L based on the quintuple structure;

[0013] minL=L1+L2+γL3+L4

[0014] Among them, γ is an adjustable hyperparameter used to adjust the proportion of L3 in the loss function L, and L1 represents the text feature F T With image feature F I The loss between modal data is measured using the boundary-based hinge loss function and the sigmoid cross entropy loss function. L2 represents the text feature F T The loss between the internal modal data is measured using the boundary-based hinge loss function and the sigmoid cross entropy loss function. L3 represents the loss between the hash code F and the low-dimensional feature H. L4 represents the loss obtained by orthogonal regularization of the weight W and the bias B.

[0015] S5: Use a large batch of quintuple data to train the retrieval model;

[0016] The retrieval model includes an image feature learning network, a text feature learning network, a hash layer and a query retrieval process;

[0017] The five-tuple data includes: query anchor samples, negative samples of the same modality, positive samples of the same modality, negative samples of different modalities, and positive samples of different modalities; the training batch size, i.e., batch_size, is set to 8192, the number of training Epochs is set to 2000, and the training data is input into the retrieval model in a five-tuple mode, and the model is trained by optimizing the total objective function L and updating parameters such as weights and biases;

[0018] S6: Use the trained retrieval model to perform the cross-modal hash retrieval task.

[0019] Further, in step S3, the low-dimensional features are mapped to hash codes of the same number of bits by a threshold function, wherein the first fully connected layer uses a tanh activation function, and the second layer uses a sigmoid function. The process of obtaining low-dimensional features of the same dimension can be described as:

[0020]

[0021] Among them, the tanh activation function used in the C1 layer is and They are the weights and biases of the parameters in the C1 layer. The C2 layer uses the sigmoid activation function. and are the weights and biases of the parameters in the C2 layer, respectively. f represents the output of the feature extraction part of the modal data, namely, the output of the image features by the DenseNet201 image feature learning network fused with SENet attention and the output of the text data features by the text feature learning network based on the BoW model.

[0022] The threshold function is:

[0023] Further, in step S4, the text feature F T With image feature F I The calculation formula of the loss L1 between modal data is:

[0024] L1=l1+λ(l2+l3)

[0025]

[0026]

[0027]

[0028] Among them, l1 represents the loss of the distance between modalities; λ is an adjustable hyperparameter used to control the proportion of loss terms l2 and l3 to the L1 loss term; n represents the number of quintuple samples, and the hyperparameter β represents the measure of and The boundary value of the loss between Represents the low-dimensional features of the query text F qT The low-dimensional features of the retrieved positive image The distance between Represents the low-dimensional features of the query text F qT The low-dimensional features of the retrieved negative image The distance between

[0029]

[0030]

[0031]

[0032]

[0033] in, Indicates T i + and I i + The distance between i + and I i + They represent the i-th positive text and the i-th positive image respectively.

[0034] Further, in step S4, the text feature F T The calculation formula of the loss L2 between the modal data is:

[0035] L2=l4+λl5

[0036]

[0037]

[0038]

[0039]

[0040] Among them, λ is an adjustable hyperparameter used to adjust the ratio of the loss term l5 to the L2 loss term; the hyperparameter β represents the measure of and The boundary value of the loss between, n represents the number of quintuple samples, Represents low-dimensional features of query text Low-dimensional features of the retrieved positive text The distance between Represents low-dimensional features of query text Low-dimensional features of the retrieved negative text The distance between.

[0041] Further, in step S4, the calculation formula for measuring the loss L3 between the hash code F and the low-dimensional feature H is:

[0042]

[0043]

[0044]

[0045] Among them, n represents the number of quintuple samples, Low-dimensional features representing query text Its corresponding hash code And the low-dimensional features F of the retrieved text within the same modality T Its corresponding hash code H T The distance between Represents the low-dimensional features F of the retrieved image I Its corresponding hash code H I The distance between i + and T i - Represents the i-th positive example text and negative example text respectively; I i+ and T i - represent the i-th positive image and negative image respectively.

[0046] Further, in step S4, the calculation formula of the loss L4 obtained by orthogonal regularization of the weight W and the bias B is:

[0047]

[0048] Among them, W tra represents the transposed matrix of the weight matrix W, I ide represents the identity matrix, θ and ω are adjustable hyperparameters, ||·|| Fro It represents the Frobenius norm of the matrix, the subscript T represents text data, and the subscript I represents image data.

[0049] Further, in step S5, the five-tuple data is input to train the retrieval model, which specifically includes the following steps:

[0050] S51: Randomly initialize W and B, set the training batch size and training Epoch;

[0051] S52: For each query text data Randomly select m images related to its semantics Image data that is not semantically relevant Text data T related to its semantics i + , text data T that is not semantically relevant i -来 The input data is composed of m pairs of five-tuples; among them, the five-tuple input data for the training model needs to be re-selected every 30% of the total data processed, the text retrieval model is trained and the weight W and the bias term B are updated by optimizing the total objective function L;

[0052] S53: For each query image data Randomly select m images related to its semantics Image data I that is not related to its semantics i - , text data T related to its semantics i + , text data T that is not semantically relevant i - To form m pairs of five-tuple input data; among them, every 30% of the total data processed needs to re-select the five-tuple input data for the training model, train the image retrieval model and update the weight W and bias term B by optimizing the total objective function L.

[0053] Further, in step S6, a cross-modal hash retrieval task is performed using a trained retrieval model, specifically including: inputting the modal data such as images and texts to be queried into the trained retrieval model, and obtaining the corresponding hash code after being processed by the model, and then performing an XOR operation on these hash codes to obtain the Hamming distance between the modal data, and sorting these Hamming distances to obtain the semantic relevance between the query modal data and the modal data in the retrieval database. The modal data ranked in the top column has a higher degree of semantic similarity, and the modal data ranked in the top column is taken as the retrieval result of this query.

[0054] The beneficial effects of the present invention are as follows: by adopting the quintuple pattern, the present invention can establish semantic relations between modal data from both inter-modal and intra-modal aspects, and the semantic information within modal data of the same category is also used as the index basis for cross-modal hash retrieval; the image feature learning network structure is optimized by using the DenseNet dense network, which can more efficiently identify and extract image data features than the previous VGG-Net; at the same time, the SENet attention in the channel domain attention mechanism is integrated into the image feature learning network, so that the extracted modal data features are more refined, and the trained model significantly improves the retrieval result accuracy of cross-modal hash retrieval and improves the precision of retrieval.

[0055] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0057] Figure 1 Schematic diagram of the five-tuple measuring the semantic relationship between the modality and the modality;

[0058] Figure 2 This is a model framework structure diagram of the cross-modal hash retrieval method of five-tuple images and texts based on the DenseNet network that integrates the attention mechanism of the present invention. DETAILED DESCRIPTION

[0059] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0060] See also Figure 1-2 The present invention provides a cross-modal hash retrieval method for five-tuple images and texts based on a DenseNet network and integrating an attention mechanism, comprising the following steps:

[0061] Step 1: Preprocess the images and text in the dataset.

[0062] The specific implementation method is described using the Wikipedia dataset and the MIRFlickr-25K dataset as examples, the same below. The Wikipedia dataset is divided, and 2173 image / text data pairs are randomly selected as training sets and retrieval sets, and the remaining 693 data pairs are used as test sets. The MIRFlickr-25K dataset is preprocessed, and tags that appear in at least 20 images are selected as the vocabulary of the bag-of-words model BoW. Data pairs containing words that are not included in the vocabulary in the entire dataset, images that lack text descriptions, and image / text data pairs that lack category labels are removed. After processing, there are still 20,819 pairs of data in the dataset that meet the experimental requirements. 1041 pairs of data pairs are randomly selected as the test set, and the remaining 19,778 pairs of data pairs are used as the retrieval dataset. 5,000 pairs of data pairs are randomly selected from the retrieval set as the training set.

[0063] Step 2: Modal data feature extraction.

[0064] like Figure 2 As shown in the figure, an image feature learning network based on the DenseNet network structure and fused with channel domain attention is used to extract 1920-dimensional features of image data, and a text feature learning network based on the bag-of-words model BoW is used to extract text features. For the Wikipedia dataset, the text data is represented as a 1000-dimensional vector, and for the MIRFlickr-25K dataset, the text data is represented as a 1386-dimensional vector.

[0065] Step 3: Hash code mapping.

[0066] like Figure 2As shown in the figure, the image features and text features extracted by the feature learning network are respectively passed through two fully connected layers to obtain low-dimensional features of the same dimension. These low-dimensional features are mapped to hash codes of the same number of bits by the threshold function. The first fully connected layer uses the tanh activation function, and the second layer uses the sigmoid function. The process of obtaining low-dimensional features of the same dimension can be described as:

[0067]

[0068] Among them, the Tanh activation function used in the C1 layer is and are the weights and biases of the parameters in layer C1. Layer C2 uses the sigmoid activation function. and are the weights and biases of the parameters in the C2 layer, respectively. Here, f represents the output of the modal data feature extraction part, that is, the output of the DenseNet201 image feature learning network fused with SENet attention for image features and the output of the text feature learning network based on the bag-of-words model BoW for text data features. The threshold function used is shown in the following formula:

[0069]

[0070] Step 4: Establish a loss function L based on the quintuple structure.

[0071] like Figure 1 As shown in the figure, the loss function L based on the quintuple structure mainly includes four parts: (1) text feature F T With image feature F I The loss between modal data is measured using the boundary-based hinge loss function and the sigmoid cross entropy loss function, and the loss function L1 can be obtained; (2) text feature F T The loss between the internal modal data of the proposed method is measured using the boundary-based hinge loss function and the sigmoid cross entropy loss function, and the loss function L2 can be obtained. (3) The loss between the hash code F and the low-dimensional feature H can be measured to obtain the loss function L3. (4) The orthogonal regularization of the weight W and the bias B can be performed to obtain the loss function L4.

[0072] (1) Calculate text feature F T With image feature F I Loss between modal data

[0073]

[0074]

[0075] in, Represents low-dimensional features of query text The low-dimensional features of the retrieved positive image The distance between Represents low-dimensional features of query text The low-dimensional features of the retrieved negative image The hinge loss function is used to evaluate the loss of the inter-modal distance.

[0076]

[0077] Among them, n represents the number of five-tuple samples input to the cross-modal hash retrieval method, and the hyperparameter β represents the measure of and The boundary value of the loss between . It can be reduced by operation The value increases simultaneously The value makes the loss value of l1 decrease. Ideally, after continuous optimization, β will eventually be and Therefore, the sigmoid cross entropy loss function is used to deal with this binary classification problem and continue to optimize and The loss term will be as follows:

[0078]

[0079]

[0080] After analysis, it can be found that The optimization result cannot approach the ideal value of 0, so we directly use The whole is treated as a loss term. If loss2 meets the optimization expectation, the sigmoid cross entropy loss function is used to process it. For this reason, the loss term l2 can be obtained

[0081]

[0082] In the input five-tuple data sample, according to T i + The relationship between and T i + There is also a relationship. Consider and T i + The distance between is also added to the objective function. and T i - There is no direct relationship to be measured.

[0083]

[0084] Combining the above loss terms, we can get the summarized inter-modal loss function:

[0085] L1=l1+λ(l2+l3)

[0086] Where λ is an adjustable hyperparameter used to control the proportion of loss terms l2 and l3 to the L1 loss term.

[0087] (2) Calculate text feature F T The loss between the modal data

[0088]

[0089]

[0090] The distance between the low-dimensional features of the query text and the retrieved text in the same modality data is measured by the above two formulas. Represents low-dimensional features of query text Low-dimensional features of the retrieved positive text The distance between Represents low-dimensional features of query text Low-dimensional features of the retrieved negative text The distance between them. The boundary-based hinge loss function is also used to evaluate the loss of intra-modal distance:

[0091]

[0092] Among them, the hyperparameter β represents the measure of and The boundary value of the loss between modal data and n represents the number of 5-tuple samples. Since the processing logic is the same as that of calculating the loss between modal data, the following loss terms can be obtained:

[0093]

[0094] The total intra-modal loss term is:

[0095] L2=l4+λl5

[0096] Among them, λ is an adjustable hyperparameter used to control the ratio of the loss term l5 to the L2 loss term.

[0097] (3) Measuring the loss between the hash code F and the low-dimensional feature H

[0098] The unique discrete nature of hash codes means that there will inevitably be error loss in the process of converting low-dimensional features to hash codes. To address this problem, consider adding the distance between the hash code and the feature to the objective function. Through the constraint of this loss term, the loss of features in the process of converting to hash codes can be reduced, so that the hash code can better represent the characteristics of the modal data. The trade-off between the two is shown in the following formula:

[0099]

[0100]

[0101] in, Low-dimensional features representing query text Its corresponding hash code And the low-dimensional features F of the retrieved text within the same modality T The distance between its corresponding hash code Ht; Represents the low-dimensional features F of the retrieved image I Its corresponding hash code H I The distance between them. Combining the above two formulas, we can get the loss term between the hash code H and the low-dimensional feature F:

[0102]

[0103] (4) Orthogonal regularization of weight W and bias B

[0104] Regularization is introduced to reduce the overfitting problem during model training and improve the generalization ability of the model. Because the orthogonal matrix has the property of orthogonal transformation and norm preservation, the norm relationship between vectors multiplied by the orthogonal matrix will not change, so the orthogonal regularization process is introduced to orthogonally regularize the weight matrix W and use it as a loss term. Since the bias B does not participate in the multiplication between vectors, this process is not performed. The Frobenius norm of the matrix is ​​used to measure this part.

[0105]

[0106] Among them, W tra represents the transposed matrix of the weight matrix W, I ide represents the identity matrix, θ and ω are tunable hyperparameters.

[0107] Adding the loss functions of these four parts can get the total target loss function L:

[0108] minL=L1+L2+γL3+L4

[0109] Among them, γ is an adjustable hyperparameter used to adjust the proportion of L3 in the loss function L.

[0110] Step 5: Use a large batch method to input quintuple data to train the model.

[0111] The quintuple includes: query anchor samples, negative samples of the same modality, positive samples of the same modality, negative samples of different modalities, and positive samples of different modalities. The training batch size, batch_size, is set to 8192, the number of training Epochs is set to 2000, and the training data is input into the retrieval model in a quintuple mode. The model is trained by optimizing the total objective function L and updating parameters such as weights and biases. The DenseNet network-based quintuple image-text cross-modal hash retrieval method with fusion attention mechanism mainly includes two retrieval tasks: text query image and image query text. To this end, the model is trained for these two tasks respectively. The specific process is as follows:

[0112] Step 5.1: Randomly initialize W and B, set the training batch size to 8192, and set the training epoch to 2000. For the setting of hyperparameters, the number of hash code bits is set to 16, 32, and 64, and the corresponding β is 6, 8, and 10, respectively, θ = 0.000l, γ = λ = ω = 0.01. In order to reduce the number of hyperparameters and speed up the convergence of the model, γ, λ, and ω are set to the same value. The Adam optimization algorithm is used to optimize and update the parameters in the model. The learning rate on all data sets is set to the default 0.0001 of the Adam algorithm.

[0113] Step 5.2: For each query text data Randomly select m images related to its semantics Image data that is not semantically relevant Text data T related to its semantics i + , text data T that is not semantically relevant i - To form m pairs of quintuples Input data; where the five-tuple input data for the training model is reselected every time 30% of the total data is processed. Train the text retrieval task network and update the weight W and bias term B by optimizing the overall objective function L.

[0114] Step 5.3: For each query image data Randomly select m images related to its semantics Image data that is not semantically relevant Text data T related to its semantics i + , text data that is not semantically relevant To form m pairs of quintuples Input data; where the five-tuple input data for the training model is re-selected every time 30% of the total data is processed. Train the image retrieval task network and update the weight W and bias term B by optimizing the overall objective function L.

[0115] Step 6: Use the trained model to perform the cross-modal hash retrieval task.

[0116] like Figure 2 As shown, the present invention inputs the modal data such as images and texts to be queried into a trained retrieval model, and the corresponding hash code can be obtained after being processed by the model. Then, the Hamming distance between the modal data can be obtained by performing an XOR operation on these hash codes. The semantic correlation between the queried modal data and the modal data in the retrieval database can be obtained by sorting these distances. The higher the semantic similarity of the modal data ranked in the front, the modal data ranked in the front is taken as the retrieval result of this query.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A cross-modal hash retrieval method based on fusion attention mechanism and DenseNet network, characterized in that: The method comprises the following steps: S1: Preprocess the images and texts in the dataset; S2: Extracting modal data features, specifically including: using an image feature learning network based on fusion channel domain attention and DenseNet network structure to extract image features, and using a text feature learning network based on the BoW model to extract text features; S3: Hash code mapping: The image features and text features extracted in step S2 are respectively passed through two fully connected layers to obtain low-dimensional features of the same dimension; S4: Establish a loss function L based on the quintuple structure; minL=L1+L2+γL3+L4 Among them, γ is an adjustable hyperparameter used to adjust the proportion of L3 in the loss function L, and L1 represents the text feature F T With image feature F I The loss between modal data is measured using the boundary-based hinge loss function and the sigmoid cross entropy loss function. L2 represents the text feature F T The loss between the internal modal data is measured using the boundary-based hinge loss function and the sigmoid cross entropy loss function. L3 represents the loss between the hash code F and the low-dimensional feature H. L4 represents the loss obtained by orthogonal regularization of the weight W and the bias B. Text feature F T With image feature F I The calculation formula of the loss L1 between modal data is: L1=l1+λ(l2+l3) Among them, l1 represents the loss of the distance between modalities; λ is an adjustable hyperparameter used to control the proportion of loss terms l2 and l3 to the L1 loss term; n represents the number of quintuple samples, and the hyperparameter β represents the measure of and The boundary value of the loss between Represents low-dimensional features of query text The low-dimensional features of the retrieved positive image The distance between Represents low-dimensional features of query text The low-dimensional features of the retrieved negative image The distance between in, express and The distance between and Represent the i-th positive example text and the i-th positive example image respectively; Text feature F T The calculation formula of the loss L2 between the modal data is: L2=l4+λl5 Among them, λ is an adjustable hyperparameter used to adjust the ratio of the loss term l5 to the L2 loss term, and the hyperparameter β represents the measure of and The boundary value of the loss between, n represents the number of quintuple samples, Represents low-dimensional features of query text Low-dimensional features of the retrieved positive text The distance between Represents low-dimensional features of query text Low-dimensional features of the retrieved negative text The distance between The calculation formula for measuring the loss L3 between the hash code F and the low-dimensional feature H is: Among them, n represents the number of quintuple samples, Low-dimensional features representing query text Its corresponding hash code And the low-dimensional features F of the retrieved text within the same modality T Its corresponding hash code H T The distance between Represents the low-dimensional features F of the retrieved image I Its corresponding hash code H I The distance between and Represent the i-th positive example text and negative example text respectively; and Represent the i-th positive image and negative image respectively; S5: input quintuple data to train the retrieval model; The retrieval model includes an image feature learning network, a text feature learning network, a hash layer and a query retrieval process; The five-tuple data includes: query anchor samples, negative samples of the same modality, positive samples of the same modality, negative samples of different modalities, and positive samples of different modalities; S6: Use the trained retrieval model to perform the cross-modal hash retrieval task.

2. The cross-modal hash retrieval method according to claim 1, characterized in that: In step S3, the low-dimensional features are mapped to hash codes of the same number of bits by a threshold function, wherein the first fully connected layer uses a tanh activation function and the second layer uses a sigmoid function. The process of obtaining low-dimensional features of the same dimension is described as follows: Among them, the tanh activation function used in the C1 layer is and They are the weights and biases of the parameters in the C1 layer. The C2 layer uses the sigmoid activation function. and are the weights and biases of the parameters in the C2 layer, respectively, and f represents the output of the feature extraction part of the modal data; The threshold function is:

3. The cross-modal hash retrieval method according to claim 1, characterized in that: In step S4, the loss L4 obtained by orthogonal regularization of weights ... and bias B is calculated as: Among them, W tra represents the transposed matrix of the weight matrix W, I ide represents the identity matrix, θ and ω are hyperparameters that can be adjusted, ‖·‖ Fro It represents the Frobenius norm of the matrix, the subscript T represents text data, and the subscript I represents image data.

4. The cross-modal hash retrieval method according to claim 1, characterized in that: In step S5, the five-tuple data is input to train the retrieval model, which specifically includes the following steps: S51: Randomly initialize W and B, set the training batch size and training Epoch; S52: For each query text data Randomly select m images related to its semantics Image data that is not semantically relevant Text data related to its semantics Text data that is not relevant to its semantics To form m pairs of five-tuple input data; among them, every 30% of the total data processed needs to re-select the five-tuple input data for the training model, train the text retrieval model and update the weight W and bias term B by optimizing the total objective function L; S53: For each query image data Randomly select m images related to its semantics Image data that is not semantically relevant Text data related to its semantics Text data that is not relevant to its semantics To form m pairs of five-tuple input data; among them, every 30% of the total data processed needs to re-select the five-tuple input data for the training model, train the image retrieval model and update the weight W and bias term B by optimizing the total objective function L.

5. The cross-modal hash retrieval method according to claim 1, characterized in that: In step S6, a cross-modal hash retrieval task is performed using a trained retrieval model, specifically including: inputting the image and text modal data to be queried into the trained retrieval model, obtaining the corresponding hash code after processing by the model, and then performing an XOR operation on these hash codes to obtain the Hamming distance between the modal data, and sorting these Hamming distances to obtain the semantic relevance between the query modal data and the modal data in the retrieval database. The modal data ranked in the top column has a higher degree of semantic similarity, and the modal data ranked in the top column is taken as the retrieval result of this query.

Citation Information

Patent Citations

  • Cross-modal hash retrieval method based on triple deep networks

    CN108170755A

  • Image-text cross-modal hash retrieval method based on large-batch training

    CN111209415A

  • Cross-modal pedestrian re-identification method based on difficult quintuple

    CN111597876A

  • Universal cross-modal retrieval model based on deep hash

    CN113076465A

  • Five-tuple entity relationship extraction method and device

    CN114297400A