A deep cross-modal hashing retrieval method based on image enhancement technology

Abstract

The application discloses a deep cross-modal hash retrieval method based on an image enhancement technology, and comprises the following steps: S1, obtaining a target image, obtaining a category embedding vector and a feature embedding vector based on the target image, and obtaining a text feature based on text; S2, obtaining an image embedding vector based on the category embedding vector and the feature embedding vector; S3, training the image embedding vector and the text feature to obtain a retrieval model; and S4, inputting retrieval text into the retrieval model to obtain the target image. The application has the beneficial effect that the average precision of multi-modal hash retrieval is improved by enhancing image-text feature representation.

Description

Technical Field

[0001] This invention belongs to the field of information retrieval, and specifically relates to a deep cross-modal hash retrieval method based on image enhancement technology. Background Technology

[0002] Information retrieval is the primary way users search for and obtain information. Driven by technologies such as big data, artificial intelligence, and social networks, information retrieval technology has been greatly improved. However, the ever-increasing volume of data has increased the difficulty of information retrieval, making it even more challenging to find accurate and comprehensive information. In many retrieval applications, data includes not only text information but also images and other data.

[0003] In existing technologies, multimodal hashing is applied to approximate nearest neighbor retrieval for multimodal data. Cross-Modal Hashing (CMH) is widely used, but the CMH method is independent of hash code generation learning when extracting multimodal features, making it incompatible with hash code learning. Deep Cross-Modal Hashing (DCMH) includes two deep neural networks in its feature learning part, one for image and one for text. The deep neural network for images is modified from Convolutional Neural Networks (CNNs). For text, each segment of text is first split and represented as a vector, which is then used as input to a deep neural network with two fully connected layers. The extracted image and text features are trained together with the hash code generation process, with the overall goal of preserving image-text similarity information in the hash code. Finally, Approximate Nearest Neighbor (ANN) is used to query image-text similarity. However, DCMH lacks the preservation of image-text semantics during feature extraction. In summary, when performing retrieval tasks without semantic representation of images and text, it is often difficult to generate accurate retrieval results based on the searcher's intent. There is also the problem that the average accuracy of multimodal hash retrieval cannot be improved by enhancing the representation of image and text features.

[0004] For example, a Chinese patent document, "Training Method, Encoding Method, Apparatus, and Electronic Equipment for Cross-Modal Hash Models," with publication number CN114398980A and application date of January 13, 2022, describes an invention that uses a cross-modal hash model to perform dimensionality reduction hash encoding on multiple sample pairs, resulting in multiple hash encoding pairs. For each hash encoding pair, a target data hash point with a higher weight is determined from the hash data point pairs at each position within the hash encoding pair. A binary code for the hash encoding pair is determined based on each target hash data point. The total quantization loss of the cross-modal hash model is determined based on the similarity matrix corresponding to each hash encoding pair and multiple sample pairs, as well as the difference between each hash encoding pair and its corresponding binary code. The parameters of the cross-modal hash model are updated based on the total quantization loss. This application can improve the encoding accuracy of the cross-modal hash model, thereby saving computational resources used to calculate the similarity between the encoding results of different samples. However, it cannot improve the average accuracy of multimodal hash retrieval by enhancing the representation of image and text features. Summary of the Invention

[0005] To address the shortcomings of existing technologies in improving the average accuracy of multimodal hash retrieval by enhancing image and text feature representation, this invention proposes a deep cross-modal hash retrieval method based on image enhancement technology, which improves the average accuracy of multimodal hash retrieval by enhancing image and text feature representation.

[0006] The following is the technical solution of the present invention: a deep cross-modal hash retrieval method based on image enhancement technology, comprising the following steps:

[0007] S1: Obtain the target image, obtain the category embedding vector and feature embedding vector based on the target image, and obtain text features based on the text;

[0008] S2: Obtain the image embedding vector based on the category embedding vector and the feature embedding vector;

[0009] S3: The retrieval model is obtained by training the image embedding vector and text features;

[0010] S4: Input the search text into the search model to obtain the target image.

[0011] This scheme adds semantic information to image features, including information about objects within the image and image features. Semantic information is also added to text feature representations, including contextual information and the order of elements within the original text. This results in an enhanced image feature representation, which is trained together with the hash code generation process. This enhanced representation is sensitive to text order, textual context, and the actual content of the image. While preserving image-text similarity information in the generated hash codes, it enables the model to more accurately generate results based on the searcher's intent. By enhancing image-text feature representations, the average accuracy of multimodal hash retrieval is improved.

[0012] As a preferred option, in S1, the EfficientNet model is used to extract features from the target image to obtain a feature embedding vector.

[0013] Preferably, in S1, obtaining the category embedding vector based on the target image includes the following steps:

[0014] S101: Use the InceptionV3 model to obtain the target image classification probability and category label;

[0015] S102: Determine the target image category label based on classification probability and threshold;

[0016] S103: Category labels are generated into category embedding vectors using the word2vec model.

[0017] Preferably, S102 includes the following steps:

[0018] S1021: Set threshold;

[0019] S1022: If the classification probability is greater than the threshold, the embedding vector of the category label corresponding to the classification probability is used as the embedding of the target image; otherwise, the embedding of the target image is set to the zero vector or a preset value.

[0020] Preferably, in S1, obtaining text features based on the text includes the following steps:

[0021] S111: Convert each representation in the text into a corresponding unique ID;

[0022] S112: Obtain semantically enhanced text features by passing each representation through a pre-trained BERT model.

[0023] Preferably, S111 includes the following steps:

[0024] S1111: Decompose the text into several representations;

[0025] S1112: Add to the beginning and end of the sentence to fill in the sentence so that the total length of the sentence is equal to the maximum length;

[0026] S1113: Convert each representation into a corresponding unique ID in the BERT model.

[0027] As a preferred embodiment, in S112, the calculation expression of the BERT model is as follows:

[0028]

[0029] In the above formula, y j For the j-th segment of text, The feature output corresponding to the j-th text segment is [CLS], which is located at the beginning of each text sequence y. In BERT, the output is an aggregated representation of the subsequent text. The [SEP] identifier is used to separate the two inputs.

[0030] Preferably, S2 includes the following steps:

[0031] S21: Concatenate the category embedding vector and the feature embedding vector to obtain the concatenated vector;

[0032] S22: The concatenated vector is passed through a fully connected layer to obtain the image embedding vector.

[0033] As a preferred option, the objective function of S3 is as follows:

[0034]

[0035] stB∈{-1,+1} c×n

[0036]

[0037]

[0038]

[0039] In the above formula, F *i and G is the image embedding vector. *j and For text features, It is image x i binary hash code, It is text y ij The binary hash code, where γ and η are hyperparameters.

[0040] The beneficial effect of this invention is that it improves the average accuracy of multimodal hash retrieval by enhancing the representation of image and text features. Attached Figure Description

[0041] Figure 1 The flowchart of a deep cross-modal hash retrieval method based on image enhancement technology is shown in this invention.

[0042] Figure 2 The present invention provides a flowchart of the model construction for a deep cross-modal hash retrieval method based on image enhancement technology. Detailed Implementation

[0043] The technical solution of the present invention will be further described in detail below through embodiments and in conjunction with the accompanying drawings.

[0044] Example: Figure 1 As shown, a deep cross-modal hashing retrieval method based on image enhancement technology includes the following steps:

[0045] S1: Obtain category embedding vector and feature embedding vector based on the target image, and obtain text features based on the text;

[0046] S2: Obtain the image embedding vector based on the category embedding vector and the feature embedding vector;

[0047] S3: The retrieval model is obtained by training the image embedding vector and text features;

[0048] S4: Input the search text into the search model to obtain the target image.

[0049] In step S1, such as Figure 2 As shown, feature embedding vectors are obtained based on the target image. The EfficientNet model (a network model) is used to extract features from the image, and the output image feature embedding vector is represented as follows: Depending on the size of the target image dataset and the limitations of computational resources, different EfficientNet models are employed to flexibly adapt to different scenarios. Compared to traditional CNNs (Convolutional Neural Networks), the EfficientNet model has higher accuracy and better generalization ability, achieving better results in feature extraction tasks, while also preserving, to some extent, the high-level semantic information contained in the image.

[0050] The category embedding vector is obtained based on the target image. A CNN classifier is trained by fine-tuning the InceptionV3 model (a network model) on the ImageNet dataset (a computer vision dataset). This classifier incorporates object information contained in the image, resulting in the image category label z. i And the corresponding classification probability P(z|x). For the classification result, a threshold theta is set. If the classification probability is greater than the set threshold, the embedding vector of its corresponding class label is used as the embedding of the image; otherwise, its embedding is set to a zero vector or other pre-set value.

[0051] The Transformer model (a transformer model) is used to train the word2vec model (a word vector model). Specifically, the encoder of a Transformer model can be viewed as a function that maps a sequence of words w_1, w_2, ..., w_n to a sequence of vectors h_1, h_2, ..., h_n in a high-dimensional vector space, i.e.:

[0052] h1,h2,...,h n =Transformer(w1,w2,...,w n ) (Formula 1)

[0053] These vector sequences are used as training samples to train a word2vec model using the CBOW algorithm. For the CBOW algorithm, the average of the context vectors for each word can be used as input, and a softmax classifier is used to predict the word. This model predicts the target word based on the context words, where each word's context vector is w_0. i Obtain its word2vec vector f in its joint high-dimensional vector space W (w i The image's category label y is used to generate an image category embedding vector f using the word2vec model. W (z),

[0054] Text features are obtained based on the text. Each token (representation) in the text Y is converted into a corresponding unique ID, and a semantically enhanced text feature representation is obtained through a pre-trained BERT (Bidirectional Encoder Representation from Transformers) model. Includes the following steps:

[0055] First, the `tokens_to_ids` method of BertTokenizer is used to obtain the unique ID corresponding to each token in the text Y. This method decomposes the text Y into tokens. A [CLS] token is added to the beginning of the sentence, a [SEP] token is added to the end, and the sentence is padded with a [PAD] token to make the total length equal to the maximum length. Each token is then converted into the corresponding ID in the BERT model.

[0056] Then, the tokens generated in the previous step are passed through a pre-trained BERT model to obtain semantically enhanced text feature representations. BERT is a pre-trained language representation model. It emphasizes that instead of using traditional one-way language models (such as LSTM) or shallow concatenation of two one-way language models (such as Bi-LSTM) for pre-training, it uses a Masked Language Model (MLM) to generate deep bi-directional language representations. A BERT model requires a significant amount of time to train on a large dataset, so a pre-trained BERT is typically used, connected to an additional output layer to obtain a model for text feature extraction tasks. The calculation formula for the BERT model is shown in Equation (2).

[0057]

[0058] In the above formula, y j For the j-th segment of text, The feature output corresponding to the j-th text segment is [CLS], which is located at the beginning of each text sequence y. In BERT, the output is an aggregated representation of the subsequent text. The [SEP] identifier is used to separate the two inputs.

[0059] Based on the calculation method in formula (2), the semantically enhanced text feature representation is obtained. This is because each token input into BERT is represented as a sum of three embeddings: token embeddings, segment embeddings, and position embeddings. This means that each word, after being processed by the BERT pre-trained model, not only contains its own information but also the global semantic information and positional information of the text.

[0060] In step S2, the image embedding vector is obtained based on the category embedding vector and the feature embedding vector. The image category embedding vector is then concatenated with the image feature embedding vector generated by the EfficientNet model to obtain the semantically enhanced embedding vector of the image, i.e., f(x) = [F...]. E (x),f W (z)]. Then, a fully connected layer is used to obtain semantically enhanced image embedding vectors.

[0061] In step S3, the image embedding vector and text features are trained to obtain the retrieval model. The text feature representation obtained in step S1 and the semantically enhanced image obtained in step S2 are trained together with the hash code learning process in the DCMH method to obtain an enhanced cross-modal retrieval model. The main approach is as follows:

[0062] The DCMH method is an end-to-end learning framework that seamlessly integrates feature learning and hash code learning. Compared to previous methods, it is more compatible with hash code learning and achieves better performance. The goal of the DCMH method is to obtain binary hash codes that preserve image-text similarity information. The objective function is shown in equation (3).

[0063]

[0064] stB∈{-1,+1} c×n (Formula 3)

[0065] in, F *i and G is the image embedding vector. *j and For text features. It is image x i binary hash code, It is text y ij The binary hash code. γ and η are hyperparameters. The first term... It is the negative log-likelihood of cross-modal similarity, which optimizes cross-modal similarity by preserving cross-modal similarity in S with both image and text feature representations; the second term Used to optimize the quality of binary hash codes. The third term η(||F1||F 2 +||G1||F 2 ) is a hash code balancing term used to keep each bit of the hash code balanced across all training points, avoiding the situation where some bits are overactivated.

[0066] All parameters are learned from this objective function, which integrates semantically enhanced feature learning and hash code learning into the same deep learning framework, ultimately obtaining a deep cross-modal hash retrieval model based on image enhancement.

[0067] In step S4, the search text is input into the retrieval model to obtain the target image. By inputting the search text into the model, target images with high similarity are matched. If there are several target images, they are displayed in descending order of similarity.

[0068] After obtaining the feature representation X of the image and the feature representation Y of the text, for each image xi of the image modality, an image classifier using a Convolutional Neural Network (CNN) fine-tuned with the InceptionV3 model is used to obtain the corresponding category label and classification probability. Then, classification results greater than a set threshold are extracted. For example, if an image has classification results of: apple -0.6, strawberry -0.4, ball -0.1, and the classification probability threshold is set to 0.5, then apple is considered the classification result for that image. This classification result is then converted into word vectors using a word2vec model trained with a Transformer model. The representation of the embedded image generated using the image feature extractor of the EfficientNet model. The images are concatenated and finally processed through a fully connected layer to obtain a semantically enhanced image feature representation. For each segment of text y in the text modality j Each token is converted into its corresponding unique ID, and a text feature representation containing contextual semantic information is obtained through a pre-trained BERT model.

[0069] The image feature representation enhancement method according to this invention adds semantic information to image features, encompassing both information about objects contained in the image and image feature information. Previous methods only included image feature information. The semantically enhanced image embedding in this method improves the accuracy of image-text retrieval.

[0070] The text feature representation enhancement method according to this invention adds semantic information to the text feature representation, including contextual information and order information in the original text. Previous methods only included information about whether a word exists in the text, without the above information. The enhanced retrieval method, through this approach, will produce the desired "apple" results when people search for something like "apple".

[0071] The deep cross-modal hash retrieval method based on image enhancement technology described in this invention simultaneously obtains enhanced image feature representations, which are then used in conjunction with the hash code generation process for training. Previous methods used CNN-like models to extract image features, generated text features using bag-of-words vectors through fully connected layers, and jointly trained with hash code generation to retrieve query-related results to some extent. However, this approach is insensitive to text order, contextual information, and image content. For example, when a user's query terms contain significant word order information or the query image content is complex, accurate retrieval results are difficult to obtain. This method, on the other hand, is sensitive to text order, text contextual information, and the actual content of the image. While preserving image-text similarity information in the generated hash code, it enables the model to more accurately generate results based on the searcher's intent.

[0072] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A deep cross-modal hash retrieval method based on image enhancement technology, characterized in that, Includes the following steps: S1: Obtain the target image, obtain the category embedding vector and feature embedding vector based on the target image, use the InceptionV3 model to obtain the target image classification probability and category label, determine the target image category label based on the classification probability and threshold, generate category embedding vectors for the category label through the word2vec model, obtain text features based on the text, decompose the text into several representations, add to the beginning and end of the sentence, fill in the sentence, convert each representation into a corresponding unique ID in the BERT model, and obtain semantically enhanced text features for each representation through the pre-trained BERT model; S2: Based on the category embedding vector and the feature embedding vector, the image embedding vector is obtained. The image category embedding vector is concatenated with the image feature embedding vector generated by the EfficientNet model to obtain the semantically enhanced embedding vector of the image. S3: Train the image embedding vector and text features to obtain the retrieval model. Train the text feature representation obtained in step S1 and the semantically enhanced image obtained in step S2 together with the hash code learning process in the DCMH method to obtain the enhanced cross-modal retrieval model. S4: Input the search text into the search model to obtain the target image.

2. The deep cross-modal hash retrieval method based on image enhancement technology according to claim 1, characterized in that, In S1, the EfficientNet model is used to extract features from the target image to obtain feature embedding vectors.

3. The deep cross-modal hash retrieval method based on image enhancement technology according to claim 1, characterized in that, S1 includes the following steps: S11: Set threshold; S12: If the classification probability is greater than the threshold, the embedding vector of the category label corresponding to the classification probability is used as the embedding of the target image; otherwise, the embedding of the target image is set to the zero vector or a preset value.

4. The deep cross-modal hash retrieval method based on image enhancement technology according to claim 1, characterized in that, In S1, the computational expression for the BERT model is as follows: In the above formula, y j For the j-th segment of text, The feature output corresponding to the j-th text segment is [CLS], which is located at the beginning of each text sequence y. In BERT, the output is an aggregated representation of the subsequent text. The [SEP] identifier is used to separate the two inputs.

5. The deep cross-modal hash retrieval method based on image enhancement technology according to claim 1, characterized in that, S2 includes the following steps: S21: Concatenate the category embedding vector and the feature embedding vector to obtain the concatenated vector; S22: The concatenated vector is passed through a fully connected layer to obtain the image embedding vector.

6. A deep cross-modal hash retrieval method based on image enhancement technology according to claim 1 or 5, characterized in that, The objective function of S3 is as follows: s.t.B∈{-1,+1} c×n In the formula: S ij These are predefined original similarity labels between images and text, where F is the image feature matrix and G is the text feature matrix. i It is an image feature vector, G j It is a text feature vector, Θ ij It is F i With G j The inner product similarity is calculated, where B is the binary hash code matrix to be optimized. (x) and B (y) These are image modality and text modality, respectively, θ x It is the parameter set of the text modality feature extraction network, where γ and η are the weight coefficients of the error term; and These are the squares of the Euclidean distances between the hash code and the image features, and the text features, respectively. and These are the squares of the sum of the elements in each row of the image and text feature matrices, respectively. T is the transpose of the matrix, c is the hash code length, and n is the number of samples.

7. The deep cross-modal hash retrieval method based on image enhancement technology according to claim 1, characterized in that, S4 includes: matching target images with high similarity by inputting search text into the model; if there are several target images, displaying the target images in descending order of similarity.

8. A deep cross-modal hash retrieval method based on image enhancement technology according to claim 1, characterized in that, S1 includes: using the EfficientNet model to extract features from the image, and the output image feature embedding vector is represented as follows: Depending on the size of the target image dataset and the limitations of computing resources, different EfficientNet models are used to flexibly apply to different scenarios.

Images