Deep hash image retrieval method based on prompt reinforcement learning

Abstract

The invention discloses a deep Hash image retrieval method based on prompt reinforcement learning, and mainly relates to the field of Hash media retrieval. Comprising the following steps: S1, respectively and correspondingly dividing an image data set and category label information corresponding to the image data set into a training set, a test set and a retrieval set; s2, constructing a deep hash image retrieval architecture based on prompt reinforcement learning; s3, inputting the training set images into the image network; s4, obtaining a retrieval model through training; s5, inputting the test set picture and the retrieval set picture into the trained retrieval model to respectively obtain hash codes of the test set and the retrieval set; s6, the Hash code of one test image is taken, and a Hamming distance vector is obtained through calculation; s7, sorting numerical values in the Hamming distance vector to obtain a retrieval result of the test image; according to the method, the network can effectively align the bimodal features, important information in the image features is enhanced, and more accurate hash codes with discrimination are generated.

Description

Technical Field

[0001] The present invention relates to the field of deep hash media retrieval, and in particular to a deep hash image retrieval method based on prompt reinforcement learning. Background Art

[0002] With the rapid development of the Internet, the emergence of massive image data has brought great challenges to traditional image retrieval technology. In this context, how to efficiently and accurately retrieve images that meet user needs from a huge image library has become an important research topic in the field of computer vision and information retrieval. As an efficient image retrieval method, hash image retrieval can significantly reduce storage and computing costs by compressing high-dimensional image features into low-dimensional binary hash codes, thereby accelerating the large-scale image retrieval process.

[0003] Although powerful neural networks such as CNN and Transformer have brought many breakthroughs to hash image retrieval, their processing objects only contain image data and randomly initialize each area of ​​the image. The corresponding label data only participates in the optimization of the network as supervisory information, so the image feature expression ability obtained in the feature extraction stage is always limited. This makes the network unable to accurately identify meaningful target information for images with complex scenes, especially images where the target object (foreground information) and the image background (background information) are very similar or the background information is extremely complex, which ultimately leads to a greatly reduced discrimination of the hash code, and even misleading hash codes are generated due to incorrect target recognition. Summary of the invention

[0004] The purpose of the present invention is to solve the problems existing in the prior art and to provide a deep hash image retrieval method based on prompt reinforcement learning, which can help the network to more accurately identify the target objects corresponding to the real labels, generate more accurate and discriminative hash codes, and enable the network to more comprehensively capture the complex patterns and relationships in bimodal information.

[0005] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0006] A deep hash image retrieval method based on hint reinforcement learning comprises the following steps:

[0007] S1. Divide the image dataset and the category label information corresponding to the image dataset into a training set, a test set, and a retrieval set respectively, where each sample includes an image and a corresponding category label. The training set and the test set have the same number of samples for each category, and the other images are used as the retrieval set.

[0008] S2. Construct a deep hash image retrieval architecture based on prompt-enhanced learning, including a CLIP network, a multi-frequency adaptive fusion module, a hash layer and a loss function. The CLIP network includes an image encoder and a text encoder. The CLIP network is used to learn the features of the image and the prompt text. The hash layer includes a fully connected layer, a layer normalization layer and an activation function tanh(·). The hash layer is used to learn the construction of the hash function and obtain the hash code of the input image and the prompt text. The loss function includes an information-noise contrast estimation loss function, a cosine contrast loss function, a similarity affinity loss function, a reconstruction loss function and a quantization loss function. The loss function is used to guide the optimization direction of the hash function.

[0009] S3, input the training set images into the deep hash image retrieval network based on prompt reinforcement learning, first find the corresponding category name through the label of the training image, and construct the prompt according to the category name;

[0010] S4, using the training set, according to the deep hash image retrieval architecture based on prompt reinforcement learning, to obtain a deep hash image retrieval model based on prompt reinforcement learning through training;

[0011] S5, input the test set and retrieval set images into the trained deep hash image retrieval model based on prompt reinforcement learning to obtain the hash codes of the test set and retrieval set respectively;

[0012] S6, taking a hash code of a test image, calculating the Hamming distance between the hash code and the hash code of the retrieval set, and obtaining a Hamming distance vector;

[0013] S7. Sort the values ​​in the Hamming distance vector in ascending order, output the corresponding original images in order, and obtain the retrieval result of the test image.

[0014] Preferably, in a deep hash image retrieval architecture based on prompt-enhanced learning, the image is passed through an image encoder to obtain a class label of the image feature, and the prompt corresponding to the image is passed through a text encoder to obtain a class label of the text feature, wherein the image encoder uses a ViT-B-16 model and the text encoder uses a Bert model, and their weights are frozen at the same time; the obtained class labels of the image features and text features are fused and aligned through a multi-frequency adaptive fusion module, and the prompt text information is embedded in the image information; the image information with prompt enhancement and the prompt information affected by the image enter the subsequent work of the two branches respectively, during which the image-text positive similarity pairs are first brought closer and the negative similarity pairs are pushed away through the information noise contrast estimation loss and the cosine contrast loss, and after the two-modal information hash codes are obtained respectively through the hash layer, the hash codes are optimized through the similarity pair affinity loss, the bit affinity loss and the quantization loss; finally, the decoding features obtained through the image and text decoders are compared and optimized with the encoding features through the reconstruction loss.

[0015] Preferably, the features obtained by the image encoder and the hint encoder are F i x With F i p , expressed as: F i x ,F i p = CLIP(x i ,p i ), where i represents an image-prompt text pair in the set and CLIP(·) represents the CLIP network framework.

[0016] Preferably, in the data preprocessing stage, the category name corresponding to each image is found through the label, and the category name is used as a prompt word to generate a simple prompt for each image. The prompt word has two characteristics: first, the sentence includes all the category names contained in the image as prompt words; second, all words except the prompt words are set as learnable parameters; during the training process, the network is searched for the prompt weight that best suits the network, and by aligning the image with the prompt, the prompt text features complement the image features.

[0017] Preferably, the multi-frequency adaptive fusion module includes a naive Fourier decomposition and a Transformer layer, and the Fourier coefficients are A normal variable with parameters set to be learnable, where inputdim is the input dimension size;

[0018] The image and the prompt feature are transformed by FourierKAN(·) and then concat to obtain the fusion feature.

[0019]

[0020] Next, the fused features are aggregated through a layer of Transformer to obtain the aggregated features.

[0021]

[0022] Separate the aggregated features to obtain image features and prompt features

[0023]

[0024] Compared with the prior art, the beneficial effects of the present invention are:

[0025] The present invention proposes a deep hash image retrieval method based on prompt reinforcement learning. Different from previous work, the present invention introduces a prompt learning method and adds a parallel prompt text encoder in addition to the image encoder. On the one hand, the introduction of prompt text breaks the characteristic of the single data modality in the network training stage in previous work, and uses the prompt sentence as a task-independent modality to assist the network in learning image features; on the other hand, the prompt uses the category name corresponding to the image as a prompt word, emphasizes the target object through the prompt word, and helps the network to more accurately distinguish foreground information and background information. At the same time, due to the asymmetry of image information and prompt text information, the present invention adopts a multi-frequency adaptive fusion module to align and fuse image features and prompt features. It not only maps the image and prompt text to the same frequency domain feature space, but also uses a combination of bi-sine (i.e., cosine and sine) functions to fully capture the complex patterns and relationships in the bimodal information, and combines the learnable Fourier coefficients to adaptively learn the frequency components that are most conducive to feature alignment and fusion. It is worth noting that the image and text features are not only mapped to one frequency domain feature space, but are simultaneously mapped to multiple different frequency domain feature spaces to capture the multi-scale information in the input features. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a diagram of the deep hashing image retrieval architecture based on hint reinforcement learning. DETAILED DESCRIPTION

[0027] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.

[0028] Embodiment: The present invention is a deep hash image retrieval method based on prompt reinforcement learning, and the specific implementation steps are as follows:

[0029] Step 1: Divide the image dataset and its corresponding category label information into three parts: training set, test set and retrieval set. Each sample includes an image and its corresponding category label. The training set and test set have the same number of samples for each category, and the other images are used as retrieval sets.

[0030] Step 2: Construct a deep hash image retrieval architecture based on prompt reinforcement learning, which includes the CLIP network (including image encoder and text encoder), multi-frequency adaptive fusion module, hash layer (including fully connected layer, layer normalization layer and activation function tanh(·)) and loss function (including information noise contrast estimation loss function, cosine contrast loss function, similarity affinity loss function, reconstruction loss function and quantization loss function). The CLIP network is used to learn the features of the image and prompt text, the hash layer is used to learn the construction of the hash function and obtain the hash code of the input image and prompt text, and the loss function is used to guide the optimization direction of the hash function.

[0031] Step 3: Input the training set images into the deep hash image retrieval network based on prompt reinforcement learning. First, find the corresponding category name through the label of the training image, and construct the prompt according to the category name;

[0032] Step 4: Using the training set, according to the deep hash image retrieval architecture based on prompt reinforcement learning, a deep hash image retrieval model based on prompt reinforcement learning is obtained through training;

[0033] Step 5: Input the test set and retrieval set images into the trained deep hash image retrieval model based on prompt reinforcement learning to obtain the hash codes of the test set and retrieval set respectively;

[0034] Step 6: Take a hash code of a test image, calculate the Hamming distance between the hash code and the hash code of the retrieval set, and obtain the Hamming distance vector;

[0035] Step 7: Sort the values ​​in the Hamming distance vector from small to large, output the corresponding original images in order, and obtain the retrieval result of the test image.

[0036] Next, the prompt text construction and multi-frequency adaptive fusion module mentioned in the above steps will be described in detail. First of all, the present invention, like most other methods, uses x n To represent the image sample, y n Indicates the corresponding label. Assuming that there are N samples in the data set, the present invention has an image set The corresponding prompt text set is The tag set is in, is One-Hot encoding, N c Represents the number of categories of labels. n With p n After the hash layer and function, the corresponding real-valued hash code h is obtained n With g n , the corresponding binary representation can be obtained through the symbolic function and N bits is the length of the hash code, and the sign function is defined as follows:

[0037]

[0038] The deep hash image retrieval architecture based on hint reinforcement learning in step 2 above is as follows Figure 1 As shown in Figure 2, we use the CLIP model framework to establish a connection between the image and the prompt text through contrastive learning.

[0039] like Figure 1 As shown in the figure, the image and the corresponding prompt are respectively passed through the image encoder and the text encoder to obtain the class labels of the image features and text features. The image encoder uses the ViT-B-16 model, and the text encoder uses the Bert model, and their weights are frozen at the same time. The obtained class labels of the image features and text features are fused and aligned through the multi-frequency adaptive fusion module. In this process, the prompt text information is embedded into the image information to enhance the important part of the image information. In order to feed back this effect to the network and hash code, the image information with prompt enhancement and the prompt information affected by the image enter the subsequent work of the two branches respectively. During this period, the image-text positive similarity pairs are first brought closer and the negative similarity pairs are pushed away through the information noise contrast estimation loss and the cosine contrast loss. After the two-modal information hash codes are obtained through the hash layer, the hash codes are optimized through the similarity pair affinity loss, bit affinity loss and quantization loss. Finally, the decoded features obtained by the image and text decoders are compared and optimized with the encoded features through the reconstruction loss to avoid the interaction of the two-modal features causing them to differ greatly from the original single-modal features.

[0040] The detailed description of each part is as follows:

[0041] Tips for learning networking:

[0042] Assume that the features obtained by the image encoder and the hint encoder are and It can be expressed as:

[0043]

[0044] Where i represents an image-prompt text pair in the set, and CLIP(·) represents the CLIP network framework. In the present invention, all network weights are frozen for both the image and text encoders, and no parameter update is performed. The prompt module adopts a dynamic region prompt mechanism, constructs prompt words only through class labels, and adds a learnable prompt tensor.

[0045] Specifically, in the data preprocessing stage, the present invention finds the corresponding category name through the label of each image, which displays the important information in the expression image. The category name is used as a prompt word to generate a simple prompt sentence for each image, which has two characteristics: first, the sentence includes all the category names contained in the image as prompt words; second, all words except the prompt words are set as learnable parameters. These two characteristics make the prompt sentence concise and flexible, which can help the network distinguish foreground information and background information more accurately.

[0046] During the training process, the network searches for the most suitable prompt weights for the network. In addition, by aligning the image with the prompt, the prompt text features complement the image features, thereby improving the retrieval accuracy.

[0047] Multi-frequency adaptive fusion module:

[0048] In the present invention, the multi-frequency adaptive fusion module includes a naive Fourier decomposition and a Transformer layer, wherein the grid size gridsize in the naive Fourier decomposition is set to 300, assuming that the Fourier coefficient is A normal variable with parameters set to be learnable, where inputdim is the input dimension size.

[0049] Assume the input is X, define a linear mapping W θ , generate the Fourier transform grid vector k, where the dimension of k is gridsize, k = [1,2,...,gridsize], and perform cosine and sine transforms on the input to obtain:

[0050]

[0051] in Represents matrix multiplication. Concatenating the results of cosine and sine transforms gives:

[0052] Z=Concat(C,S)

[0053] Using the Einstein summation convention, we can calculate the Fourier transform result:

[0054]

[0055] Then, the Fourier transform feature y required by the present invention can be obtained through linear transformation:

[0056] Among them, ⊙ represents element-by-element multiplication; the above operation is represented by FourierKAN(·), and the image and the prompt feature are fused by concat after Fourier transformation.

[0057]

[0058] Next, the fused features are aggregated through a layer of Transformer to obtain the aggregated features.

[0059]

[0060] Separate the aggregated features to obtain image features and prompt features

[0061]

[0062] The final result and The image features enhanced by the prompts and the prompt features affected by the images. The embedding of the prompt text makes the two modal features blend with each other, bringing clearer important identification to the image features and significantly improving the accuracy and distinguishability of the image hash code.

[0063] The present invention introduces the method of prompt learning into the field of deep hash image retrieval, breaking the unified paradigm of previous work, adding task-independent modal data as auxiliary information, helping the network to more accurately identify the target object corresponding to the true label, and generate a more accurate and discriminative hash code. The present invention proposes a multi-frequency adaptive fusion module, which aligns and fuses the two modal features of image and prompt text into the same frequency domain feature space, and extracts multi-scale features through multiple different frequency domain feature spaces. The dual-chord combination function and learnable Fourier coefficients enable the network to more comprehensively capture the complex patterns and relationships in the bimodal information.

Claims

1. A deep hash image retrieval method based on hint reinforcement learning, characterized in that: Includes steps: S1. Divide the image dataset and the category label information corresponding to the image dataset into a training set, a test set, and a retrieval set respectively, where each sample includes an image and a corresponding category label. The training set and the test set have the same number of samples for each category, and the other images are used as the retrieval set. S2. Construct a deep hash image retrieval architecture based on prompt-enhanced learning, including a CLIP network, a multi-frequency adaptive fusion module, a hash layer and a loss function. The CLIP network includes an image encoder and a text encoder. The CLIP network is used to learn the features of the image and the prompt text. The hash layer includes a fully connected layer, a layer normalization layer and an activation function tanh(·). The hash layer is used to learn the construction of the hash function and obtain the hash code of the input image and the prompt text. The loss function includes an information-noise contrast estimation loss function, a cosine contrast loss function, a similarity affinity loss function, a reconstruction loss function and a quantization loss function. The loss function is used to guide the optimization direction of the hash function. S3, input the training set images into the deep hash image retrieval network based on prompt reinforcement learning, first find the corresponding category name through the label of the training image, and construct the prompt according to the category name; S4, using the training set, according to the deep hash image retrieval architecture based on prompt reinforcement learning, to obtain a deep hash image retrieval model based on prompt reinforcement learning through training; S5, input the test set and retrieval set images into the trained deep hash image retrieval model based on prompt reinforcement learning to obtain the hash codes of the test set and retrieval set respectively; S6, taking a hash code of a test image, calculating the Hamming distance between the hash code and the hash code of the retrieval set, and obtaining a Hamming distance vector; S7. Sort the values ​​in the Hamming distance vector in ascending order, output the corresponding original images in order, and obtain the retrieval result of the test image.

2. The deep hash image retrieval method based on prompt reinforcement learning according to claim 1, characterized in that: In the deep hash image retrieval architecture based on prompt reinforcement learning, the image is passed through the image encoder to obtain the class label of the image feature, and the corresponding prompt of the image is passed through the text encoder to obtain the class label of the text feature. The image encoder uses the ViT-B-16 model, and the text encoder uses the Bert model, and their weights are frozen at the same time; the obtained class labels of image features and text features are fused and aligned through the multi-frequency adaptive fusion module, and the prompt text information is embedded in the image information; the image information with prompt enhancement and the prompt information affected by the image enter the subsequent work of the two branches respectively. During this period, the image-text positive similarity pairs are first brought closer and the negative similarity pairs are pushed away through the information noise contrast estimation loss and the cosine contrast loss. After the two-modal information hash codes are obtained respectively through the hash layer, the hash codes are optimized through the similarity pair affinity loss, bit affinity loss and quantization loss; finally, the decoding features obtained through the image and text decoders are compared and optimized with the encoding features through the reconstruction loss.

3. The deep hash image retrieval method based on prompt reinforcement learning according to claim 2 is characterized in that: The features obtained by the image encoder and the hint encoder are F i x With F i p , expressed as: F i x ,F i p = CLIP(x i ,p i ), where i represents an image-prompt text pair in the set and CLIP(·) represents the CLIP network framework.

4. The deep hash image retrieval method based on prompt reinforcement learning according to claim 3 is characterized in that: In the data preprocessing stage, the corresponding category name is found through the label of each image, and the category name is used as a prompt word to generate a simple prompt for each image. The prompt word has two characteristics: first, the sentence includes all the category names contained in the image as prompt words; second, other words except the prompt word are set as learnable parameters; During the training process, the network searches for the most suitable prompt weights for the network, and by aligning the image with the prompt, the prompt text features complement the image features.

5. The deep hash image retrieval method based on prompt reinforcement learning according to claim 2 is characterized in that: The multi-frequency adaptive fusion module includes naive Fourier decomposition and Transformer layer, and the Fourier coefficients are A normal variable with parameters set to be learnable, where inputdim is the input dimension size; The image and the prompt feature are transformed by FourierKAN(·) and then concatenated to obtain the fusion feature F i Four : F i Four =Concat(FourierKAN(F i x ),FourierKAN(F i p )); Next, the fused features are aggregated through a layer of Transformer to obtain the aggregated features. Separate the aggregated features to obtain image features and prompt features

Images