An adaptive enhanced weakly supervised hash retrieval method and system

Abstract

The present invention discloses an adaptive enhanced weakly supervised hash retrieval method and system, which inputs a query image for online retrieval into a trained retrieval model and outputs an image retrieval result; the retrieval model is trained as follows: an image-text database is constructed; in the first stage, features are extracted from images and texts to obtain image expression features and text expression features, and the image expression features are sequentially passed through a deep hash network and a projection network to obtain a reconstructed user text annotation vector; in the second stage, the similarity between the reconstructed user text annotation vector corresponding to each image and each user text annotation in a text annotation vector data set is calculated, and based on the similarity sorting, it is determined whether the text annotation vector data set should be updated, and the updated text annotation vector data set is used to guide the training of the deep hash network and the projection network in the first stage; the weakly supervised hash retrieval method and system improve retrieval accuracy.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an adaptively enhanced weakly supervised hash retrieval method and system. Background Art

[0002] Image retrieval technology aims to return database images similar to the user-entered query image from a large-scale database in real time. To ensure real-time response and efficient storage, images are usually expressed using compact representations. The most common image compact representation technology is binary hashing, which represents the image as a binary code. This can effectively reduce the storage space occupied by the image representation, and the Hamming distance between binary codes can be efficiently calculated by existing CPU architectures. Therefore, the binary hashing method can simultaneously ensure real-time response and efficient storage of the retrieval system. In the binary hashing method, a discriminative binary hash function needs to be learned to improve the retrieval accuracy of the binary hashing method, thereby ensuring satisfactory retrieval results.

[0003] Existing binary hashing methods can be divided into three categories based on the use of supervisory information during the hash function learning process: unsupervised, semi-supervised, and supervised. Unsupervised hashing methods do not use any supervisory information during hash function learning, while supervised hashing methods use supervisory information to train hash functions. Semi-supervised hashing methods use supervisory information from partial data during training. Generally speaking, the more supervisory information used during learning, the more discriminative the hash function and the better the retrieval accuracy of the binary hashing method. However, collecting accurate supervisory information typically requires significant human and material resources.

[0004] In recent years, weakly supervised hashing methods have begun to be studied. These methods use readily available, noisy user text annotations as weak supervision to guide the learning of hash functions. Existing image sharing websites allow users to provide simple text annotations when uploading images. These descriptions typically describe diverse information such as image content, attributes, camera information, weather conditions, and location, and can be used as weak supervision for hash function learning. Extracting supervisory information relevant to image retrieval tasks from these diverse text descriptions is a key challenge facing existing weakly supervised hashing methods. Based on how user text annotations are applied, existing weakly supervised hashing methods can be divided into two categories: direct application methods and update application methods. Direct application methods directly use user text annotations as supervision information and, through designing objective functions and network architectures, learn to embed the semantic information contained in these user text annotations into the hash function. Update application methods design methods to update user text annotations during the hash function learning process, using the updated user text annotations as more accurate supervision information to guide the hash function learning. Generally speaking, update application methods are able to distill more accurate supervision information, thereby achieving superior retrieval accuracy.

[0005] On the other hand, vision-language pre-training has already developed a series of powerful foundational models. Existing research on vision-language pre-training has mostly explored different model architectures and pre-training objectives, achieving excellent performance in various downstream tasks (such as image captioning, visual question answering, and image-text matching).

[0006] However, there is currently little work exploring the application of visual language pre-training based models in large-scale image retrieval tasks. Existing supervised and semi-supervised hashing methods require the use of manually annotated supervisory information, which is labor-intensive and resource-intensive. While unsupervised hashing methods do not require the use of manually annotated supervisory information, the lack of supervisory information during hash function learning results in weak semantic discriminability and generally inferior retrieval accuracy to hashing methods that utilize supervised information. Weakly supervised hashing methods utilize user text annotations that come with the image. Although these user annotations are not entirely relevant to the retrieval task and may contain some task-irrelevant text annotations as noise, they still provide a certain amount of content supervision information, and therefore generally offer better retrieval accuracy than unsupervised hashing methods.

[0007] Among weakly supervised hashing methods, direct application methods apply noisy user text annotations without filtering. This interferes with the learning process of the hash function due to task-irrelevant user annotations. While retrieval performance is superior to unsupervised hashing methods, retrieval accuracy is generally inferior to update application methods. Furthermore, existing update application methods are relatively rare. While they can update user text annotations to a certain extent, their update process typically ignores the contextual similarity of the text or only considers the similarity between the image content and the text. Therefore, the updated user text annotations are not fully relevant to the image retrieval task and cannot guarantee image retrieval accuracy. Furthermore, existing weakly supervised hashing methods typically use traditional deep convolutional models as their backbone networks, and their retrieval accuracy is limited by the semantic discrimination capabilities of traditional convolutional models. Summary of the Invention

[0008] Based on the technical problems existing in the background technology, the present invention proposes an adaptive enhanced weakly supervised hash retrieval method and system to improve the retrieval accuracy.

[0009] The present invention proposes an adaptive enhanced weakly supervised hash retrieval method, which inputs the query image of online retrieval into the trained retrieval model and outputs the image retrieval result;

[0010] The retrieval model is trained as follows:

[0011] Step 1: Obtain images and corresponding user text annotations, and build an image-text database based on all images and corresponding user text annotations, wherein the image-text database includes a training image set and a text annotation vector dataset;

[0012] Step 2, first stage retrieval: Based on the frozen visual encoder in the visual language pre-training basic model, feature extraction is performed on the input image to obtain image expression features. Based on the frozen text encoder in the visual language pre-training basic model, feature extraction is performed on the input user text annotation features to obtain text expression features. The image expression features are converted into binary codes through the deep hash network, and the binary codes are converted into reconstructed user text annotation vectors through the projection network. The first stage loss is constructed to optimize the first stage deep hash network and projection network for training;

[0013] Step 3: The second stage of redefining user text annotations: Calculate the similarity between the reconstructed user text annotation vector corresponding to each image and each user text annotation in the text annotation vector dataset, and sort based on the similarity to obtain the text annotation phrase w with the minimum similarity in the current text annotation vector dataset. i , and the text annotation phrase w that does not have the maximum similarity in the current text annotation vector dataset m , by calculating the text annotation phrase w i and text annotation phrase wm The exchange reward and non-exchange reward between them are used to determine whether to mark the phrase w through text according to the relative size of the exchange reward and the non-exchange reward. m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, and using the updated text annotation vector dataset to guide the training of the deep hashing network and projection network in the first stage.

[0014] In step 2, the process of obtaining a binary code from image expression features through a deep hash network is modeled as a probabilistic decision-making process. The output of the deep hash network is regarded as the probability of outputting a bit '1' at each bit position. Based on the output probability, multiple binary codes are sampled, and each binary code is regarded as a probabilistic trial and error process.

[0015] Furthermore, in step 2, the first-stage loss includes strategy loss, balance loss, and cross-modal discriminative loss;

[0016] The strategy loss directly optimizes the reward of the binary code, specifically:

[0017]

[0018] Among them, Θ h represents the deep hashing network parameters to be learned, Represents the binary code of the i-th image in the c-th trial and error process Rewards received;

[0019] The balance loss achieves the compactness and high information content of the binary code by optimizing the probability that each bit has an equal output of '0' or '1', specifically:

[0020]

[0021] where B∈{0,1} N×L is the binary code matrix of the entire training set, · represents dot product, N is the number of images in the training set, L is the bit length, 1 and 0.5 represent vectors with values ​​of all 1 and 0.5;

[0022] The cross-modal discriminative loss is to output binary code to embed semantic discriminative information, specifically:

[0023]

[0024] where Θ m represents the parameters of the projection network, r i represents the reconstructed user text annotation vector corresponding to the i-th image, t i is the user text annotation vector corresponding to the i-th image in the text annotation vector dataset, t jis the user text annotation vector corresponding to the jth image in the text annotation vector dataset, and τ is a manually set heat hyperparameter;

[0025] The final first-stage optimization target L(Θ h ,Θ m ) is to optimize the sum of the three losses mentioned above, specifically:

[0026] L(Θ,Θm)=Li(Θ,Θm)+αLp(Θh)+βL(Θh);

[0027] where α and β represent hyperparameters that control the relative importance of policy loss and cross-modal discriminative loss.

[0028] Calculate the retrieval accuracy of each trial-and-error process as a reward, use the binary code of the trial-and-error process as a query, calculate the Hamming distance between the query binary code and the binary code in the training image set, sort the images in the training set in ascending order of the Hamming distance, and obtain the retrieval results;

[0029] Furthermore, in calculating the retrieval accuracy of each trial-and-error process as a reward, in order to calculate the retrieval accuracy of the trial-and-error process, in the trial-and-error process, user text annotations are used as weak supervision. If the binary code used as the query and the image-text database include common user text annotations, the retrieval accuracy calculated for the binary code is used as a reward, maximizing the output probability of the trial-and-error process with the maximum reward, thereby enabling the deep hash network to generate distinguishable binary codes.

[0030] Furthermore, in step 3, by calculating the text annotation phrase w i and text annotation phrase w m The exchange reward and non-exchange reward between them are used to determine whether the text annotation phrase w m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, specifically:

[0031] If the exchange reward is less than or equal to the non-exchange reward, then the text is annotated with the phrase w m Do not replace the text annotation phrase w i , if the exchange reward is greater than the non-exchange reward, then the text annotation phrase w m Replace the text annotation phrase w i ;

[0032] When the text is annotated with the phrase w m Do not replace the text annotation phrase w iWhen , the mean of all user text annotations in the text annotation vector dataset is used as the query vector, and the training image set is used as the retrieval database. The cosine similarity between the query vector and each user text annotation vector in the retrieval database is calculated, and the retrieval database is sorted in descending order based on the similarity to obtain the retrieval results;

[0033] When the text is annotated with the phrase w m Replace the text annotation phrase w i When the text annotation phrase w in the text annotation vector dataset is i Remove and add text annotation phrase w m , the mean of all user text annotations in the updated text annotation vector dataset is used as the query vector, the training image set is used as the retrieval database, the cosine similarity between the query vector and each user text annotation vector in the retrieval database is calculated, and the retrieval database is sorted in descending order based on the similarity to obtain the retrieval results.

[0034] An adaptively enhanced weakly supervised hash retrieval system that inputs online query images into a trained retrieval model and outputs image retrieval results.

[0035] The retrieval model is trained as follows:

[0036] Step 1: Obtain images and corresponding user text annotations, and build an image-text database based on all images and corresponding user text annotations, wherein the image-text database includes a training image set and a text annotation vector dataset;

[0037] Step 2, first stage retrieval: Based on the frozen visual encoder in the visual language pre-training basic model, feature extraction is performed on the input image to obtain image expression features. Based on the frozen text encoder in the visual language pre-training basic model, feature extraction is performed on the input user text annotation features to obtain text expression features. The image expression features are converted into binary codes through the deep hash network, and the binary codes are converted into reconstructed user text annotation vectors through the projection network. The first stage loss is constructed to optimize the first stage deep hash network and projection network for training;

[0038] Step 3: The second stage of redefining user text annotations: Calculate the similarity between the reconstructed user text annotation vector corresponding to each image and each user text annotation in the text annotation vector dataset, and sort based on the similarity to obtain the text annotation phrase w with the minimum similarity in the current text annotation vector dataset. i , and the text annotation phrase w that does not have the maximum similarity in the current text annotation vector dataset m , by calculating the text annotation phrase w i and text annotation phrase w mThe exchange reward and non-exchange reward between them are used to determine whether to use text to mark the phrase w according to the relative size of the exchange reward and the non-exchange reward. m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, and using the updated text annotation vector dataset to guide the training of the deep hashing network and projection network in the first stage.

[0039] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the weakly supervised hash retrieval method as described above when executing the computer program.

[0040] A computer-readable storage medium stores a plurality of classification programs, wherein the plurality of classification programs are used to be called by a processor and execute the weakly supervised hash retrieval method as described above.

[0041] The advantages of the adaptively enhanced weakly supervised hash retrieval method and system provided by the present invention are: by iteratively and alternately optimizing the weak hash function learning method and the user text annotation adaptive update method, and using retrieval accuracy as a reward to optimize the two-stage training process, the user text annotation can be more relevant to the retrieval task and the image content, and serve as more accurate supervisory information to guide the learning of the hash function. Furthermore, the learned deep hash network can generate more discriminative binary codes, thereby obtaining better online retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a structural schematic diagram of the present invention; DETAILED DESCRIPTION

[0043] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0044] To address the problems of existing binary hashing methods, we design an adaptively enhanced weakly supervised hash retrieval method. This method alternately and iteratively learns hash functions and updates user text annotations. By designing a probabilistic decision-making hash function learning method and an adaptive update method for user text annotations, we directly optimize the retrieval accuracy during training, obtain a more discriminative hash function, and thus improve the retrieval accuracy during testing. Specifically,

[0045] Applying the image-text alignment capabilities of the visual language pre-training basic model, a weak hash function learning method and a user text annotation adaptive update method with alternating iterative optimization are designed. Specifically, the weak hash function learning method in this embodiment is modeled as a probabilistic decision-making learning method, which uses the updated user text annotations as semantic supervision to directly optimize the retrieval accuracy of the hash function. The user text annotation adaptive update method in this embodiment is also modeled as a probabilistic decision-making update method, which uses the binary code generated by the hash function to calculate the neighbor relationship between images, so that the updated user text annotations can maximize the image neighbor relationship, and thus can directly optimize the retrieval accuracy. By iteratively and alternately optimizing the weak hash function learning method and the user text annotation adaptive update method, this embodiment can effectively obtain user text annotations that are directly related to the retrieval task, and obtain a hash function with more semantic discrimination, thereby further improving the accuracy of image retrieval. This embodiment is described in detail below.

[0046] like Figure 1 As shown in FIG, the present invention proposes an adaptive enhanced weakly supervised hash retrieval method, which inputs the query image of the online retrieval into the trained retrieval model and outputs the image retrieval result;

[0047] The retrieval model consists of a two-stage learning process with alternating iterative optimization: learning a deep weakly supervised hash function and adaptively updating user text annotations. In the first stage, the deep weakly supervised hash function is directly optimized using user text annotations as weak supervision information. In the second stage, the user text annotations are adaptively updated based on the binary codes learned in the previous stage and the reconstructed user text annotation vectors.

[0048] The retrieval model is trained as follows:

[0049] Step 1: Obtain images and corresponding user text annotations, and build an image-text database based on all images and corresponding user text annotations, wherein the image-text database includes a training image set and a text annotation vector dataset;

[0050] Step 2, first stage retrieval: Based on the frozen visual encoder in the visual language pre-training basic model, feature extraction is performed on the input image to obtain image expression features. Based on the frozen text encoder in the visual language pre-training basic model, feature extraction is performed on the input user text annotation features to obtain text expression features. The image expression features are converted into binary codes through the deep hash network, and the binary codes are converted into reconstructed user text annotation vectors through the projection network. The first stage loss is constructed to optimize the first stage deep hash network and projection network for training;

[0051] The deep hashing network must contain a fully connected layer and an activation function layer to ensure that its output is in the range [0, 1]. To embed task-specific information in the binary code output, a projection network is designed to reconstruct the user-annotated text vector from the binary code output of the deep hashing network. This projection network consists of three fully connected layers. The first two layers each use a ReLU activation function, and the output of the last layer must be L2 normalized.

[0052] The goal of the first phase is to learn a binary hash function that generates distinguishable binary codes for retrieval tasks. Based on the image-text alignment capabilities of user text annotation and the pre-trained visual language model, this example designs three losses for training and optimizing the hash function (deep hashing network and projection network). These losses are: policy loss, balance loss, and cross-modal discriminative loss.

[0053] (a1) Strategy loss

[0054] First, to overcome the discontinuous objective function and optimization difficulties associated with binary codes, this embodiment designs a novel strategic loss to directly optimize the reward of binary codes. Specifically, the binarization process is modeled as a probabilistic decision-making process. The output of the deep hashing network is considered the probability of outputting a '1' bit at each bit position. Based on this probability, multiple binary codes are sampled, and each binary code is treated as a probabilistic trial-and-error process.

[0055] In order to calculate the retrieval accuracy of the trial-and-error process, user text annotations are used as weak supervision. That is, the query image (the query is represented by the binary code of the query image) and the image-text database are considered to be related if they contain common user text annotations. The retrieval accuracy is calculated as the reward of the trial-and-error process. Through the strategy loss, the output probability of the trial-and-error process with the maximum reward is maximized.

[0056] The strategy loss directly optimizes the reward of the binary code, specifically:

[0057]

[0058] Among them, Θ h represents the deep hashing network parameters to be learned, The binary code representing the i-th training sample (i.e. image) in the c-th trial-and-error process The rewards received are: The retrieval accuracy obtained by searching the image-text database for the query vector.

[0059] Specifically, the retrieval accuracy is obtained by using the binary code of the trial-and-error process as the query, calculating the Hamming distance between the query binary code and the binary code in the training image set, sorting the images in the training image set in ascending order of the Hamming distance, obtaining the retrieval results, and calculating the accuracy of the retrieval results based on the training set labels.

[0060] (a2) Cross-modal discriminative loss

[0061] Then, in order to embed semantic distinctiveness directly in the binary code, this embodiment designs a cross-modal discriminative loss so that the reconstructed user-annotated text vector and the user-text annotation vector of the image are semantically distinctive, that is, the similarity between the reconstructed user-annotated text vector and the user-text annotation vector of the input image is as large as possible, and the similarity between the user-text annotation vectors of different images is as small as possible, while optimizing the deep hashing network and the projection network.

[0062] The cross-modal discriminative loss is to output binary code to embed semantic discriminative information, specifically:

[0063]

[0064] where Θ m represents the parameters of the projection network, r i Represents the reconstructed user text annotation vector, t i The user text annotation vector in the text annotation vector dataset is obtained by aggregating the text phrase vectors corresponding to each image, that is, where w i,k is the kth text phrase w in the text annotation set of the i-th image i,k Text expression features, N k is the total number of text phrases in the text annotation set of the i-th image, and τ is a manually set heat hyperparameter.

[0065] (a3) Finally, to maintain the compactness of the output binary code, this embodiment designs a balanced loss so that each bit position has the same probability of outputting a bit '0' or '1', maximizing the amount of information that can be represented by each bit position.

[0066] That is, the balanced loss achieves the compactness and high information content of the binary code by optimizing the probability that each bit has an equal output of '0' or '1', specifically:

[0067]

[0068] where B∈{0,1} N×Lis the binary code matrix of the entire training set, · represents dot product, N is the total number of images in the training image set, L is the bit length, 1 and 0.5 represent vectors with values ​​of all 1 and 0.5.

[0069] The final first-stage optimization target L(Θ h ,Θ m ) is to optimize the sum of the three losses mentioned above, specifically:

[0070] L(Θ,Θm)=Li(Θ,Θm)+αLp(Θh)+βL(Θh);

[0071] Where α and β represent hyperparameters that control the relative importance of the corresponding losses.

[0072] By simultaneously optimizing the above three losses, the first-stage deep weakly supervised hash function learning module (first-stage loss) designed in this embodiment can simultaneously optimize the deep hash network and the projection network.

[0073] Step 3: The second stage of redefining user text annotations: Calculate the similarity between the reconstructed user text annotation vector corresponding to each image and each user text annotation in the text annotation vector dataset, and sort based on the similarity to obtain the text annotation phrase w with the minimum similarity in the current text annotation vector dataset. i , and the text annotation phrase w that does not have the maximum similarity in the current text annotation vector dataset m , by calculating the text annotation phrase w i and text annotation phrase w m The exchange reward and non-exchange reward between them are used to determine whether to use text to mark the phrase w according to the relative size of the exchange reward and the non-exchange reward. m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, and using the updated text annotation vector dataset to guide the training of the deep hashing network and projection network in the first stage.

[0074] In the second stage, based on the binary codes learned in the previous stage and the reconstructed user text annotation vector, each image's user text annotation is adaptively redefined by analyzing its neighbor relationships. By calculating the Hamming distance between the binary code and the database image's binary code, and the Euclidean distance between the reconstructed text vector and the database text annotation vector, a relatively reliable image neighbor relationship is obtained from the K neighboring images with the minimum distance. Through the neighbor relationship analysis of each image, its user text annotation is calculated to ensure consistency with the neighboring user annotations and is then adaptively updated.

[0075] The learning goal of the second phase is to redefine user text annotations and filter out noise from the existing user annotation set for each image, namely, user text annotations that are irrelevant to the retrieval task or the image content. Through the training of the first phase, the binary codes and reconstructed user text annotation vectors can carry a certain degree of semantic distinctiveness. Based on the output of the first phase, this embodiment designs a probabilistic decision-making user text annotation update method, which iteratively removes the text annotation with the smallest reward from the current text annotation vector dataset and adds the text annotation with the largest reward.

[0076] Specifically, for each image, given the reconstructed user text annotation vector, the similarity between it and the user text annotations in the entire text annotation vector dataset is calculated in turn. This similarity can be regarded as the correlation between the user text annotation and the image. Consider the text annotation phrase w with the maximum correlation (similarity) that is not in the current image user annotation set. m , and the text annotation phrase w with the minimum relevance (similarity) in the current image user annotation set i By calculating the exchange reward and non-exchange reward of the two user text annotation phrases, it is decided whether to exchange the user text annotations, and then the user text annotation set is updated. In this embodiment, exchange and non-exchange are two trial and error processes.

[0077] This embodiment uses retrieval accuracy as a reward for different trial-and-error processes. The reward is calculated by the image retrieval accuracy of the training image set obtained by using the text annotation phrase representation as the query. Whether to exchange is selected based on the relative size of the exchange reward and the non-exchange reward. If the exchange reward is less than or equal to the non-exchange reward, then no exchange (i.e., no replacement) is performed. If the exchange reward is greater than the non-exchange reward, then exchange (i.e., replacement) is performed.

[0078] When there is no exchange (i.e. no replacement), the mean of all user text annotations in the text annotation vector dataset is used as the query vector, and the training image set is used as the retrieval database. The cosine similarity between the query vector and each user text annotation vector in the database image is calculated, and the retrieval database is sorted in descending order based on the similarity to obtain the retrieval results.

[0079] When exchanging (i.e. replacing), the text annotation vector dataset is first updated to remove w i , add w m , and calculate the mean of all user text annotations in the updated text annotation vector dataset as the query vector, use the training image set as the retrieval database, calculate the cosine similarity between the query vector and each user text annotation vector in the database image, and sort the retrieval database in descending order based on the similarity to obtain the retrieval results.

[0080] Unlike the first stage, which uses user text annotations as supervisory information, the second stage uses the binary code output from the first stage to calculate the database image with the minimum Hamming distance as the neighbor image of the current image, that is, the related image. This is then used to calculate the retrieval accuracy of the two swapping and non-swapping processes as rewards. If the swapping process obtains more rewards, the text annotation vector dataset and training image set of the current image are updated; otherwise, the original text annotation vector dataset and training image set remain unchanged.

[0081] By iteratively updating the text annotation vector dataset for each image, the user text annotations for the entire training image set are ultimately more relevant to the retrieval task and the image content, and can serve as better supervisory information to guide the deep hash function learning in the first phase. Therefore, this embodiment iteratively interleaves the two phases until convergence.

[0082] Through steps one to three, this embodiment proposes a weakly supervised hash retrieval method, which trains a deep hash network and a projection network by alternating iterative optimization of a weak hash function learning method and a user text annotation adaptive update method, and adaptively updates the user text annotation. After training, the deep hash network can be used for large-scale image retrieval to extract the binary code of each image in a large-scale image database, which can significantly reduce the storage consumption of the large-scale image database. When searching online, the user gives a query image, and its binary code is extracted through the deep hash network. The similarity with the database image can then be calculated by calculating the Hamming distance between the binary codes. The database images are sorted in order from small to large according to the Hamming distance to obtain the retrieval results. Since the Hamming distance calculation only involves XOR sum operations, the distance calculation efficiency is high, ensuring the real-time performance of large-scale image retrieval. Therefore, the adaptive enhanced weakly supervised hash proposed in this embodiment has low storage consumption and high computational efficiency.

[0083] On the other hand, this embodiment iteratively and alternately optimizes the weak hash function learning method and the user text annotation adaptive update method, and uses retrieval accuracy as a reward to optimize the two-stage training process. User text annotation can be more relevant to the retrieval task and the image content, and serve as more accurate supervisory information to guide the learning of the hash function. Furthermore, the learned deep hash network can generate more discriminative binary codes, thereby obtaining better online retrieval accuracy.

[0084] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. An adaptive enhanced weakly supervised hash retrieval method, characterized in that: Input the query image retrieved online into the trained retrieval model and output the image retrieval result; The retrieval model is trained as follows: Step 1: Obtain images and corresponding user text annotations, and build an image-text database based on all images and corresponding user text annotations, wherein the image-text database includes a training image set and a text annotation vector dataset; Step 2, first stage retrieval: Based on the frozen visual encoder in the visual language pre-training basic model, feature extraction is performed on the input image to obtain image expression features. Based on the frozen text encoder in the visual language pre-training basic model, feature extraction is performed on the input user text annotation features to obtain text expression features. The image expression features are converted into binary codes through the deep hash network, and the binary codes are converted into reconstructed user text annotation vectors through the projection network. The first stage loss is constructed to optimize the first stage deep hash network and projection network for training; Step 3: The second stage of redefining user text annotations: Calculate the similarity between the reconstructed user text annotation vector corresponding to each image and each user text annotation in the text annotation vector dataset, and sort based on the similarity to obtain the text annotation phrase w with the minimum similarity in the current text annotation vector dataset. i , and the text annotation phrase w that does not have the maximum similarity in the current text annotation vector dataset m , by calculating the text annotation phrase w i and text annotation phrase w m The exchange reward and non-exchange reward between them are used to determine whether to use text to mark the phrase w according to the relative size of the exchange reward and the non-exchange reward. m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, and using the updated text annotation vector dataset to guide the training of the deep hashing network and projection network in the first stage.

2. The adaptive enhanced weakly supervised hash retrieval method according to claim 1, characterized in that In step 2, the process of obtaining a binary code from image expression features through a deep hash network is modeled as a probabilistic decision-making process. The output of the deep hash network is regarded as the probability of outputting a bit '1' at each bit position. Based on the output probability, multiple binary codes are sampled, and each binary code is regarded as a probabilistic trial and error process.

3. The adaptive enhanced weakly supervised hash retrieval method according to claim 1, characterized in that In step 2, the first-stage loss includes strategy loss, balance loss, and cross-modal discriminative loss; The strategy loss directly optimizes the reward of the binary code, specifically: Among them, Θ h represents the deep hashing network parameters to be learned, Represents the binary code of the i-th image in the c-th trial and error process Rewards received; The balance loss achieves the compactness and high information content of the binary code by optimizing the probability that each bit has an equal output of '0' or '1', specifically: where B∈{0,1} N×L is the binary code matrix of the entire training set, · represents dot product, N is the number of images in the training set, L is the bit length, 1 and 0.5 represent vectors with values ​​of all 1 and 0.5; The cross-modal discriminative loss is to output binary code to embed semantic discriminative information, specifically: where Θ m represents the parameters of the projection network, r i represents the reconstructed user text annotation vector corresponding to the i-th image, t i is the user text annotation vector corresponding to the i-th image in the text annotation vector dataset, t j is the user text annotation vector corresponding to the jth image in the text annotation vector dataset, and τ is a manually set heat hyperparameter; The final first-stage optimization target L(Θ h ,Θ m ) is to optimize the sum of the three losses mentioned above, specifically: L(Θ h ,I m )=L i (I h ,I m )+αL p (I h )+βL b (I h ) where α and β represent hyperparameters that control the relative importance of policy loss and cross-modal discriminative loss.

4. The adaptive enhanced weakly supervised hash retrieval method according to claim 2, characterized in that: In calculating the retrieval accuracy of each trial-and-error process as a reward, in order to calculate the retrieval accuracy of the trial-and-error process, in the trial-and-error process, user text annotations are used as weak supervision. If the binary code used as the query and the image-text database include common user text annotations, the retrieval accuracy calculated for the binary code is used as a reward, maximizing the output probability of the trial-and-error process with the maximum reward, thereby enabling the deep hash network to generate distinguishable binary codes.

5. The adaptive enhanced weakly supervised hash retrieval method according to claim 1, characterized in that: In step 3, by calculating the text annotation phrase w i and text annotation phrase w m The exchange reward and non-exchange reward between them are used to determine whether the text annotation phrase w m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, specifically: If the exchange reward is less than or equal to the non-exchange reward, then the text is annotated with the phrase w m Do not replace the text annotation phrase w i , if the exchange reward is greater than the non-exchange reward, then the text annotation phrase w m Replace the text annotation phrase w i ; When the text is annotated with the phrase w m Do not replace the text annotation phrase w i When , the mean of all user text annotations in the text annotation vector dataset is used as the query vector, and the training image set is used as the retrieval database. The cosine similarity between the query vector and each user text annotation vector in the retrieval database is calculated, and the retrieval database is sorted in descending order based on the similarity to obtain the retrieval results; When the text is annotated with the phrase w m Replace the text annotation phrase w i When the text annotation phrase w in the text annotation vector dataset is i Remove and add text annotation phrase w m , the mean of all user text annotations in the updated text annotation vector dataset is used as the query vector, the training image set is used as the retrieval database, the cosine similarity between the query vector and each user text annotation vector in the retrieval database is calculated, and the retrieval database is sorted in descending order based on the similarity to obtain the retrieval results.

6. An adaptively enhanced weakly supervised hash retrieval system, characterized in that: Input the query image retrieved online into the trained retrieval model and output the image retrieval result; The retrieval model is trained as follows: Step 1: Obtain images and corresponding user text annotations, and build an image-text database based on all images and corresponding user text annotations, wherein the image-text database includes a training image set and a text annotation vector dataset; Step 2, first stage retrieval: Based on the frozen visual encoder in the visual language pre-training basic model, feature extraction is performed on the input image to obtain image expression features. Based on the frozen text encoder in the visual language pre-training basic model, feature extraction is performed on the input user text annotation features to obtain text expression features. The image expression features are converted into binary codes through the deep hash network, and the binary codes are converted into reconstructed user text annotation vectors through the projection network. The first stage loss is constructed to optimize the first stage deep hash network and projection network for training; Step 3: The second stage of redefining user text annotations: Calculate the similarity between the reconstructed user text annotation vector corresponding to each image and each user text annotation in the text annotation vector dataset, and sort based on the similarity to obtain the text annotation phrase w with the minimum similarity in the current text annotation vector dataset. i , and the text annotation phrase w that does not have the maximum similarity in the current text annotation vector dataset m , by calculating the text annotation phrase w i and text annotation phrase w m The exchange reward and non-exchange reward between them are used to determine whether to use text to mark the phrase w according to the relative size of the exchange reward and the non-exchange reward. m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, and using the updated text annotation vector dataset to guide the training of the deep hashing network and projection network in the first stage.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the weakly supervised hash retrieval method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of classification programs, which are used to be called by a processor and execute the weakly supervised hash retrieval method according to any one of claims 1 to 5.

Images