Self-adaptive enhanced weak supervision Hash retrieval method and self-adaptive enhanced weak supervision Hash retrieval system

Abstract

The invention discloses an adaptive enhanced weak supervision Hash retrieval method and system, and the method comprises the steps: inputting an online retrieved query image into a trained retrieval model, and outputting an image retrieval result; the training of the retrieval model comprises the following steps: constructing an image text database; in the first stage, feature extraction is performed on an image and a text to obtain an image expression feature and a text expression feature, and the image expression feature sequentially passes through a deep hash network and a projection network to obtain a reconstructed user text annotation vector; and the second stage: calculating the similarity between the reconstructed user text labeling vector corresponding to each image and each user text labeling in the text labeling vector data set, and judging whether to update the text labeling vector data set or not based on similarity sorting. Guiding the training of the deep hash network and the projection network in the first stage by using the updated text labeling vector data set; according to the weak supervision Hash retrieval method and system, the retrieval precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an adaptively enhanced weakly supervised hash retrieval method and system. Background Art

[0002] Image retrieval technology aims to return database images similar to the query image input by the user in real time from a large-scale database. In order to ensure real-time response and efficient storage, images are usually expressed using compact representations. The most common image compact representation technology is binary hashing, which represents images as binary codes, which can effectively reduce the storage space occupied by image representations, and the Hamming distance between binary codes can be efficiently calculated by the existing CPU architecture. Therefore, the binary hashing method can simultaneously ensure the real-time response and efficient storage of the retrieval system. In the binary hashing method, a discriminative binary hash function needs to be learned to improve the retrieval accuracy of the binary hashing method, thereby ensuring a satisfactory retrieval effect.

[0003] According to the use of supervisory information in the hash function learning process, the existing binary hash methods can be divided into three categories: unsupervised hash methods, semi-supervised hash methods, and supervised hash methods. Among them, the unsupervised hash method does not use any supervisory information in the hash function learning process, the supervised hash method uses supervisory information to train the hash function, and the semi-supervised hash method uses supervisory information of part of the data during the training process. In general, the more supervisory information used in the learning process, the stronger the hash function discrimination of the binary hash method and the better the retrieval accuracy. However, it usually takes a lot of manpower and material resources to collect accurate supervisory information.

[0004] In recent years, weakly supervised hashing methods have begun to be studied, using easily available and noisy user text annotation information as weak supervision information to guide the learning of hash functions. Existing image sharing websites allow users to make simple text annotation descriptions when uploading images. These text descriptions usually describe diverse information such as image content, attributes, camera information, weather conditions, geographic location, etc., which can be used as weak supervision information for hash function learning. How to mine supervision information related to image retrieval tasks from these diverse text descriptions is a key challenge faced by existing weakly supervised hashing methods. According to the application method of user text annotations, existing weakly supervised hashing methods can be divided into two categories, one is direct application method and the other is update application method. Among them, the direct application method directly applies user text annotations as supervision information, and learns to embed the semantic information contained in user text annotations into the hash function by designing the objective function and network architecture. In the process of hash function learning, the update application method designs a method to update the user text annotations, and uses the updated user text annotation information as more accurate supervision information to guide the hash function to learn better. Generally speaking, the update application method can usually distill more accurate supervision information, thereby achieving better retrieval accuracy.

[0005] On the other hand, visual language pre-training technology has trained a series of powerful basic models. Most of the existing research on visual language pre-training technology explores different model architectures and pre-training objectives, and has achieved excellent performance in different downstream tasks (such as image caption generation, visual question answering, image-text matching, etc.).

[0006] However, there is little work exploring the application of visual language pre-training basic models in large-scale image retrieval tasks. At the same time, existing supervised and semi-supervised hashing methods require the use of manually annotated supervisory information, which consumes manpower and material resources. Although unsupervised hashing methods do not require the use of manually annotated supervisory information, due to the lack of supervisory information guidance in the hash function learning process, the semantic discrimination of the hash function is weak, and the retrieval accuracy is generally not as good as the hashing method that uses supervisory information. Weakly supervised hashing methods use user text annotations that come with the image. Although these user annotations are not completely related to the retrieval task and may contain certain text annotations that are not related to the task as noise, they can still provide certain content supervision information, so the retrieval accuracy is generally better than the unsupervised hashing method.

[0007] In weakly supervised hashing methods, direct application methods directly apply noisy user text annotations without screening. The learning process of the hash function is disturbed by user annotations that are irrelevant to the task. Although the retrieval performance is worse than that of the unsupervised hashing method, the retrieval accuracy is generally not as good as that of the update application method. On the other hand, there are relatively few existing update application methods. Although they can update user text annotations to a certain extent, their update process usually does not consider the contextual similarity of the text or only considers the similarity between the image content and the text. Therefore, the updated user text annotations are not completely related to the image retrieval task and cannot guarantee the image retrieval accuracy. In addition, existing weakly supervised hashing methods usually use traditional deep convolutional models as the backbone network, and their retrieval accuracy is limited by the semantic distinction ability of traditional convolutional models. Summary of the invention

[0008] Based on the technical problems existing in the background technology, the present invention proposes an adaptive enhanced weakly supervised hash retrieval method and system to improve the retrieval accuracy.

[0009] The present invention proposes an adaptive enhanced weakly supervised hash retrieval method, which inputs the query image of online retrieval into the trained retrieval model and outputs the image retrieval result;

[0010] The training of the retrieval model is as follows:

[0011] Step 1: Obtain images and corresponding user text annotations, and build an image-text database based on all images and corresponding user text annotations, wherein the image-text database includes a training image set and a text annotation vector data set;

[0012] Step 2, retrieval in the first stage: extract features of the input image based on the frozen visual encoder in the visual language pre-training basic model to obtain image expression features, extract features of the input user text annotation based on the frozen text encoder in the visual language pre-training basic model to obtain text expression features, obtain binary codes from the image expression features through the deep hash network, and obtain reconstructed user text annotation vectors from the binary codes through the projection network, and construct the first stage loss to optimize the first stage deep hash network and projection network for training;

[0013] Step 3: The second stage of redefining user text annotations: Calculate the similarity between the reconstructed user text annotation vector corresponding to each image and each user text annotation in the text annotation vector dataset, and sort based on the similarity to obtain the text annotation phrase w with the minimum similarity in the current text annotation vector dataset. i , and the text annotation phrase w that does not have the maximum similarity in the current text annotation vector dataset m , by calculating the text annotation phrase w i and the text annotation phrase wm The exchange reward and non-exchange reward between them are used to determine whether to mark the phrase w through the text according to the relative size of the exchange reward and the non-exchange reward. m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, and using the updated text annotation vector dataset to guide the training of the deep hashing network and projection network in the first stage.

[0014] In step 2, the process of obtaining binary codes from image expression features through a deep hash network is modeled as a probabilistic decision process, and the output of the deep hash network is regarded as the probability of outputting bit '1' at each bit position; based on the output probability, multiple binary codes are sampled, and each binary code is regarded as a probabilistic trial and error process.

[0015] Further, in step 2, the first stage loss includes strategy loss, balance loss and cross-modal discriminative loss;

[0016] The strategy loss directly optimizes the reward of the binary code, specifically:

[0017]

[0018] Among them, Θ h represents the deep hashing network parameters to be learned, Represents the binary code of the i-th image in the c-th trial and error process Rewards received;

[0019] The balance loss achieves compactness and high information content of binary code by optimizing the probability that each bit has an equal output of '0' or '1', specifically:

[0020]

[0021] where B∈{0,1} N×L is the binary code matrix of the entire training set, · represents dot product, N is the number of images in the training set, L is the bit length, 1 and 0.5 represent vectors with values ​​of all 1 and 0.5;

[0022] The cross-modal discriminative loss is to output binary codes to embed semantic discriminative information, specifically:

[0023]

[0024] where Θ m represents the parameters of the projection network, r i represents the reconstructed user text annotation vector corresponding to the i-th image, t i is the user text annotation vector corresponding to the i-th image in the text annotation vector dataset, t jis the user text annotation vector corresponding to the jth image in the text annotation vector dataset, and τ is a manually set heat hyperparameter;

[0025] Finally, the first stage optimization target L(Θ h ,Θ m ) is to optimize the sum of the three losses mentioned above, specifically:

[0026] L(Θ,Θm)=Li(Θ,Θm)+αLp(Θh)+βL(Θh);

[0027] where α and β represent hyperparameters that control the relative importance of policy loss and cross-modal discriminative loss.

[0028] Calculate the retrieval accuracy of each trial-and-error process as a reward, use the binary code of the trial-and-error process as a query, calculate the Hamming distance between the query binary code and the binary code in the training image set, sort the images in the training set in ascending order of the Hamming distance, and obtain the retrieval results;

[0029] Furthermore, in calculating the retrieval accuracy of each trial and error process as a reward, in order to calculate the retrieval accuracy of the trial and error process, in the trial and error process, user text annotations are used as weak supervision. If the binary code used as the query and the image text database include common user text annotations, the retrieval accuracy calculated for the binary code is used as a reward, maximizing the output probability of the trial and error process with the maximum reward, thereby enabling the deep hash network to generate distinguishable binary codes.

[0030] Furthermore, in step 3, by calculating the text annotation phrase w i and the text annotation phrase w m The exchange reward and non-exchange reward between them are used to determine whether the text annotation phrase w m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, specifically:

[0031] If the exchange reward is less than or equal to the non-exchange reward, then the text annotation phrase w m Do not replace the text annotation phrase w i , if the exchange reward is greater than the non-exchange reward, then the text annotation phrase w m Replace the text annotation phrase w i ;

[0032] When the text is annotated with a phrase w m Do not replace the text annotation phrase w iWhen , the mean of all user text annotations in the text annotation vector dataset is used as the query vector, and the training image set is used as the retrieval database. The cosine similarity between the query vector and each user text annotation vector in the retrieval database is calculated, and the retrieval database is sorted in descending order based on the similarity to obtain the retrieval results.

[0033] When the text is annotated with a phrase w m Replace the text annotation phrase w i When the text annotation phrase w in the text annotation vector dataset is i Remove and add text annotation phrase w m , the mean of all user text annotations in the updated text annotation vector dataset is used as the query vector, the training image set is used as the retrieval database, the cosine similarity between the query vector and each user text annotation vector in the retrieval database is calculated, and the retrieval database is sorted in descending order based on the similarity to obtain the retrieval results.

[0034] An adaptive enhanced weakly supervised hash retrieval system that inputs the query image of online retrieval into the trained retrieval model and outputs the image retrieval result;

[0035] The training of the retrieval model is as follows:

[0036] Step 1: Obtain images and corresponding user text annotations, and build an image-text database based on all images and corresponding user text annotations, wherein the image-text database includes a training image set and a text annotation vector data set;

[0037] Step 2, retrieval in the first stage: extract features of the input image based on the frozen visual encoder in the visual language pre-training basic model to obtain image expression features, extract features of the input user text annotation based on the frozen text encoder in the visual language pre-training basic model to obtain text expression features, obtain binary codes from the image expression features through the deep hash network, and obtain reconstructed user text annotation vectors from the binary codes through the projection network, and construct the first stage loss to optimize the first stage deep hash network and projection network for training;

[0038] Step 3: The second stage of redefining user text annotations: Calculate the similarity between the reconstructed user text annotation vector corresponding to each image and each user text annotation in the text annotation vector dataset, and sort based on the similarity to obtain the text annotation phrase w with the minimum similarity in the current text annotation vector dataset. i , and the text annotation phrase w that does not have the maximum similarity in the current text annotation vector dataset m , by calculating the text annotation phrase w i and the text annotation phrase w mThe exchange reward and non-exchange reward between them are used to determine whether to use the text annotation phrase w according to the relative size of the exchange reward and the non-exchange reward. m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, and using the updated text annotation vector dataset to guide the training of the deep hashing network and projection network in the first stage.

[0039] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the weakly supervised hash retrieval method as described above when executing the computer program.

[0040] A computer-readable storage medium stores a plurality of classification programs, wherein the plurality of classification programs are used to be called by a processor and execute the weakly supervised hash retrieval method as described above.

[0041] The advantages of the adaptively enhanced weakly supervised hash retrieval method and system provided by the present invention are: by iteratively and alternately optimizing the weak hash function learning method and the user text annotation adaptive update method, and using the retrieval accuracy as a reward to optimize the two-stage training process, the user text annotation can be more relevant to the retrieval task and the image content, and serve as more accurate supervisory information to guide the learning of the hash function. Further, the learned deep hash network can generate more discriminative binary codes, thereby obtaining better online retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a structural schematic diagram of the present invention; DETAILED DESCRIPTION

[0043] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0044] Aiming at the problems existing in the existing binary hashing methods, an adaptive enhanced weakly supervised hashing retrieval method is designed. The hashing function is learned and the user text annotations are updated alternately and iteratively. By designing a probabilistic decision-making hashing function learning method and a user text annotation adaptive updating method, the retrieval accuracy is directly optimized during the training process to obtain a more discriminative hashing function, thereby improving the retrieval accuracy during the test. Specifically:

[0045] Applying the image-text alignment capabilities of the visual language pre-training basic model, a weak hash function learning method and a user text annotation adaptive update method with alternating iterative optimization are designed. Specifically, the weak hash function learning method in this embodiment is modeled as a probabilistic decision-making learning method, which uses the updated user text annotations as semantic supervision to directly optimize the retrieval accuracy of the hash function. The user text annotation adaptive update method in this embodiment is also modeled as a probabilistic decision-making update method, which uses the binary code generated by the hash function to calculate the neighbor relationship between images, so that the updated user text annotation can maximize the image neighbor relationship, and then directly optimize the retrieval accuracy. By iteratively and alternately optimizing the weak hash function learning method and the user text annotation adaptive update method, this embodiment can effectively obtain user text annotations directly related to the retrieval task, and obtain a hash function with more semantic distinction, thereby further improving the accuracy of image retrieval. This embodiment is described in detail below.

[0046] like Figure 1 As shown, the present invention proposes an adaptive enhanced weakly supervised hash retrieval method, which inputs the query image of online retrieval into the trained retrieval model and outputs the image retrieval result;

[0047] The retrieval model consists of a two-stage learning process of alternating iterative optimization, which consists of deep weakly supervised hash function learning and adaptive updating of user text annotations. In the first stage, the deep weakly supervised hash function is directly optimized using user text annotations as weakly supervised information. In the second stage, the user text annotations are adaptively updated based on the binary codes learned in the previous stage and the reconstructed user text annotation vectors.

[0048] The retrieval model is trained as follows:

[0049] Step 1: Obtain images and corresponding user text annotations, and build an image-text database based on all images and corresponding user text annotations, wherein the image-text database includes a training image set and a text annotation vector data set;

[0050] Step 2, retrieval in the first stage: extract features of the input image based on the frozen visual encoder in the visual language pre-training basic model to obtain image expression features, extract features of the input user text annotation based on the frozen text encoder in the visual language pre-training basic model to obtain text expression features, obtain binary codes from the image expression features through the deep hash network, and obtain reconstructed user text annotation vectors from the binary codes through the projection network, and construct the first stage loss to optimize the first stage deep hash network and projection network for training;

[0051] The deep hash network must contain a fully connected layer and an activation function layer to ensure that its output is in the range of [0,1]. In order to embed task-specific information in the output binary code, a projection network is designed to reconstruct the user-annotated text vector from the output binary code of the deep hash network. The projection network contains three fully connected layers, each of the first two layers has a ReLU activation function, and the output of the last layer needs to be L2 modulus normalized.

[0052] The learning goal of the first stage is to learn a binary hash function to generate distinguishable binary codes for retrieval tasks. Based on the image-text alignment capabilities of user text annotation and visual language pre-training basic models, this embodiment designs three types of losses for training and optimizing hash functions (deep hashing network and projection network). The three types of losses are: strategy loss, balance loss, and cross-modal discriminative loss.

[0053] (a1) Strategy loss

[0054] First, in order to overcome the discontinuous objective function and optimization difficulties caused by binary codes, this embodiment designs a novel strategy loss to directly optimize the reward of binary codes. Specifically, the binarization process is modeled as a probabilistic decision process, and the output of the deep hash network is regarded as the probability of outputting bit '1' at each bit position. Based on this probability, multiple binary codes are sampled, and each binary code is regarded as a probabilistic trial and error process.

[0055] In order to calculate the retrieval accuracy of the trial and error process, user text annotations are used as weak supervision, that is, the query image (the query is represented by the binary code of the query image) and the image text database are considered to be related if they contain common user text annotations. The retrieval accuracy is calculated as the reward of the trial and error process. Through the strategy loss, the output probability of the trial and error process with the maximum reward is maximized.

[0056] The strategy loss directly optimizes the reward of the binary code, specifically:

[0057]

[0058] Among them, Θ h represents the deep hashing network parameters to be learned, Represents the binary code of the i-th training sample (i.e. image) in the c-th trial and error process The rewards received include The retrieval accuracy obtained by searching the image-text database for the query vector.

[0059] Specifically, the retrieval accuracy is obtained by taking the binary code of the trial-and-error process as the query, calculating the Hamming distance between the query binary code and the binary code in the training image set, sorting the images in the training image set in ascending order of the Hamming distance, obtaining the retrieval result, and calculating the accuracy of the retrieval result based on the training set label.

[0060] (a2) Cross-modal discriminative loss

[0061] Then, in order to embed semantic distinctiveness directly in the binary code, this embodiment designs a cross-modal discriminative loss so that the reconstructed user-annotated text vector and the user text annotation vector of the image are semantically distinctive, that is, the similarity between the reconstructed user-annotated text vector and the user text annotation vector of the input image is as large as possible, and the similarity between the user text annotation vectors of different images is as small as possible, while optimizing the deep hashing network and the projection network.

[0062] The cross-modal discriminative loss is to output binary codes to embed semantic discriminative information, specifically:

[0063]

[0064] where Θ m represents the parameters of the projection network, r i represents the reconstructed user text annotation vector, t i is the user text annotation vector in the text annotation vector dataset, which is obtained by aggregating the text phrase vector corresponding to each image, that is, where w i,k is the kth text phrase w in the text annotation set of the i-th image i,k The text expression features, N k is the total number of text phrases in the text annotation set of the i-th image, and τ is an artificially set heat hyperparameter.

[0065] (a3) Finally, in order to maintain the compactness of the output binary code, this embodiment designs a balanced loss so that each bit position has the same probability of outputting a bit '0' or '1', maximizing the amount of information that can be represented by each bit position.

[0066] That is, the balanced loss achieves the compactness and high information content of the binary code by optimizing the probability that each bit has an equal output of '0' or '1', specifically:

[0067]

[0068] where B∈{0,1} N×Lis the binary code matrix of the entire training set, · represents dot product, N is the total number of images in the training image set, L is the bit length, and 1 and 0.5 represent vectors with values ​​of all 1 and 0.5.

[0069] Finally, the first stage optimization target L(Θ h ,Θ m ) is to optimize the sum of the three losses mentioned above, specifically:

[0070] L(Θ,Θm)=Li(Θ,Θm)+αLp(Θh)+βL(Θh);

[0071] Where α and β represent hyperparameters that control the relative importance of the corresponding losses.

[0072] By simultaneously optimizing the above three losses, the first-stage deep weakly supervised hash function learning module (first-stage loss) designed in this embodiment can simultaneously optimize the deep hash network and the projection network.

[0073] Step 3: The second stage of redefining user text annotations: Calculate the similarity between the reconstructed user text annotation vector corresponding to each image and each user text annotation in the text annotation vector dataset, and sort based on the similarity to obtain the text annotation phrase w with the minimum similarity in the current text annotation vector dataset. i , and the text annotation phrase w that does not have the maximum similarity in the current text annotation vector dataset m , by calculating the text annotation phrase w i and the text annotation phrase w m The exchange reward and non-exchange reward between them are used to determine whether to use the text annotation phrase w according to the relative size of the exchange reward and the non-exchange reward. m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, and using the updated text annotation vector dataset to guide the training of the deep hashing network and projection network in the first stage.

[0074] In the second stage, based on the binary code learned in the previous stage and the reconstructed user text annotation vector, the user text annotation of each image is adaptively redefined by analyzing the neighbor relationship. By calculating the Hamming distance between the binary code and the database image binary code, and the Euclidean distance between the reconstructed text vector and the database text annotation vector, a relatively reliable image neighbor relationship can be obtained from the K neighbor images with the minimum distance. Through the neighbor relationship analysis of each image, its user text annotation is calculated by the consistency with the neighbor user annotation, and then adaptively updated.

[0075] The learning goal of the second stage is to redefine the user text annotations and filter out the noise in the existing user annotation set of each image, that is, the user text annotations that are irrelevant to the retrieval task or the image content. Through the training of the first stage, the binary code output and the reconstructed user text annotation vector can carry a certain semantic distinction. Based on the output of the first stage, this embodiment designs a probabilistic decision-making user text annotation update method, which iteratively removes the text annotation with the smallest reward from the current text annotation vector data set and adds the text annotation with the largest reward.

[0076] Specifically, for each image, given the reconstructed user text annotation vector, the similarity between it and the user text annotations in the entire text annotation vector dataset is calculated in turn. This similarity can be regarded as the correlation between the user text annotation and the image. Consider the text annotation phrase w with the maximum correlation (similarity) that is not in the current image user annotation set m , and the text annotation phrase w with the minimum relevance (similarity) in the current image user annotation set i By calculating the exchange reward and non-exchange reward of the two user text annotation phrases, it is decided whether to exchange the user text annotations, and then the user text annotation set is updated. In this embodiment, exchange and non-exchange are two trial and error processes.

[0077] This embodiment uses retrieval accuracy as a reward for different trial and error processes. The reward is calculated by the image retrieval accuracy of the training image set obtained by using the text annotation phrase representation as the query. Whether to exchange is selected based on the relative size of the exchange reward and the non-exchange reward. If the exchange reward is less than or equal to the non-exchange reward, then no exchange (i.e., no replacement) is performed. If the exchange reward is greater than the non-exchange reward, then exchange (i.e., replacement) is performed.

[0078] When there is no exchange (i.e. no replacement), the mean of all user text annotations in the text annotation vector dataset is used as the query vector, and the training image set is used as the retrieval database. The cosine similarity between the query vector and each user text annotation vector in the database image is calculated, and the retrieval database is sorted in descending order based on the similarity to obtain the retrieval results.

[0079] When exchanging (i.e. replacing), the text annotation vector dataset is first updated to remove w i , add w m , and calculate the mean of all user text annotations in the updated text annotation vector dataset as the query vector, use the training image set as the retrieval database, calculate the cosine similarity between the query vector and each user text annotation vector in the database image, and sort the retrieval database in descending order based on the similarity to obtain the retrieval results.

[0080] Different from the first stage, which uses user text annotation as the definition of supervisory information, the second stage uses the output binary code of the first stage to calculate the database image with the minimum Hamming distance as the neighbor image of the current image, that is, the related image, and then uses it to calculate the retrieval accuracy of the above two exchange and non-exchange processes as a reward. If the exchange process obtains more rewards, the text annotation vector dataset and training image set of the current image are updated; otherwise, the original text annotation vector dataset and training image set are maintained unchanged.

[0081] By iteratively updating the text annotation vector dataset of each image, the user text annotation of the entire training image set can finally be more relevant to the retrieval task and the image content, and can serve as better supervision information to guide the deep hash function learning in the first stage. Therefore, this embodiment iteratively interleaves the two stages until convergence.

[0082] Through steps one to three, this embodiment proposes a weakly supervised hash retrieval method, which trains a deep hash network, a projection network, and adaptively updates user text annotations by alternately iteratively optimizing a weak hash function learning method and a user text annotation adaptive update method. After the training is completed, the deep hash network can be used for large-scale image retrieval, extracting the binary code of each image in a large-scale image database, and can significantly reduce the storage consumption of a large-scale image database. When searching online, the user gives a query image, extracts its binary code through a deep hash network, and then calculates the similarity with the database image by calculating the Hamming distance between the binary codes, and sorts the database images in order from small to large according to the Hamming distance to obtain the retrieval results. Since the Hamming distance calculation only involves XOR sum operations, the distance calculation efficiency is relatively high, which ensures the real-time performance of large-scale image retrieval. Therefore, the adaptive enhanced weakly supervised hash proposed in this embodiment has lower storage consumption and higher computational efficiency.

[0083] On the other hand, this embodiment iteratively and alternately optimizes the weak hash function learning method and the user text annotation adaptive update method, and uses retrieval accuracy as a reward to optimize the two-stage training process. User text annotation can be more relevant to the retrieval task and the image content, and serve as more accurate supervisory information to guide the learning of the hash function. Further, the learned deep hash network can generate more discriminative binary codes, thereby obtaining better online retrieval accuracy.

[0084] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An adaptive enhanced weakly supervised hash retrieval method, characterized in that: Input the query image retrieved online into the trained retrieval model and output the image retrieval result; The training of the retrieval model is as follows: Step 1: Obtain images and corresponding user text annotations, and build an image-text database based on all images and corresponding user text annotations, wherein the image-text database includes a training image set and a text annotation vector data set; Step 2, retrieval in the first stage: extract features of the input image based on the frozen visual encoder in the visual language pre-training basic model to obtain image expression features, extract features of the input user text annotation based on the frozen text encoder in the visual language pre-training basic model to obtain text expression features, obtain binary codes from the image expression features through the deep hash network, and obtain reconstructed user text annotation vectors from the binary codes through the projection network, and construct the first stage loss to optimize the first stage deep hash network and projection network for training; Step 3: The second stage of redefining user text annotations: Calculate the similarity between the reconstructed user text annotation vector corresponding to each image and each user text annotation in the text annotation vector dataset, and sort based on the similarity to obtain the text annotation phrase w with the minimum similarity in the current text annotation vector dataset. i , and the text annotation phrase w that does not have the maximum similarity in the current text annotation vector dataset m , by calculating the text annotation phrase w i and the text annotation phrase w m The exchange reward and non-exchange reward between them are used to determine whether to use the text annotation phrase w according to the relative size of the exchange reward and the non-exchange reward. m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, and using the updated text annotation vector dataset to guide the training of the deep hashing network and projection network in the first stage.

2. The adaptive enhanced weakly supervised hash retrieval method according to claim 1, characterized in that: In step 2, the process of obtaining binary codes from image expression features through a deep hash network is modeled as a probabilistic decision process, and the output of the deep hash network is regarded as the probability of outputting bit '1' at each bit position; based on the output probability, multiple binary codes are sampled, and each binary code is regarded as a probabilistic trial and error process.

3. The adaptive enhanced weakly supervised hash retrieval method according to claim 1, characterized in that: In step 2, the first stage loss includes strategy loss, balance loss and cross-modal discriminative loss; The strategy loss directly optimizes the reward of the binary code, specifically: Among them, Θ h represents the deep hashing network parameters to be learned, Represents the binary code of the i-th image in the c-th trial and error process Rewards received; The balance loss achieves compactness and high information content of binary code by optimizing the probability that each bit has an equal output of '0' or '1', specifically: where B∈{0,1} N×L is the binary code matrix of the entire training set, · represents dot product, N is the number of images in the training set, L is the bit length, 1 and 0.5 represent vectors with values ​​of all 1 and 0.5; The cross-modal discriminative loss is to output binary codes to embed semantic discriminative information, specifically: where Θ m represents the parameters of the projection network, r i represents the reconstructed user text annotation vector corresponding to the i-th image, t i is the user text annotation vector corresponding to the i-th image in the text annotation vector dataset, t j is the user text annotation vector corresponding to the jth image in the text annotation vector dataset, and τ is a manually set heat hyperparameter; Finally, the first stage optimization target L(Θ h ,Θ m ) is to optimize the sum of the three losses mentioned above, specifically: L(Θ h ,I m )=L i (I h ,I m )+αL p (I h )+βL b (I h ) where α and β represent hyperparameters that control the relative importance of policy loss and cross-modal discriminative loss.

4. The adaptive enhanced weakly supervised hash retrieval method according to claim 2, characterized in that: In calculating the retrieval accuracy of each trial and error process as a reward, in order to calculate the retrieval accuracy of the trial and error process, in the trial and error process, the user text annotation is used as weak supervision. If the binary code used as the query and the image text database include common user text annotations, the retrieval accuracy calculated for the binary code is used as a reward, maximizing the output probability of the trial and error process with the maximum reward, so that the deep hash network generates distinguishable binary codes.

5. The adaptive enhanced weakly supervised hash retrieval method according to claim 1, characterized in that: In step 3, by calculating the text annotation phrase w i and the text annotation phrase w m The exchange reward and non-exchange reward between them are used to determine whether the text annotation phrase w m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, specifically: If the exchange reward is less than or equal to the non-exchange reward, then the text annotation phrase w m Do not replace the text annotation phrase w i , if the exchange reward is greater than the non-exchange reward, then the text annotation phrase w m Replace the text annotation phrase w i ; When the text is annotated with a phrase w m Do not replace the text annotation phrase w i When , the mean of all user text annotations in the text annotation vector dataset is used as the query vector, and the training image set is used as the retrieval database. The cosine similarity between the query vector and each user text annotation vector in the retrieval database is calculated, and the retrieval database is sorted in descending order based on the similarity to obtain the retrieval results. When the text is annotated with a phrase w m Replace the text annotation phrase w i When the text annotation phrase w in the text annotation vector dataset is i Remove and add text annotation phrase w m , the mean of all user text annotations in the updated text annotation vector dataset is used as the query vector, the training image set is used as the retrieval database, the cosine similarity between the query vector and each user text annotation vector in the retrieval database is calculated, and the retrieval database is sorted in descending order based on the similarity to obtain the retrieval results.

6. An adaptive enhanced weakly supervised hash retrieval system, characterized in that: Input the query image retrieved online into the trained retrieval model and output the image retrieval result; The training of the retrieval model is as follows: Step 1: Obtain images and corresponding user text annotations, and build an image-text database based on all images and corresponding user text annotations, wherein the image-text database includes a training image set and a text annotation vector data set; Step 2, retrieval in the first stage: extract features of the input image based on the frozen visual encoder in the visual language pre-training basic model to obtain image expression features, extract features of the input user text annotation based on the frozen text encoder in the visual language pre-training basic model to obtain text expression features, obtain binary codes from the image expression features through the deep hash network, and obtain reconstructed user text annotation vectors from the binary codes through the projection network, and construct the first stage loss to optimize the first stage deep hash network and projection network for training; Step 3: The second stage of redefining user text annotations: Calculate the similarity between the reconstructed user text annotation vector corresponding to each image and each user text annotation in the text annotation vector dataset, and sort based on the similarity to obtain the text annotation phrase w with the minimum similarity in the current text annotation vector dataset. i , and the text annotation phrase w that does not have the maximum similarity in the current text annotation vector dataset m , by calculating the text annotation phrase w i and the text annotation phrase w m The exchange reward and non-exchange reward between them are used to determine whether to use the text annotation phrase w according to the relative size of the exchange reward and the non-exchange reward. m Replace the text annotation phrase w i , thereby updating the text annotation vector dataset, and using the updated text annotation vector dataset to guide the training of the deep hashing network and projection network in the first stage.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the weakly supervised hash retrieval method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the weakly supervised hash retrieval method as described in any one of claims 1 to 5.

Images