Unsupervised fine-grained image hash encoding method, storage medium and electronic device

Abstract

The present invention discloses an unsupervised fine-grained image hash encoding method, storage medium, and electronic device. The method includes the following steps: improving the existing contrastive learning method through an asymmetric data augmentation method to construct an asymmetric contrastive learning framework adapted to fine-grained objects; introducing Fisher vectors to construct end-to-end part-level dense contrastive learning to capture and model the part-level information of fine-grained objects; and constructing a self-consistent hash learning module to introduce reconstruction tasks into hash learning based on an encoder-decoder structure to learn comprehensive features. The present invention uses a contrastive learning method based on asymmetric augmentation to model the part-level information of an image using Fisher vectors and part-level dense contrastive learning, constructing a self-consistent hash learning module to guide hash learning to retain complete and important overall image feature information, thereby achieving higher image retrieval accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, covering two directions: fine-grained image retrieval and unsupervised hash learning. Specifically, it relates to an unsupervised hash coding method based on asymmetric enhancement in fine-grained image retrieval. Background Art

[0002] Fine-grained image retrieval is an important part of the field of fine-grained image analysis, and the fine-grained image retrieval task has received widespread attention from researchers in recent years. In addition to requiring the correct estimation of subcategories, this task also requires ranking all instances in the database, where images from the same subcategory are prioritized based on the subtle details presented in the query. Compared to general image retrieval, which focuses on retrieving similar images based on content similarity (such as texture, color, and shape), fine-grained image retrieval focuses on retrieving images of the same category type (such as lower species of the same animal or the same car model). What makes it more challenging is that objects in fine-grained categories can have subtle differences in posture, scale, and orientation.

[0003] In practical applications of fine-grained image retrieval, they face significant time-consuming drawbacks when searching for nearest neighbors in a wide range of image databases. Image hashing is a popular and effective approximate nearest neighbor search technique that is very suitable for large-scale fine-grained data. By mapping high-dimensional feature vectors into low-dimensional binary hash codes so that the neighbor relationships in the original space are preserved as much as possible, the computational and storage overhead in the retrieval system can be greatly reduced, and it has attracted widespread attention from researchers. However, it is worth noting that these methods still rely on fine-grained category labels, which are difficult to obtain in practice and often require professionals to spend a lot of time labeling. This means extremely high costs, which limits the application scenarios and prevents popularization. Therefore, unsupervised hashing that does not rely on fine-grained category labels will significantly reduce the labeling cost of fine-grained retrieval.

[0004] However, current unsupervised hashing algorithms have achieved good performance in data retrieval, but are limited to coarse-grained data retrieval. In many cases, such as searching for dog images, people not only want to retrieve images of dogs rather than other animals, but also want to know the breed of the dog, such as Corgi or Samoyed. In this case, existing unsupervised hashing methods mostly rely on traditional contrastive learning, which uses a random data augmentation strategy to transform a given image to produce two views of the same example, which are considered as positive sample pairs, while other examples (usually from different categories) are considered as negative sample pairs. Each sample is passed through the same neural network and hash mapping to generate a corresponding hash code. By narrowing the hash codes between positive sample pairs and widening the distance between the hash codes between negative sample pairs, a binary hash code with similarity preservation can be generated.

[0005] However, certain types of augmentation can disrupt the original discriminative patterns of a specific fine-grained category, or even fundamentally alter that category. This approach introduces confusion in the context of self-supervised learning of fine-grained objects and lacks specific design for fine-grained data, resulting in very low retrieval accuracy on fine-grained datasets. Therefore, it is necessary to propose a new unsupervised hashing method that can achieve high-accuracy responses in fine-grained retrieval environments, which is of practical significance. Summary of the Invention

[0006] The purpose of the present invention is to provide an unsupervised fine-grained image hash coding method based on asymmetric enhancement.

[0007] The technical solution for achieving the purpose of the present invention is:

[0008] An unsupervised fine-grained image hash coding method based on asymmetric enhancement includes the following steps:

[0009] Step 1: Using an asymmetric data augmentation method, we construct an asymmetric contrastive learning framework that is adaptable to fine-grained objects.

[0010] Step 2: Fisher vectors are introduced to construct end-to-end part-level dense contrastive learning to capture and model part-level information of fine-grained objects.

[0011] Step 3: Construct a self-consistent hashing learning module and introduce the reconstruction task into hashing learning based on the encoder-decoder structure.

[0012] A computer storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0013] An electronic device includes a memory and one or more processors, wherein the memory is used to store one or more programs; when the one or more programs are executed by the one or more processors, the method described is implemented

[0014] Compared with the existing technology, the significant advantages of this invention are as follows: (1) targeting the characteristics of fine-grained image data, a novel asymmetric enhancement method is used to construct positive and negative sample pairs and integrate them into the contrastive learning framework; (2) dense contrastive learning and Fisher vectors are used to capture and model the part-level information of fine-grained objects and learn more discriminative representations; (3) the reconstruction task is introduced into hash learning, and the codec structure is used to guide hash learning to retain relatively complete and important overall image feature information; BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a schematic diagram of an unsupervised fine-grained image hash coding method based on asymmetric enhancement according to the present invention. DETAILED DESCRIPTION

[0016] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0017] like Figure 1 As shown, the present invention provides an unsupervised fine-grained image hash coding method based on asymmetric enhancement, which specifically includes the following steps:

[0018] Step 1: Using an asymmetric data augmentation method, we construct an asymmetric contrastive learning framework that is adaptable to fine-grained objects.

[0019] Specifically, for sample I, asymmetric enhancement creates three enhanced views: anchor view I (a) , positive sample view I (p) and negative sample view I (n) . Among them I (a) Only through the Resize operation on I generation, I (p) Only generated by RandomCrop operation on I, I (n) It undergoes two destructive enhancements, namely ColorJitter and ElasticTransform operations, to induce it to produce substantial changes related to fine-grained objects.

[0020] During the training process, for a sample I in a mini-batch i (i=(1,2,…,N), first specify it as the anchor view Then a positive sample view is generated, relative to the anchor view, the anchor views and positive sample views of the remaining N-1 samples and the original negative sample view Composed with The related 2N-1 negative sample views. Then, the similarity between the binary hash codes u generated by the model is calculated:

[0021]

[0022] Among them, I i is a sample in a mini-batch, i=(1,2,…,N), and Sample I i Anchor view, positive sample view and negative sample view of Corresponding to The hash code of Corresponding to The hash code of Corresponding to The hash code of . is the contrast loss of the positive view. The temperature hyperparameter η is set to 0.3.

[0023] Step 2: Fisher vectors are introduced to construct end-to-end part-level dense contrastive learning to capture and model part-level information of fine-grained objects.

[0024] Target parts, such as the red head and spotted tail of a bird, play a crucial role in the representation of fine-grained visual objects. Capturing these discriminative target parts and subsequently deriving strong part-level features is crucial for accurate fine-grained image hashing. Specifically, for an input sample I, by feeding it into a pure convolutional neural network model, we can obtain a 3D activation tensor through the following steps:

[0025] T=Φ CNN (I)∈R C×H×W #(2)

[0026] Where C, H and W are the depth, height and width of the activation tensor respectively. T can also be viewed as a set of H×W depth descriptors, represented as X = {x t}, t = (1, 2, ..., H × W). Subsequently, for X, we use Gaussian Mixture Model (GMM) to cluster these deep descriptors into K clusters shared across all categories, where each cluster may correspond to a specific part-level semantics. Mathematically, we use λ = {ω k ,μ k ,σ k ; k = 1, ..., K} denotes the parameters of a GMM with K components / clusters, where ω k , μ k and σ k are the mixing weight, mean vector and covariance matrix of the kth Gaussian component respectively. It is worth noting that the mixing weight ω k Subject to the following constraints:

[0027]

[0028] This is also used as the depth descriptor x associated with the kth cluster t After assignment, the mean of the depth descriptors within each cluster is calculated to obtain the part-level prototype:

[0029]

[0030] Among them, Ω k represents the index set of deep descriptors corresponding to the kth cluster. We then perform dense contrastive learning by extending the original contrastive loss function, Formula (1), to a part-oriented paradigm, which can be expressed as follows:

[0031]

[0032] Among them, p i,k Indicates I i The k-th part-level representation of , where the superscripts a, p, and n represent the anchor, positive sample, and negative sample negative views, respectively. is the dense contrast loss for the positive view.

[0033] Then, these K components are clustered into a high-dimensional vector containing high-order statistical information, and the mean vector u of the kth Gaussian component is used k and the covariance matrix σ k , we can get the first-order and second-order statistics as follows:

[0034]

[0035]

[0036] Among them, γ t (k) represents soft allocation, i.e. x t The probability generated by the kth Gaussian. The final Fisher vector f FV is from all K clusters and In order to obtain a comprehensive image representation, the deep features T are aggregated through two fully connected layers to obtain the image-level information f FC , and combine it with the part-level information f FV The combination is performed to obtain the complete image representation f, which is used to generate the hash code.

[0037] Step 3: Construct a self-consistent hashing learning module and introduce the reconstruction task into hashing learning based on the encoder-decoder structure.

[0038] The self-consistency principle states that autonomous intelligent systems should strive to achieve highly self-consistent models that minimize the internal differences between observed data and reproduced data from the external world. In other words, the intelligent system should be able to skillfully reconstruct the distribution of observed data from its compressed representation, to the point where, despite its best efforts, the internal differences become indistinguishable. Therefore, according to the self-consistency principle, the expectation is to learn more comprehensive features that cover not only holistic visual patterns but also valuable fine-grained patterns.

[0039] Self-consistent hashing learning module, we use the encoder matrix W to project f into the q-dimensional latent space to obtain the internal latent representation v. Following the self-consistency principle, we use the decoder to reconstruct f:

[0040]

[0041] where F = {f1; f2; ...; f N} represents the image embedding in a mini-batch, and V corresponds to the potential representation v in a mini-batch. Directly optimizing Equation (8) with hard constraints is difficult to perform. Therefore, we relax the constraints into soft constraints, and the learning objective can be rewritten as:

[0042]

[0043] Finally, we can generate the q-bit binary hash code u from the latent representation v as follows:

[0044] u=sgn(tanh(v))#(10).

Claims

1. An unsupervised fine-grained image hash coding method, comprising the following steps: Step 1: Using an asymmetric data augmentation method, we construct an asymmetric contrastive learning framework that is adaptable to fine-grained objects. Step 2: Fisher vectors are introduced to construct end-to-end part-level dense contrastive learning to capture and model part-level information of fine-grained objects. Step 3: Construct a self-consistent hashing learning module and introduce the reconstruction task into hashing learning based on the encoder-decoder structure; Step 2 uses the Gaussian mixture model λ={ω k , μ k ,σ k ; k = 1, ..., K} for the deep feature descriptor X = {x t }, t=(1, 2, ..., H×W) for modeling, where ω k , μ k and σ k are the mixture weights, mean vectors, and covariance matrices of the k-th Gaussian component respectively; H and W are the height and width of the activation tensor respectively; and the part-level prototype corresponding to each cluster is calculated, Among them, Ω k represents the index set of depth descriptors corresponding to the k-th cluster; Then, traditional contrastive learning was extended to part-level dense contrastive learning. Among them, p i，k Indicates I i The k-th component-level representation of Respectively represent p i，k Anchor, positive, and negative sample views; is the dense contrast loss of the positive sample view, and the temperature hyperparameter η is set to 0.3; Based on the modeling parameters, high-order statistical information is calculated and aggregated into Fisher vectors to obtain part-level information; Among them, γ t (k) represents soft allocation, i.e. x t The probability generated by the kth Gaussian; the final Fisher vector f FV is from all K clusters and To obtain a comprehensive image representation, the deep features T are aggregated through two fully connected layers to obtain the image-level information f FC , and combine it with the part-level information f FV The combination is performed to obtain the complete image representation f, which is used to generate the hash code.

2. The unsupervised fine-grained image hash coding method according to claim 1, characterized in that: Step 1: For each sample I, three enhanced views are created using an asymmetric enhancement method: an anchor view, a positive view, and a negative view. The anchor view is generated by applying only the Resize operation to sample I, the positive view is generated by applying only the RandomCrop operation to sample I, and the negative view undergoes two destructive enhancements, namely ColorJitter and ElasticTransform, to induce substantial changes related to fine-grained objects. Each view is passed through the model to generate a corresponding hash code, and the similarity between the hash codes u generated by the model is calculated: Among them, I i is a sample in a mini-batch, i = (1, 2, ..., N), and Sample I i Anchor view, positive sample view and negative sample view of Corresponding to The hash code of Corresponding to The hash code of Corresponding to The hash code of is the contrastive loss for positive views; the temperature hyperparameter η is set to 0.

3.

3. The unsupervised fine-grained image hash coding method according to claim 1, characterized in that: Step 3 introduces the reconstruction task into hash learning through the encoder-decoder structure, constructs a self-consistent hash learning module, uses the encoder matrix W to project f into the q-dimensional latent space to obtain the internal latent representation v; following the self-consistency principle, uses the decoder to reconstruct f: where F = {f1; f2; ...; f N } represents the image embedding in a mini-batch, and V corresponds to the latent representation v in a mini-batch; Rewrite the learning objective through soft constraints as follows: The expectation is that hashing learns more comprehensive features, covering both overall visual patterns and fine-grained patterns; Finally, a q-bit binary hash code u is generated based on the latent representation v: u=sgn(tanh(v))#(7).

4. A computer storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

5. An electronic device, characterized in that: The method comprises a memory and one or more processors, wherein the memory is used to store one or more programs; when the one or more programs are executed by the one or more processors, the method according to any one of claims 1 to 3 is implemented.

Images