A self-distillation guided ranking consistent self-supervised learning method suitable for image retrieval

Abstract

The application discloses a self-distillation guided ranking consistency self-supervised learning method suitable for image retrieval, which mainly comprises four modules of an image feature extractor, a self-distillation-based quantization module, ranking consistency learning and feature similarity, and comprises the following steps: acquiring an original feature vector of an image by using the feature extractor; generating a reconstructed soft quantization vector and a reconstructed hard quantization vector by using the self-distillation-based quantization module, the two vectors forming a teacher signal and a student signal with the original feature respectively, and constructing a self-distillation relationship; subsequently, using the ranking information formed by the soft quantization vector and the original feature to guide the ranking information corresponding to the hard quantization, so that the retrieval result of the hard quantization approximates the retrieval result of the soft quantization; meanwhile, introducing a feature similarity constraint to make the soft and hard quantizations different in the feature level, and assisting in optimizing the ranking consistency learning. The self-distillation guided ranking consistency method effectively reduces the gap between the retrieval performances of the soft and hard quantizations, and effectively improves the retrieval precision while ensuring the retrieval efficiency.

Description

Technical Field

[0001] This invention belongs to the field of image retrieval technology, and more specifically, relates to a self-supervised learning method for image retrieval that is self-distilled and guided to achieve consistent ranking. Technical Background In recent years, with the development of information technology, multimedia data on the Internet has exploded. Among them, image data has attracted widespread attention from all walks of life due to its rich content, simple acquisition methods, and processing and analysis methods. However, image data is usually high-dimensional and complex in structure. Faced with a vast and diverse image library, how to achieve efficient and accurate retrieval to meet user query needs has become an urgent problem to be solved. Directly searching in the original real-valued feature space for large-scale data would incur extremely high computational and storage costs. Among numerous search algorithms, approximate nearest neighbor search stands out due to its advantages of fast computation speed, low storage cost, and high retrieval accuracy.

[0002] There are two main approaches to approximate nearest neighbor search: hashing and vector quantization. The hashing method uses Hamming distance to calculate the distance between binary codes; however, this method has a limitation: it can only represent the distance with a few distinct values, making complex distance representations impossible. To alleviate this problem, vector quantization-based methods are proposed, utilizing quantized real-valued vectors for distance measurement. Among these, product quantization is one of the best methods.

[0003] The core idea of ​​product quantization is to decompose the feature space containing high-dimensional feature vectors into a Cartesian product of multiple subspaces, and then perform vector quantization independently on each subspace. Product quantization achieves asymmetric approximate distance calculation through a lookup table, significantly improving computational efficiency while reducing storage overhead. However, the hard quantization operation in traditional product quantization, due to its non-differentiability, hinders its end-to-end joint optimization with deep neural networks, thus limiting further improvements in retrieval performance.

[0004] With the development and large-scale application of deep learning, deep quantization coding methods have gradually attracted attention. Deep quantization solves the problem of non-differentiability in traditional product quantization during the training phase by employing a differentiable soft quantization strategy. During training, soft quantization generates a continuous probability distribution, enabling gradient backpropagation and jointly optimizing the codebook and network parameters. In the retrieval phase, a non-differentiable hard quantization strategy is used, combined with a pre-computed lookup table to achieve efficient approximate distance calculation. This method combines the efficient retrieval capabilities of product quantization with the semantic discrimination advantages of deep learning, providing a new solution for large-scale image retrieval.

[0005] Based on the above background, this invention designs a self-supervised learning method for image retrieval that uses self-distillation-guided ranking consistency. Specifically, firstly, features are extracted from a batch of images to obtain feature vectors for the corresponding original images. Then, the feature vectors are quantized using a self-distillation-based quantization module to obtain corresponding soft-quantized and hard-quantized vectors, which are then normalized. Based on this, a ranking consistency constraint is introduced, allowing the hard-quantized vector to improve retrieval accuracy while maintaining high retrieval efficiency. Furthermore, feature similarity constraints provide a more stable foundation for ranking consistency learning, thereby further improving retrieval accuracy. Summary of the Invention

[0006] In view of this, the purpose of this invention is to propose a self-supervised learning method for image retrieval that is guided by self-distillation to achieve ranking consistency. This method constructs a ranking consistency learning module to guide the hard-quantized retrieval results to approximate the soft-quantized retrieval results, thereby improving the retrieval accuracy under hard quantization while maintaining the efficient retrieval characteristics of hard quantization. At the same time, it introduces feature similarity constraints to align soft and hard-quantized features at the feature level, thereby further improving the retrieval accuracy.

[0007] To achieve the above objectives, the specific technical solutions implemented by the present invention include: The system consists of four modules: an image feature extractor, a self-distillation-based quantization module, a ranking consistency learning module, and a feature similarity module.

[0008] The image feature extractor includes extracting feature vectors from the images. All images in the batch need to undergo the following processing to obtain the image... For example, the specific steps include: Step A1: Adjust the image The size of the image is converted to 3×224×224; Step A2: Extract images using a convolutional neural network. eigenvectors ; The self-distillation-based quantization module includes quantizing the feature vectors of the original image. The feature vectors of all images in the batch need to undergo the following processing to obtain the feature vectors... For example, it includes a soft quantization process, a hard quantization process, and a self-distillation construction process, with specific steps including: (1) Softening process Step B1: Given feature vectors and a codebook containing M codebooks The feature vector is divided into M equal-dimensional sub-vectors. ; Step B2: Normalize all codebooks and subvectors; Step B3: For the m-th sub-vector, calculate its relationship with the corresponding codebook. The similarity of all codewords in the code is calculated, and the soft-assigned weights are obtained by using the softmax function. Step B4: Use soft-assignment weights to perform weighted summation on the codewords to obtain the corresponding sub-reconstruction soft quantization vector; Step B5: Concatenate all the sub-reconstructed soft-quantized vectors to obtain the feature vector. The corresponding reconstructed soft quantization vector ; (2) Hard quantization process: Step C1: Given the feature vector and a codebook containing M codebooks The feature vector is divided into M equal-dimensional sub-vectors. ; Step C2: Normalize all codebooks and subvectors; Step C3: For the m-th sub-vector, calculate its relationship with the corresponding codebook. The similarity of all codewords in the code is calculated, and the codeword with the highest similarity is selected to obtain the corresponding codeword index. Step C4: Obtain the sub-reconstructed hard quantization vector based on the selected codeword index; Step C5: Concatenate all the sub-reconstructed hard-quantized vectors to obtain the feature vector. The corresponding reconstructed hard quantization vector ; (3) Self-distillation construction Step D1: Given the feature vector of the original image Reconstructing the soft quantization vector Reconstructing hard-quantized vectors ; Step D2: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full , , Normalize each feature vector separately to obtain the normalized feature vectors. Soft quantization vector Hard quantization vector ; Step D3: Reconstruct the soft-quantized vectors of all images in the current batch after normalization. and the reconstructed hard-quantized vector after normalization Treating it as a small-scale image library, the normalized original feature vector of each image in the batch is used. As a query; Step D4: Calculate the feature vector of the original image With reconstructing soft quantization vectors The cosine similarity is used to obtain the soft-quantized similarity matrix. As a signal from the teacher; Step D5: Calculate the feature vector of the original image With reconstructing hard quantized vectors The cosine similarity is used to obtain the hard-quantized similarity matrix. As a signal for students; The ranking consistency learning module includes comparing the differences between soft and hard quantization vectors in the ranking results and using these differences to constrain model training. Specific steps include: Step E1: Given a soft-quantized similarity matrix and hard-quantized similarity matrix ; Step E2: For each query image, calculate its similarity in the soft-quantized similarity matrix. Based on the similarity value, the samples are sorted from largest to smallest similarity, and the indices of the top k most similar samples are selected to form an index set I. Step E3: Use index set I to perform soft quantization on the similarity matrix and hard-quantized similarity matrix The columns are reordered, meaning only the columns corresponding to index I are retained, to obtain the top-k soft-quantized similarity matrix. Similarity matrix with top-k hard quantization ; Step F4: Based on the top-k soft quantization similarity matrix Construct pairwise similarity difference matrices Before constructing the pairwise similarity difference matrix, remove the first and second similarity results corresponding to each query; Step F5: Based on the top-k hard-quantized similarity matrix Constructing pairwise similarity difference matrices Before constructing the pairwise similarity difference matrix, remove the first and second similarity results corresponding to each query; Step F6: Based on and Calculate the inconsistent pairwise similarity difference loss separately. Consistent pairwise similarity difference loss ; The feature similarity includes calculating the similarity between the reconstructed soft-quantized vector and the reconstructed hard-quantized vector, specifically including the following steps: Step G1: Given a batch of reconstructed soft-quantization vectors for images and reconstructing hard quantized vectors ; Step G2: Calculate the reconstructed soft quantization for this batch of images. With reconstructing hard quantized vectors Mean square error loss between ; Attached image description: Figure 1 This is a schematic diagram of the self-supervised learning method for self-distillation-guided sorting consistency of the present invention.

[0009] Figure 2 This is a schematic diagram of the soft quantization process in the self-distillation-based quantization module of the present invention.

[0010] Figure 3 This is a schematic diagram of the hard quantization process in the self-distillation-based quantization module of the present invention. Specific implementation methods: To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the technical solutions, accompanying drawings, and embodiments.

[0011] This invention proposes a self-supervised learning method for image retrieval that guides ranking consistency through self-distillation, comprising four modules: an image feature extractor, a self-distillation-based quantization module, a ranking consistency learning module, and a feature similarity module. The complete process is as follows: Figure 1 As shown: Within a batch, each image first obtains a feature vector through a convolutional neural network, and then generates corresponding reconstructed soft-quantized and reconstructed hard-quantized vectors through a self-distillation-based quantization module. The original feature vectors, reconstructed soft-quantized vectors, and reconstructed hard-quantized vectors are then normalized to obtain normalized original feature vectors, soft-quantized vectors, and hard-quantized vectors. Subsequently, using the original feature vectors as queries, the similarity between them and the soft-quantized and hard-quantized vectors within the same batch is calculated, generating corresponding similarity matrices. Based on these similarity matrices, difference matrices are calculated to construct a ranking consistency loss, making the retrieval results of hard quantization approximate those of soft quantization. Simultaneously, a feature similarity loss between soft and hard quantized vectors is introduced to align the two types of quantization representations at the feature level. The ranking consistency loss and feature similarity loss complement each other, allowing hard-quantized vectors to improve retrieval accuracy while maintaining computational efficiency.

[0012] More specifically, the following is combined with Figure 1 , 2 Section 3 provides a detailed description of the self-supervised learning method of the present invention, which is guided by distillation and achieves consistent sorting.

[0013] (1) Quantization module based on self-distillation The self-distillation-based quantization module includes a soft quantization process, a hard quantization process, and a self-distillation construction process, wherein: The specific structure of the soft quantization process is as follows: Figure 2 As shown, given a codebook containing M codebooks The m-th codebook It contains K codewords , with feature vector For example: Step H1: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full Divide into M equal-dimensional sub-vectors , where the m-th subvector And the subvector is quantized using the Mth codebook; Step H2: In the m-th subspace, for subvectors and All codewords in the code are normalized to... and For example, the specific calculation formula is as follows: , , where the function This indicates that a vector is being normalized. Step H3: Calculate the soft allocation weights , its origin The calculation is as follows: ,in, It is a non-negative parameter used to scale the input to softmax; Step H4: For each subvector The codewords are weighted and summed using soft-assignment weights to obtain the corresponding sub-reconstruction soft quantization vector. The specific calculation formula is as follows: ; Step H5: Reconstruct all sub-soft quantized vectors The concatenation yields the reconstructed soft-quantized vector. ; The specific process of hard quantization is as follows: Figure 3 As shown, given a codebook containing M codebooks The m-th codebook It contains K codewords , with feature vector For example: Step I1: Divide into M equal-dimensional sub-vectors , where the m-th subvector And the subvector is quantized using the Mth codebook; Step I2: In the m-th subspace, for subvectors and All codewords in the code are normalized to... and For example, the specific calculation formula is as follows: , , where the function This indicates that a vector is being normalized. Step I3: For the m-th sub-vector Calculate its relationship with the corresponding codebook The similarity of all codewords in the dataset is calculated, and the codeword with the highest similarity is selected to obtain its index. The specific calculation formula is as follows: ; Step I4: Based on the selected codeword index From the corresponding codebook The corresponding codeword is selected as the reconstructed hard quantization vector of the subvector. The specific calculation formula is as follows: ; Step I5: Reconstruct hard-quantized vectors from all sub-vectors The concatenation yields the reconstructed hard-quantized vector. ; The specific process of self-distillation construction is as follows: Figure 1 As shown, the specific process is as follows: Step J1: Given the feature vector of the original image, the reconstructed soft-quantization vector, and the reconstructed hard-quantization vector, taking the nth sample in the batch as an example, their corresponding vectors are represented as follows: , , ,in, Represents the original feature vector. This indicates the reconstruction of the soft-quantized vector. This indicates the reconstruction of the hard-quantized vector; Step J2: Normalize the feature vector of the original image, the reconstructed soft-quantized vector, and the reconstructed hard-quantized vector, respectively. The specific calculation formula is shown below: For the original feature vector: , For reconstructing the soft-quantized vector: , For reconstructing hard-quantized vectors: , where the function This represents the L2 normalization operation for vectors; Step J3: Reconstruct the soft-quantized vectors from all normalized images in a batch. and the reconstructed hard-quantized vector after normalization Treating it as a small-scale image library, the normalized original feature vector of each image in the batch is used. As a query vector; Step J4: Calculate the feature vector of the original image With reconstructing soft quantization vectors The cosine similarity is used to obtain the soft-quantized similarity matrix. The specific calculation formula is as follows: ; Step J5: Calculate the feature vector of the original image. With reconstructing hard quantized vectors The cosine similarity is used to obtain the hard-quantized similarity matrix. The specific calculation formula is as follows: ; (2) Order Consistency Learning Module The specific process of learning order consistency is as follows: Figure 1 As shown, the specific process is as follows: Step K1: Given a soft-quantized similarity matrix and hard-quantized similarity matrix ; Step K2: Because soft quantization has higher retrieval accuracy, for each query image... Based on its soft-quantized similarity matrix Based on the similarity value, samples are sorted from highest to lowest similarity, and the indices of the top k most similar samples are selected to form an index set I. The specific calculation formula is shown below: ,in, The function sorts the similarity values ​​from largest to smallest and returns the top k indices. Step K3: Use index set I to perform soft quantization on the similarity matrix and hard-quantized similarity matrix Reorder the columns, keeping only those corresponding to index I, to obtain the top-k soft-quantized similarity matrix. Similarity matrix with top-k hard quantization The specific calculation formula is as follows: , ,in, This indicates that the values ​​in the similarity matrix are sorted according to index I; Step K4: To enhance the robustness of the ranking supervision, before constructing the difference matrix, the first column of the top-k soft-quantized similarity matrix is ​​removed, and the result is based on the first top-k soft-quantized similarity matrix. Construct pairwise similarity difference matrices The specific calculation formula is as follows: ; Step K5: To enhance the robustness of the ranking supervision, before constructing the difference matrix, the first column of the top-k soft-quantized similarity matrix is ​​removed, and the result is based on the first top-k hard-quantized similarity matrix. Constructing pairwise similarity difference matrices The specific calculation formula is as follows: ; Step K6: Based on and Calculate the inconsistent pairwise similarity difference loss separately. Consistent pairwise similarity difference loss The specific calculation formula is as follows:

[0014] ,in, Let represent the Heaviside leap function, and m be the smoothing term. Finally, during the training phase, the total loss function is: ,in, For hyperparameters; (3) Feature Similarity Module The specific process of feature similarity is as follows: Figure 1 As shown, the specific process is as follows: Step L1: Given a batch of reconstructed soft-quantized vectors of images and reconstructing hard quantized vectors ,in, and These represent reconstructing the soft quantization vector and reconstructing the hard quantization vector, respectively. Step L2: Calculate the mean squared error loss for each pair of reconstructed soft-quantized vectors and reconstructed hard-quantized vectors in the batch. The specific calculation formula is as follows: Where n represents the batch size; The specific embodiments described above further illustrate the purpose and technical solutions of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A self-supervised learning method for image retrieval with self-distillation-guided ranking consistency, characterized in that, The method includes: an image feature extractor, a self-distillation-based quantization module, a ranking consistency learning module, and a feature similarity module; The image feature extractor includes: using a convolutional neural network to extract features from the input image to obtain the feature vector of the original image; The self-distillation-based quantization module includes: inputting the feature vector of the extracted original image into a quantization module composed of M codebooks, and obtaining the reconstructed soft quantization vector and reconstructed hard quantization vector of the corresponding image through soft quantization mechanism and hard quantization mechanism respectively. The teacher signal formed by the reconstructed soft quantization vector and the original feature vector is used to guide the learning of the student signal formed by the reconstructed hard quantization vector, thereby realizing the self-distillation guidance of soft quantization on hard quantization. The ranking consistency learning module includes: treating the soft and hard vectors of a batch of images as a small-scale image library; using the original feature vectors of each image in the batch as queries; calculating the similarity between the query and the soft and hard vectors in the image library; and using the ranking information formed by the soft vectors and the original feature vectors to provide self-supervised guidance for the ranking information formed by the hard vectors and the original feature vectors, thereby constraining the hard retrieval results to approximate the soft retrieval results. The feature similarity includes aligning the feature representations of soft-quantized vectors and hard-quantized vectors as an auxiliary constraint, complementing the ranking consistency learning module, and jointly ensuring that the hard-quantized representation learns as much as possible the rich semantics of the soft-quantized representation while retaining efficient retrieval characteristics, thereby significantly improving the final retrieval accuracy.

2. The self-distillation-based quantization module according to claim 1, characterized in that, The quantization module includes a soft quantization process, a hard quantization process, and a self-distillation construction process, wherein: The soft quantization process includes: First, set up a codebook containing M codebooks. Given the feature vector of the original image to be quantized It is divided into several equal-dimensional sub-vectors in the dimensional direction. ,in ; Then, after normalizing each subvector and the codewords in the codebook, the m-th subvector... Based on the m-th codebook using a soft quantization mechanism Quantization to reconstruct soft quantization vector ; Finally, the M sub-reconstructed soft-quantized vectors are concatenated to obtain the feature vector. Reconstructed soft-quantized vector ; The hard quantization process includes: First, set up a codebook containing M codebooks. Given the feature vector of the original image to be quantized It is divided into several equal-dimensional sub-vectors in the dimensional direction. ,in ; Then, after normalizing each subvector and the codewords in the codebook, the m-th subvector... Based on the m-th codebook using a hard quantization mechanism Quantization to reconstruct hard-quantized vectors ; Finally, the M sub-reconstructed hard-quantized vectors are concatenated to obtain the feature vector. Reconstructed hard quantized vector ; The self-distillation construction process: The original feature vectors of a batch of images Reconstructing the soft quantization vector Reconstructing hard-quantized vectors Normalization is performed to obtain the normalized feature vector. Soft quantization vector Hard quantization vector ; Subsequently, the normalized soft quantization vector With hard quantization vector Treating it as a small-scale image library, the normalized original feature vector of each image in the batch is used. As a query, calculate its relationship with the soft-quantized vectors within the small-scale image library. The cosine similarity between them forms a soft-quantized similarity matrix. Simultaneously, it calculates its correlation with the hardened vector within a small-scale image library. The cosine similarity between them forms a hard-quantized similarity matrix. ; With the aforementioned similarity matrix As a teacher signal, the similarity matrix As a student signal, it enables the self-distillation of soft quantization onto hard quantization.

3. The ranking consistency learning module according to claim 1, characterized in that, The ranking consistency learning process is as follows: First, in a training batch, a convolutional neural network is used to extract the original feature vectors of the images in that batch. The original feature vectors of this batch of images are then fed into a self-distillation-based quantization module for quantization, resulting in a soft-quantized similarity matrix. and hard-quantized similarity matrix ; Next, for each query image, its similarity in the soft-quantized similarity matrix is ​​calculated. Based on the sorting in the matrix, obtain the index list I of the top k most similar samples, and use I to hard-quantize the similarity matrix. Similarity matrix with soft quantization Reorder the results and construct the corresponding top-k search similarity matrix. and ; Then, the similarity matrix is ​​retrieved based on the top-k similarity. and Construct two pairs of similarity difference matrices. and ; Finally, based on these two difference matrices and Calculate the inconsistent pairwise similarity difference loss separately. and consistent pairwise similarity difference loss .

4. The feature similarity module according to claim 1, characterized in that, The feature similarity process is as follows: First, in a training batch, the original feature vectors of the images in this batch are extracted using a convolutional neural network. The original feature vectors of this batch of images are then fed into a self-distillation-based quantization module for quantization to obtain reconstructed soft-quantized vectors. and reconstructed hard quantized vector ; Finally, the soft quantization vector is reconstructed through computation. and reconstructing hard quantized vectors The mean squared error between them is used to obtain the feature similarity loss. .

Images