A Feature Extraction and Deep Quantization Learning Method for Label-Free Huangmei Opera Image Retrieval

Abstract

This invention relates to the problem of unlabeled Huangmei Opera image retrieval in the field of image retrieval, and discloses a Huangmei Opera image feature extraction and deep cumulative quantization learning method based on negative view contrastive learning. This method mainly consists of five modules: random data augmentation, image feature encoder, texture feature extraction, deep cumulative quantization, and negative view-based quantization contrastive learning. Specifically, it includes: generating two positive samples and one negative sample for the input image using two positive data augmentation techniques and a patch-based negative data augmentation technique; using a pre-trained Vision Transformer model and U-Net as image feature extractors, which map the input image into a high-dimensional real-valued feature vector; then, passing through a deep cumulative quantization layer containing several codebooks, obtaining the reconstructed vector corresponding to the image feature vector through a soft allocation mechanism; furthermore, obtaining the local texture features of the corresponding Huangmei Opera image through an adaptive high-pass filter generator; and finally, negative view-based quantization contrastive learning is used to minimize the similarity between relevant reconstructed vectors and maximize the similarity between irrelevant reconstructed vectors. This invention combines a contrastive learning framework to generate true negative samples of input Huangmei Opera images and fuses the global and local texture features of the images, effectively improving the retrieval accuracy of image retrieval methods based on depth-accumulated quantization.

Description

Technical Field

[0001] This invention belongs to the field of image retrieval technology, and more specifically, relates to a feature extraction and deep quantization learning method for unlabeled Huangmei Opera image retrieval. Technical Background

[0002] With the development of digital technology, a large amount of image data of classic Huangmei Opera plays, character designs, stage scenes, etc., has been preserved and disseminated. These images not only carry rich historical and cultural information, but also provide valuable reference materials for fields such as art research, education and training, and cultural and creative design. However, due to the complex content, diverse costumes, and rich scenes of Huangmei Opera images, efficiently and accurately retrieving relevant images from massive amounts of data has become an important task.

[0003] Nearest neighbor search (NNS) is a fundamental and essential method in many technological fields, such as image retrieval, pattern recognition, and computer vision. However, when processing high-dimensional data, such as image features, exact NNS requires significant computational and storage costs. To achieve a better balance between retrieval efficiency and accuracy, approximate NNS has been proposed.

[0004] Approximate nearest neighbor search methods fall into two main categories: hash-based and quantization-based. Hash-based methods, while preserving the original similarity of feature vectors, use a hash function to convert feature vectors into compact binary codes. For this type of method, Hamming distance is used to measure the distance between binary codes. However, a limitation of this approach is that the discriminative power of Hamming distance is limited by the length of the codes used, and can only represent the similarity between feature vectors with a finite number of distance values, failing to adequately measure the complex distance relationships required for accurate feature vector similarity. Quantization-based methods address this problem better. They use an asymmetric distance based on the Euclidean distance from the query feature vector to the codebook to measure the similarity between feature vectors, ensuring retrieval accuracy while mitigating the high overhead of calculating precise Euclidean distance. Among these methods, Accumulative Quantization (AQ) is one of the best quantization methods for feature vectors, obtaining the reconstructed vector corresponding to the feature vector by accumulating M sub-vectors quantized from M codebooks.

[0005] In recent years, quantization-based deep image retrieval methods have introduced differentiable quantization techniques onto continuous depth image feature vectors, enabling direct learning of depth representations in real-valued space. For supervised image retrieval methods, achieving satisfactory retrieval accuracy requires expensive image annotation. Therefore, quantization-based unsupervised image retrieval methods have been proposed to explore the similarities between images without the need for annotation, thereby generating discriminative binary codes. In the unsupervised domain, contrastive learning is often used to learn image feature vectors without annotations to achieve superior retrieval performance.

[0006] Based on the above background, this invention designs a feature extraction and deep quantization learning method for unlabeled Huangmei Opera image retrieval, which combines deep self-supervised contrastive learning with cumulative quantization. Specifically, firstly, negative samples of the input Huangmei Opera image are generated using image patch-based negative data augmentation. These negative samples are dissimilar to all Huangmei Opera images, thus enabling the learning of more discriminative image feature vectors. Then, a codebook and quantized image feature vectors are learned through deep cumulative quantization. Furthermore, an adaptive high-pass filter generator captures local texture features of the Huangmei Opera image, providing more refined image information for the contrastive learning-based quantization model, thereby further improving retrieval performance. Summary of the Invention

[0007] In view of this, the purpose of this invention is to propose a feature extraction and deep quantization learning method for unlabeled Huangmei Opera image retrieval. This method generates true negative samples of the input Huangmei Opera image through negative data augmentation based on image patches, thereby solving the problem of false negative samples caused by randomly selected training images in the contrastive learning framework. It also incorporates a fusion filter to extract texture information reflecting the local features of the Huangmei Opera image, enhancing the image's feature discrimination capability. Furthermore, it proposes a deep cumulative quantization method to obtain more accurate reconstruction vectors and improve retrieval accuracy.

[0008] To achieve the above objectives, the specific technical solutions implemented by the present invention include:

[0009] The system consists of five modules: random data augmentation, image feature encoder, local texture feature extraction, depth-accumulating quantization layer, and quantization contrast learning based on negative view.

[0010] The random data augmentation includes generating three different views of the Huangmei Opera image using different data augmentations. Specific steps include:

[0011] All Huangmei Opera images in the batch need to undergo the following processing steps to obtain Huangmei Opera image I n For example:

[0012] Step A1: Randomly select two positive data enhancement methods, such as random cropping, rotation, color jitter, Gaussian blur, etc., denoted as t1 and t2;

[0013] Step A2: Augment t1 and t2 using positive data to transform Huangmei Opera image I n Generate two forward views and in And uniformly represented as

[0014] Step A3: Using negative data augmentation t3 based on image patches, randomly shuffle the positions of the image patches to generate a true negative view of all Huangmei Opera images.

[0015] The image feature encoder includes extracting global image features and extracting multiple scale features of the image, taking Huangmei Opera image I as an example. n For example, the specific steps include:

[0016] (1) Extract global image features

[0017] Step B1: Adjust I n The image size is set to 224×224×3.

[0018] Step B2: Extract image I using a Vision Transformer model pre-trained on the ImageNet dataset. n global feature vector x n ;

[0019] (2) Extracting multiple scale features of the image

[0020] Step C1: Extract image I using the U-Net network n Multiple scale feature vectors

[0021] Step C2: Bottom-up fusion to obtain the corresponding initial fused multi-scale information feature vector {g}. n,1 ,…,g n,l ,…,g n,L};

[0022] The texture feature extraction includes extracting local texture features from Huangmei Opera images and feature fusion, using Huangmei Opera image I... n For example, the specific steps include:

[0023] (1) Extracting local texture features from Huangmei Opera images

[0024] Step D1: Given image I n The corresponding initial fusion feature vector {gn，1 , ..., g n，l , ..., g n，L The input is fed into a 3×3 convolutional layer to obtain the initial convolutional kernel.

[0025] Step D2: Place g n,1 The input is fed into a softmax layer, which transforms the initially generated convolutions into a probability distribution W. n,1 This ensures that the weights of the low-pass filter are non-negative and normalized.

[0026] Step D3: By adjusting the weights W of the low-pass filter n,1 The inversion operation is achieved by subtracting from the unit core E, thus forming a high-pass filter;

[0027] Step D4: Feature vector g n,1 The Huangmei Opera image I is obtained after processing with a high-pass filter. n Local texture feature vectors

[0028] (2) Feature fusion

[0029] Step E1: Convert the local texture feature vectors through a fully connected layer Projected into D-dimensional space, soon The dimension is converted to D-dimensional;

[0030] Step E2: Using vector concatenation or orthogonal fusion algorithms, combine the global feature vector x... n With local texture feature vectors By fusing, the corresponding fused feature vector is obtained.

[0031] The deep cumulative quantization layer includes quantizing the feature vector using a cumulative quantization method, with specific steps including:

[0032] Step F1: Given a feature vector, The codebook C = {C1,...,C} m ,...,C M}, and will Divide into M sub-vectors in sequence

[0033] Step F2: Normalize all codebooks and subvectors;

[0034] Step F3: Calculate the soft quantization probability of each vector and the corresponding codebook;

[0035] Step F4: Quantize each sub-vector using a soft allocation mechanism to obtain the corresponding sub-reconstructed vector;

[0036] Step F5: Accumulate the sub-reconstructed vectors to obtain The corresponding reconstructed vector z n ;

[0037] The quantization contrast learning based on negative views includes joint quantization contrast learning of the original image and the negative view, and joint quantization contrast learning of the positive view and the negative view, using Huangmei Opera image I as an example. n For example, the specific steps include:

[0038] (1) Quantization comparison learning of the original image and the negative view

[0039] Step G1: Given the input Huangmei Opera image I n fused feature vector and the global feature vectors of the three views generated by it. Then, by using a deep cumulative quantization layer, the corresponding reconstructed vector z is obtained. n ,

[0040] Step G2: Calculate z respectively n and and Similarity between them;

[0041] Step G3: Calculate z n Reconstruction vectors corresponding to the forward views generated from other images The similarity between them is reduced by using a perceptual bias framework. In and z n The negative impact of similar vectors on the model training process;

[0042] Step G4: Based on the similarity calculated above, obtain the quantized contrast loss L. oib ;

[0043] (2) Quantitative comparative learning of combined positive and negative views

[0044] Step H1: Given the input Huangmei Opera image I n Global feature vectors generated from different views The corresponding reconstructed vector is obtained by using a deep cumulative quantization layer.

[0045] Step H2: Calculate separately and and The similarity between them, where y = 1, Conversely, when y = 2,

[0046] Step H3: Calculation Reconstruction vectors corresponding to the forward views generated from other images The similarity between them is reduced by using a perceptual bias framework. Zhongyu The negative impact of similar vectors on the model training process;

[0047] Step H4: Based on the similarity calculated above, obtain the quantized contrast loss L. pvb ; Attached image description:

[0048] Figure 1 This is a schematic diagram of the feature extraction and deep quantization learning method for unlabeled Huangmei Opera image retrieval according to the present invention.

[0049] Figure 2 This is a schematic diagram of the adaptive high-pass filter generator of the present invention.

[0050] Figure 3 This is a schematic diagram of the deep accumulation quantization method of the present invention.

[0051] Figure 4 This is a schematic diagram of the quantitative comparison learning of the combined original image and negative view according to the present invention.

[0052] Figure 5 This is a schematic diagram illustrating the quantitative comparison and learning of the combined positive and negative views of the present invention. Specific implementation methods:

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the technical solutions, accompanying drawings, and embodiments.

[0054] This invention proposes a feature extraction and deep quantization learning method for unlabeled Huangmei Opera image retrieval, comprising five modules: random data augmentation, image feature encoder, texture feature extraction, deep cumulative quantization layer, and quantization contrast learning based on negative views. The complete process is as follows: Figure 1As shown: First, two random positive data augmentations and patch-based negative data augmentations are employed to generate two positive views and one negative view for the input Huangmei Opera image. Then, an image feature encoder, incorporating a Vision Transformer model and U-Net, is used to extract global feature vectors and multi-scale feature vectors from the image and its corresponding views. Next, the multi-scale feature vectors are fused bottom-up to obtain an initial fused feature vector, which is then input into an adaptive high-pass filter generator to acquire the local texture features corresponding to the Huangmei Opera image. This local texture feature is then fused with the corresponding global feature vector, resulting in a feature vector containing more image information. Next, the feature vector is passed through a deep cumulative quantization layer to obtain the corresponding reconstructed vector. Finally, through quantization contrastive learning based on the negative view, the similarity between relevant reconstructed vectors is minimized, while the similarity between unrelated reconstructed vectors is maximized.

[0055] More specifically, the following is combined with Figure 1 , 2 Sections 3, 4, and 5 provide a detailed description of the feature extraction and deep quantization learning method for unlabeled Huangmei Opera image retrieval in this invention.

[0056] (1) Adaptive high-pass filter generator

[0057] The specific structure of the adaptive high-pass filter generator is as follows: Figure 2 As shown, it consists of a 3×3 convolutional layer, a softmax layer, and a filter inversion operation. The initial fused feature vector... For example:

[0058] Step I1: Obtain the initial convolution kernel through a 3×3 convolutional layer. The calculation formula is expressed as follows:

[0059] Step I2: Place g n,1 The input is fed into a softmax layer, which transforms the initially generated convolutions into a probability distribution W. n,1 This ensures that the weights of the low-pass filter are non-negative and normalized, and the calculation formula is as follows:

[0060] Step I3: By adjusting the weights W of the low-pass filter n,1 The inversion operation is achieved by subtracting from the unit kernel E, forming a high-pass filter, where k = 3 and The calculation formula is:

[0061] Step I4: Feature vector g n,1 The Huangmei Opera image I is obtained after processing with a high-pass filter. n Local texture feature vectors The calculation formula is:

[0062] Step I5: Through a fully connected layer, Converted to a D-dimensional feature vector, which is related to image I n global feature vector x n The features are concatenated to perform feature fusion and obtain a fused feature vector.

[0063] (2) Deep accumulation quantization method

[0064] The specific process of the deep accumulation quantization method is as follows: Figure 3 As shown, given M codebooks {C1,...,C...} m ,...,C M The accumulated quantization head Q, where the m-th codebook C m It contains K codewords {c m,1 ,...,c m,k ,...,c m,K And the k-th codeword in the m-th codebook By fusing feature vectors For example:

[0065] Step J1: [The text appears to be incomplete and contains several grammatical errors. A more accurate translation would require the full context.] Divide into M sub-vectors in sequence Where the m-th sub-vector And d = D M And the subvector is quantized using the Mth codebook;

[0066] Step J2: In the m-th d-dimensional subspace, for All subvectors and C m All codewords in the code are normalized to... and c m,k For example, the specific calculation formula is as follows:

[0067]

[0068] The function norm(.) represents the vector normalization operation;

[0069] Step J3: Calculate the soft quantization probability P m ={p m,1 ,...,p m,k ,...,p m,K}, which is calculated by α-softmax, and the specific calculation process is as follows:

[0070]

[0071] Here, α is a non-negative parameter used to scale the input to softmax. α-softmax, as a differentiable alternative to argmax, relaxes the discrete optimization of hard-coded assignments into a continuously differentiable form;

[0072] Step J4: Each subvector It is quantized into sub-vectors through a soft allocation mechanism. The soft allocation mechanism can be formalized as the function sa(.), and the specific calculation formula is shown below:

[0073]

[0074] Step J5: Accumulate all sub-vectors z n,m Thus obtain Reconstructed vector

[0075] (3) Schematic diagram of quantitative comparison learning between the original image and the negative view

[0076] Sample pairs constructed in the quantized contrast learning of the joint original image and the negative view are as follows: Figure 4 As shown, the vector z is reconstructed. n For example, the specific process is as follows:

[0077] Step K1: Calculate z respectively n and and The similarity between them is expressed as and Where s(.) calculates the cosine similarity, and N... B It refers to the batch size;

[0078] Step K2: Calculate z n Reconstruction vectors corresponding to the forward views generated from other images The similarity between them is reduced by using a perceptual bias framework. In and z n The negative impact of similar vectors on the model training process is calculated using the following formula:

[0079]

[0080] Where τ2 is a nonnegative parameter, ρ + This is a positive prior for bias correction;

[0081] Step K3: Based on the similarity pairs calculated above, obtain the quantization contrast loss L. oib The calculation formula is as follows:

[0082]

[0083] (4) Schematic diagram of quantitative comparison learning of combined positive and negative views

[0084] Sample pairs constructed in the quantization contrast learning of joint positive and negative views, such as... Figure 5 As shown, to reconstruct the vector For example, the specific process is as follows:

[0085] Step L1: Calculate separately and and The similarity between them is expressed as and When y = 1, Conversely, when y = 2,

[0086] Step L2: Calculation Reconstruction vectors corresponding to the forward views generated from other images The similarity between them is reduced by using a perceptual bias framework. Zhongyu The negative impact of similar vectors on the model training process is calculated using the following formula:

[0087]

[0088] Step L3: Based on the similarity pairs calculated above, obtain the quantization contrast loss L. pvb The calculation formula is as follows:

[0089]

[0090] The specific embodiments described above further illustrate the purpose and technical solutions of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A feature extraction and deep quantization learning method for label-free huangmei opera image retrieval, characterized in that, The method comprises the following steps: Random data augmentation, image feature encoder, texture feature extraction, deep cumulative quantization and negative view-based quantization contrast learning; The random data augmentation comprises: generating two positive samples for the unlabeled Huangmei Opera image by using two random forward data augmentations; in addition, a negative data augmentation based on image blocks is adopted, the input unlabeled Huangmei Opera image is divided into MxN non-overlapping image blocks and the image is randomly shuffled and reorganized to obtain the negative sample of the unlabeled Huangmei Opera image; The image feature encoder comprises: a Vision Transformer model is used as a global feature extractor of the unlabeled Huangmei Opera image, the model maps the image into a high-dimensional real-valued image feature vector through a hierarchical image block embedding mechanism, thereby obtaining the global feature vector of the image; at the same time, a U-Net is used to extract multiple scale feature vectors of the input unlabeled Huangmei Opera image; The texture feature extraction comprises: the multi-scale feature vectors extracted by the U-Net network are fused from bottom to top to obtain an initial fused feature vector, which is then input into an adaptive high-pass filter generator to obtain the local texture features in the unlabeled Huangmei Opera image; The deep cumulative quantization comprises: the extracted high-dimensional real-valued image feature vector is input into a deep cumulative quantization module composed of L codebooks, and after being quantized respectively, the quantized output vectors are accumulated to obtain the corresponding reconstruction vectors; The negative view-based quantization contrast learning comprises two kinds of negative view-based quantization contrast learning: joint original image and negative view quantization contrast learning and joint positive view and negative view quantization contrast learning; by constructing the real negative sample of the image, the negative influence of the false negative sample in the training process of the quantization model based on contrast learning is reduced; in addition, a perception debiasing framework is adopted to further reduce the adverse effects of the false negative sample problem caused by randomly selecting unlabeled training images. 2.The feature extraction and deep quantization learning method for label-free Huangmei Opera image retrieval according to claim 1, characterized in that, The texture feature extraction process is: First, given a training set containing N unlabeled Huangmei Opera images Through U-Net, extract the multiple scale feature vectors of the nth unlabeled Huangmei Opera image I n Fusion layer by layer from bottom to top to obtain the corresponding multiple scale feature vectors g n = {g n,1 …,g n,l ,…,g n,L}, wherein denotes up-sampling, and ​ Then, the feature vector g fused with multiple scale information is input to a 3x3 convolution layer n,1 , and a feature vector is output 2 where k is the number of kernel channels and high-pass filters, and k is the kernel size of the filter Second, the eigenvector g n,1 After a softmax layer, the initial generated convolution is converted into a probability distribution, so as to ensure that the weight of the filter is non-negative and normalized, and then the kernel weight of the low-pass filter is converted into a high-pass filter through a filter inversion operation. Finally, the feature vector g n,1 After the above high-pass filter processing, the unlabeled Huangmei Opera image I n Local texture features of the image 3.The feature extraction and deep quantization learning method for label-free Huangmei Opera image retrieval according to claim 1, characterized in that, The deep cumulative quantization process is: First, given the feature vector to be quantized It is equally divided into several sub-vectors in the dimension direction And the cumulative quantization head Q containing M codebooks {C1,..., CM} m ,..., CM} M , where And d = D / M; Then, each sub-vector and the code word in the code book are normalized, and the mth sub-vector By means of the soft quantization mechanism, the mth code book C m Quantized into a sub-reconstruction vector Finally, the M sub-reconstruction vectors are accumulated to obtain the eigenvector of the reconstruction vector 4.The feature extraction and deep quantization learning method for label-free Huangmei Opera image retrieval according to claim 1, characterized in that, The joint original image and negative view quantization contrast learning process is: First, in a batch, N are randomly selected from the training set I. B The training process used unlabeled Huangmei Opera images generated 2N data points through two types of random positive data augmentation and patch-based negative data augmentation. B A positive view and N B A negative view; Then, the pre-trained Vision Transformer model is used to extract the unlabeled Huangmei Opera image I n The global feature vector representation of different views is denoted as x n , Where x n First, the local texture feature vector Fusion to obtain the corresponding fusion feature vector Subsequently, through deep accumulation quantization, the corresponding reconstruction vector z n , Second, in the contrastive learning framework, for the reconstruction vector z n that corresponds to the forward view of the image is similar to the reconstruction vectors z corresponding to all the other images in the batch are dissimilar; At the same time, the perception bias framework is adopted to reduce The negative impact of similar vectors during model training process; n similar vectors during model training process; Finally, the cosine similarity between z n and its similar and dissimilar vectors are calculated as and Thus, the quantized contrastive loss L oib is obtained by combining the original image and the negative view. 5.The feature extraction and deep quantization learning method for label-free Huangmei Opera image retrieval according to claim 1, characterized in that, The joint positive view and negative view quantization contrast learning process is: First, 2N positive views and N negative views are generated for training images I n respectively, by two kinds of random positive data augmentation and negative data augmentation based on image patches. B B ​​ Then, the unlabeled Huangmei Opera image I n The global feature vectors of the three different views generated are denoted as And through the deep accumulation quantization, the corresponding reconstruction vectors are obtained Secondly, in the contrast learning framework, for the reconstruction vector It is similar to It is similar to all And the reconstruction vectors corresponding to the forward views generated by other images It is not similar, where y=1, On the contrary, y=2, At the same time, employing a perceptual bias framework to reduce the negative impact of similar vectors during model training process; Finally, by calculating and the cosine similarity between the similar vectors and the dissimilar vectors, respectively and Thus, the quantized contrastive loss L of the joint positive view and negative view is obtained pvb .

Images