General image retrieval method based on image quantization

By using first-order and second-order codebooks in image quantization models, we generate cross-domain consistent and cross-category-aware quantitative results, which solves the problem of high resource consumption in unknown domains and category image retrieval, and achieves more efficient and Accurate image retrieval.

CN118939828AActive Publication Date: 2024-11-12UESTC (SHENZHEN) ADVANCED RES INST

Patent Information

Application Number
CN202411142461.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-20
Publication Date
2024-11-12
Estimated Expiration
2044-08-20

AI Technical Summary

Technical Problem

Traditional image quantization methods require continuous annotation of new data and retraining the model when processing images in unknown fields or categories, resulting in high resource consumption and poor image retrieval performance in new fields and categories.

Method used

Using first-order codebooks and second-order codebooks, the visual feature extraction module, first-order quantization cross-domain module and second-order quantization cross-category module generates cross-domain and cross-category perception quantization results, and builds an image quantization model to improve image retrieval accuracy and efficiency.

Benefits of technology

It significantly improves the accuracy and efficiency of image retrieval, improves the generalization ability of unknown categories, reduces quantization errors and domain differences, and improves retrieval performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939828A_ABST
    Figure CN118939828A_ABST
Patent Text Reader

Abstract

The invention discloses a general image retrieval method based on image quantization, which comprises the following steps: firstly, constructing a training sample set, and then constructing an image quantization model comprising a visual feature extraction module, a first-order quantization cross-domain module, a combined feature extraction module and a second-order quantization cross-category module; in the training process, firstly, a first-order codebook is randomly initialized, cross-domain consistency features are learned through a visual feature extraction module and the first-order codebook by utilizing a contrast learning mechanism, a second-order codebook is obtained based on updating of the first-order codebook, features of new category perception are obtained through a combined feature extraction module and the second-order codebook, and the features of the new category perception are obtained. A second-order quantization cross-category module obtains quantization combination features; and during retrieval, inputting each image in the search image library and the to-be-detected image into the image quantization model to obtain a combined feature of each image, and obtaining a retrieval result based on the combined feature. According to the method, cross-domain and cross-category challenges in an image retrieval task can be effectively solved, and the accuracy and efficiency of image retrieval are remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image retrieval, and more specifically, relates to a universal image retrieval method based on image quantization. Background Art

[0002] Image quantization is a method to reduce the accuracy of image data. By counting and mapping the pixel values ​​of the image, multi-bit pixel values ​​are compressed into fewer bits, thereby reducing the image file size and image quality. At present, existing image quantization methods have been applied in many fields, especially computer vision tasks such as image recognition, detection and retrieval, and have achieved great success. However, in image retrieval, especially in real-world applications in new fields and new categories, the performance of the method is not ideal. This is because traditional image quantization methods need to continuously annotate new data and retrain models when processing images in unknown fields or categories, resulting in high resource consumption, and further improvement is needed. Summary of the invention

[0003] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a general image retrieval method based on image quantization, which adopts first-order codebook and second-order codebook to generate cross-domain consistent and cross-category perceived quantization results, thereby solving the cross-domain and cross-category challenges in image retrieval tasks and significantly improving the accuracy and efficiency of image retrieval.

[0004] In order to achieve the above-mentioned object of the invention, the general image retrieval method based on image quantization of the present invention comprises the following steps:

[0005] S1: Construct a training sample set X according to actual needs, including images of N types of objects and their corresponding category texts;

[0006] S2: Construct an image quantization model, including a visual feature extraction module, a first-order quantization cross-domain module, a combined feature extraction module, and a second-order quantization cross-category module, where:

[0007] The visual feature extraction module is used to extract the visual embedding features of the input image x. L represents the preset feature dimension, and the visual embedding feature Z is sent to the first-order quantization cross-domain module and the combined feature extraction module;

[0008] The first-order quantization cross-domain module is used to calculate the visual embedding feature Z and the first-order codebook Each first-order code word in The first-order code word with the largest similarity is used as the quantitative visual embedding feature

[0009] The combined feature extraction module is used to extract the combined feature representation ζ of the input image x and send it to the second-order quantization module; the combined feature extraction module includes an attention module, a feature fusion module, a linear layer and a feedforward neural network, wherein:

[0010] The attention module is used to use the visual embedding feature Z as the query in the attention mechanism and the second-order codebook subset As the key and value in the attention mechanism, K represents the number of item types contained in the second-order codebook subset. The attention mechanism is used to extract the image to obtain the attention vector v of the input image and send it to the feature fusion module;

[0011] The feature fusion module is used to superimpose the attention vector v onto the visual embedding feature Z to obtain the feature z=Z+v and send it to the linear layer;

[0012] The linear layer is used to perform a linear transformation on the feature z to obtain the feature LN(z) and send it to the feedforward neural network;

[0013] The feedforward neural network is used to process the feature LN(z) to obtain the combined feature representation ζ of the input image x;

[0014] The second-order quantization cross-category module is used to calculate the visual embedding feature Z and the second-order codebook Each second-order code word E i The similarity of i = 1, 2, ..., N, and the code word E is determined according to the similarity. i The weight w i , the greater the similarity, the higher the weight, and then combine to get the quantitative combination feature

[0015] S3: Randomly initialize N code words D with dimension L i ′, for N code words D i ′ is normalized to obtain the code word D i , constituting the initialization of the first-order codebook

[0016] S4: Uniformly sample from different categories of the training sample set X to obtain the training sample subset X of the current batch B , each training sample is denoted as x i,j , i = 1, 2, ..., N, j = 1, 2, ..., M, M represents the number of samples in each category;

[0017] S5: Subset the training samples to X B For each training sample x i,j Input image quantization model, the visual feature extraction module and the first-order quantization module extract the visual embedding feature Z i,j And the quantized visual embedding features are

[0018] S6: Quantized visual embedding features using N×M training samples The second-order codebook Ω is calculated based on the first-order codebook Ω (2) , the second-order codebook Ω (2) Each second-order code word E i The calculation formula is:

[0019]

[0020] S7: For the training sample subset X B For each training sample x i,j , from the second-order codebook Ω (2) Filter the second-order codebook subset that does not contain its real item type Embed the visual feature Z i,j and the second-order codebook subset The subsequent combined feature extraction module in the input image quantization model obtains the combined feature ζ i,j , and then input into the second-order quantization module based on the second-order code book Ω (2) Get quantitative combination features

[0021] S8: Calculate the loss function and then update the parameters of the image quantization model and the first-order codebook;

[0022] S9: Determine whether the training end condition is met, if so, the training ends, otherwise return to step S4;

[0023] S10: The entire second-order codebook Ω (2) As a second-order codebook subset Then each image in the image library is input into the image quantization model, based on the trained first-order codebook Ω and second-order codebook Ω (2) and the second-order codebook subset Get the combined features of each image;

[0024] S11: The entire second-order codebook Ω (2) As a second-order codebook subset Then the image to be retrieved is input into the image quantization model, based on the trained first-order codebook Ω and second-order codebook Ω (2) and the second-order codebook subset Get its combined features, then calculate the similarity between the combined features and the combined features of each image in the image library, select the R images with the largest similarity as the retrieval results and return them. The value of R is set according to actual needs.

[0025] The invention discloses a universal image retrieval method based on image quantization. First, a training sample set is constructed, and then an image quantization model including a visual feature extraction module, a first-order quantization cross-domain module, a combined feature extraction module and a second-order quantization cross-category module is constructed. During the training process, a first-order code book is randomly initialized, and then the visual feature extraction module and the first-order code book are used to learn cross-domain consistency features by using a contrast learning mechanism. A second-order code book is updated on the basis of the first-order code book, and features perceived by a new category are obtained by combining the feature extraction module and the second-order code book, and then a quantized combined feature is obtained by the second-order quantization cross-category module. During retrieval, each image in a search image library is input into the image quantization model to obtain the combined features of each image, and the image to be retrieved is input into the image quantization model to obtain its combined features, and then the similarity between the combined features and the combined features of each image in the image library is calculated, thereby obtaining a retrieval result.

[0026] The present invention has the following beneficial effects:

[0027] 1) The present invention designs a second-order codebook containing representative information of known categories, and extracts new images by combining and projecting all second-order codewords, thereby improving the generalization ability for unknown categories;

[0028] 2) The present invention proposes a cross-alignment contrastive learning objective in model training to simultaneously reduce quantization errors and inter-domain differences, encouraging the model to generate domain-invariant, new category-aware quantization code words, thereby improving retrieval performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a flowchart of a specific implementation of the general image retrieval method based on image quantization of the present invention;

[0030] Figure 2 is a structural diagram of the image retrieval model in the present invention;

[0031] Figure 3 is an example diagram of the image quantization process in the present invention;

[0032] Figure 4 Schematic diagram of multiple code books in this embodiment. DETAILED DESCRIPTION

[0033] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0034] Example

[0035] Figure 1It is a flow chart of a specific implementation of the general image retrieval method based on image quantization of the present invention.

[0036] like Figure 1 As shown, the specific steps of the general image retrieval method based on image quantization of the present invention include:

[0037] S101: Obtain training sample set:

[0038] Construct a training sample set X according to actual needs, including images of N types of objects and their corresponding category labels.

[0039] S102: Build image quantization model:

[0040] In order to realize image retrieval, the present invention constructs an image quantization model. Figure 2 It is a structural diagram of the image retrieval model in the present invention. Figure 2 As shown, the image quantization model in the present invention includes a visual feature extraction module, a first-order quantization cross-domain module, a combined feature extraction module and a second-order quantization cross-category module, and each module is described in detail below.

[0041] The visual feature extraction module is used to extract the visual embedding features of the input image x. L represents the preset feature dimension, and the visual embedding feature Z is sent to the first-order quantization cross-domain module and the combined feature extraction module. In this embodiment, the visual feature extraction module includes a visual Transformer neural network, a fully connected layer and a normalization module, where:

[0042] The visual Transformer neural network is used to extract the original features of the image and output them to the fully connected layer;

[0043] The fully connected layer is used to process the original features to obtain the initial visual embedding features. And output to the normalization module;

[0044] The normalization module is used to normalize the initial visual embedding feature Z′ to obtain the visual embedding feature The specific method of normalization processing can be set according to actual needs. In this embodiment, max-min normalization is adopted.

[0045] The first-order quantization cross-domain module is used to calculate the visual embedding feature Z and the first-order codebook Each first-order code word in , i = 1, 2, ..., N, and the first-order code word with the largest similarity is used as the quantitative visual embedding feature

[0046] The combined feature extraction module is used to extract the combined feature representation ζ of the input image x and send it to the second-order quantization module. The combined feature extraction module includes an attention module, a feature fusion module, a linear layer and a feedforward neural network, where:

[0047] The attention module is used to use the visual embedding feature Z as the query in the attention mechanism and to convert the second-order codebook subset As the key and value in the attention mechanism, K represents the number of item types contained in the second-order codebook subset. The attention mechanism is used to extract the image to obtain the attention vector v of the input image and send it to the feature fusion module. The attention mechanism is a commonly used image processing mechanism, and its specific process is not repeated here. The attention mechanism in the present invention can be expressed as:

[0048]

[0049] Among them, σ() represents the activation function, F Q (), F K (), F V () represent the linear projection functions of query, key and value respectively.

[0050] The feature fusion module is used to superimpose the attention vector v onto the visual embedding feature Z to obtain the feature z=Z+v and send it to the linear layer.

[0051] The linear layer is used to perform a linear transformation on the feature z to obtain the feature LN(z) and send it to the feedforward neural network.

[0052] The feedforward neural network is used to process the feature LN(z) to obtain the combined feature representation ζ of the input image x.

[0053] It can be seen that the combined feature representation ζ can be expressed as:

[0054] ζ=FFN(LN(Z+v))

[0055] Among them, LN() represents a linear layer and FFN() represents a feed-forward neural network.

[0056] The second-order quantization cross-category module is used to calculate the visual embedding feature Z and the second-order codebook Each second-order code word E i The similarity of i = 1, 2, ..., N, and the code word E is determined according to the similarity. i The weight w i , the greater the similarity, the higher the weight, and then combine to get the quantitative combination feature

[0057] According to the above process, the image quantization model in the present invention takes the input image as an "unknown class" sample, and then performs attention calculation on the visual embedding features and the second-order code book containing known classes to obtain the quantitative combined features of the "unknown class" samples on the known classes, thereby realizing image quantization. Figure 2 It is an example diagram of the image quantization process in the present invention. Figure 2 The projection process from known class samples to the first-order codebook and then to the second-order codebook is shown, and the weighted combination of unseen class samples through the second-order codebook generates quantized combined features. It can be seen that the present invention improves the recognition ability of unknown class samples by weighted combination of known class codewords.

[0058] S103: Initialize code book:

[0059] Randomly initialize N code words D with dimension L i ′, for N code words D i ′ is normalized to obtain the code word D i , constituting the initialization of the first-order codebook

[0060] S104: Select the current batch of training samples:

[0061] Uniformly sample from different categories of the training sample set X to obtain the current batch of training sample subsets X B , each training sample is denoted as x i,j , i = 1, 2, ..., N, j = 1, 2, ..., M, M represents the number of samples in each category. It can be seen that the total number of images in a single batch is N × M.

[0062] S105: Extracting quantized visual embedding features:

[0063] The training sample subset X B For each training sample x i,j Input image quantization model, the visual feature extraction module and the first-order quantization module extract the visual embedding feature Z i,j And the quantized visual embedding features are

[0064] S106: Calculate the second-order codebook:

[0065] Quantized visual embedding features using N×M training samples The second-order codebook Ω is calculated based on the first-order codebook Ω (2) , the second-order codebook Ω (2) Each second-order code word E i The calculation formula is:

[0066]

[0067] S107: Extracting quantitative combination features:

[0068] For the training sample subset X B For each training sample x i,j , from the second-order codebook Ω (2) Filter the second-order codebook subset that does not contain its real item type Embed the visual feature Z i,j and the second-order codebook subset The subsequent combined feature extraction module in the input image quantization model obtains the combined feature ζ i,j , and then input into the second-order quantization module based on the second-order code book Ω (2) Get quantitative combination features

[0069] S108: Update image quantization model parameters:

[0070] The loss function is calculated and then the parameters of the image quantization model and the first-order codebook are updated.

[0071] In this embodiment, in order to improve the training effect of the image quantization model, the loss function is calculated by comprehensively considering the embedded features and the combined features. The specific method is as follows:

[0072] The cross-domain loss L of the embedded feature is calculated using the following formula: Z :

[0073]

[0074] Among them, o() represents contrastive learning loss, <> represents similarity, e represents a natural constant, and τ represents a temperature coefficient.

[0075] The cross-category loss L of the combined features is calculated using the following formula: ζ :

[0076]

[0077] Then the loss function L is calculated using the following formula:

[0078] L=L Z +λL ζ

[0079] Among them, λ represents the weight.

[0080] S109: Determine whether the training end condition is met, if so, the training ends and goes to step S110, otherwise returns to step S104. The training end condition can be set according to actual needs, usually by setting it to a preset maximum number of training times or model parameter convergence.

[0081] After multiple rounds of training, the image quantization model becomes domain invariant and class-aware, thus enabling image retrieval.

[0082] S110: Extract the combined features of the image library:

[0083] The entire second-order codebook Ω (2) As a second-order codebook subset Then each image in the image library is input into the image quantization model, based on the trained first-order codebook Ω and second-order codebook Ω (2) and the second-order codebook subset Get the combined features of each image.

[0084] S111: Image retrieval:

[0085] The entire second-order codebook Ω (2) As a second-order codebook subset Then the image to be retrieved is input into the image quantization model, based on the trained first-order codebook Ω and second-order codebook Ω (2) and the second-order codebook subset Get its combined features, then calculate the similarity between the combined features and the combined features of each image in the image library, select the R images with the largest similarity as the retrieval results and return them. The value of R is set according to actual needs.

[0086] In practical applications, when the image library contains too many image categories, the second-order codebook may be too large, reducing the retrieval efficiency. In this case, multiple codebooks can be used. Figure 3 Schematic diagram of multiple codebooks in this embodiment. Figure 3 As shown in the figure, multiple codebooks refer to cutting the codebook, that is, cutting the second-order codebook with the original dimension of L into Q blocks. During the operation of the image quantization model, the visual embedding feature Z extracted by the visual feature extraction module is also cut into Q blocks, which correspond to the cut codebook one by one. Each sub-block of the visual embedding feature Z is independently quantized and learned in its corresponding sub-codebook, and finally the final quantized combined feature is obtained by concatenating the quantized combined features of multiple sub-blocks.

[0087] In order to better illustrate the technical effect of the present invention, a specific example is used to experimentally verify the present invention.

[0088] Example 1

[0089] The experimental conditions in this example are as follows: System: Ubuntu 20.04, Software: Python 3.8, Processor: CPU E5-2678v3@2.50GHz×2, memory: 256GB.

[0090] In order to more accurately evaluate the effectiveness of the present invention in image retrieval tasks, this embodiment conducts image retrieval experiments on the DomainNet dataset. Using SE-ResNet50 and ViT-Base as the basic models, the present invention is compared with several currently most advanced methods. The comparison methods include:

[0091] DPgQ, for details, see the paper "Gao L, Zhu X, Song J, et al. Beyond productquantization: Deep progressive quantization for image retrieval[J].arXivpreprint arXiv:1906.06698,2019.";

[0092] SnMpQ, for details, see the paper "Paul S, Dutta T, Biswas S. Universal cross-domain retrieval: Generalizing across classes and domains [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. 2021: 12056-12064.";

[0093] CLIP, for details, see the reference “Learning transferable visual models from natural language supervision,”

[0094] Table 1 is a comparison table of image retrieval performance in multiple fields and bit lengths between the present invention and the comparative method in Example 1.

[0095]

[0096]

[0097] Table 1

[0098] As shown in Table 1, the present invention achieves the best average precision (mAP) on all bit lengths, reaching up to 40.1%. It is worth noting that the present invention performs particularly well in retrieval tasks of unknown categories and unknown domains, with the mAP improved by up to 8.0% in retrieval tasks of unknown categories.

[0099] In addition, to further verify the performance of the present invention in complex scenarios containing distractors, experimental results show that when the search set contains distractors of known categories, the present invention can still significantly improve the retrieval accuracy, with the mAP increased by up to 6.9%. In contrast, the current most advanced methods perform poorly when dealing with scenarios containing distractors, proving the robustness and accuracy of the present invention in the face of complex retrieval tasks.

[0100] Example 2

[0101] In order to more accurately evaluate the effectiveness of the present invention in a continuous learning environment, this embodiment performs sequential task learning of image classification on the SketchyExtended dataset. Using SE-ResNet50 and ViT-Base as the original models, the present invention is compared with several existing methods. In addition to DPgQ, SnMpQ and CLIP, the comparison methods also include:

[0102] CSQ, for details, see the literature "Yuan L, Wang T, Zhang X, et al. Central similarity quantization for efficient image and video retrieval [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 3083-3092.";

[0103] GrdHash, for details, see the literature "Su S, Zhang C, Han K, et al. Greedy hash: Towards fast optimization for accurate hash coding in cnn [J]. Advances in neural information processing systems, 2018, 31.";

[0104] DSH, for details, see the literature "Liu L, Shen F, Shen Y, et al. Deep sketch hashing: Fast free-hand sketch-based image retrieval [C] / / Proceedings of the IEEE conference on computer vision and pattern recognition. 2017: 2862-2871.";

[0105] PSKD, for details, please refer to the document "Wang K, Wang Y, Xu

[0106] TCN, for details, see the reference "Wang H, Deng C, Liu T, et al. Transferable coupled network for zero-shot sketch-based image retrieval[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(12): 9181-9194.";

[0107] Table 2 is a comparison table of image classification performance of the present invention and the comparative method in Example 2.

[0108] method 16 bits 32bits 64bits original CSQ 41.3 45.9 48.0 - GrdHash 42.3 47.2 48.9 - DSH 40.7 46.1 47.7 - QUR 43.2 48.4 51.0 53.7 QUR 40.4 45.1 50.2 53.1 PSKD 43.3 49.1 51.6 54.4 TCN 43.3 49.0 51.6 54.2 CLIP 50.2 52.8 54.5 56.9 The present invention (Hash) 46.9 48.8 52.5 - The present invention 49.8 52.1 53.4 53.9 The present invention (ViT) 58.2 59.0 59.3 60.1

[0109] Table 2

[0110] As shown in Table 2, the present invention achieves the best mean average precision (mAP) on all bit lengths, reaching up to 59.3%. It is worth noting that the present invention performs particularly well in retrieval tasks of unknown categories, with a maximum mAP improvement of 8.0%, demonstrating its strong generalization ability. In addition, the present invention can still significantly improve retrieval accuracy when dealing with complex scenes containing interference items, demonstrating its robustness and accuracy in complex retrieval tasks.

[0111] By comparing the performance of different methods at various bit lengths, we can see the overall advantages of the present invention in different retrieval tasks. In particular, compared with the current most advanced methods, the present invention performs particularly well when dealing with unknown categories and fields.

[0112] Through the above data analysis and comparison, it can be known that the present invention is superior to other quantization-based image retrieval methods. These results verify the effectiveness and superiority of the present invention.

[0113] Although the above describes the illustrative specific embodiments of the present invention to facilitate the understanding of the present invention by those skilled in the art, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

Claims

1. A general image retrieval method based on image quantization, characterized in that: The following steps are involved: S1: Construct a training sample set X according to actual needs, including images of N types of objects and their corresponding category texts; S2: Construct an image quantization model, including a visual feature extraction module, a first-order quantization cross-domain module, a combined feature extraction module, and a second-order quantization cross-category module, where: The visual feature extraction module is used to extract the visual embedding features of the input image x. L represents the preset feature dimension, and the visual embedding feature Z is sent to the first-order quantization cross-domain module and the combined feature extraction module; The first-order quantization cross-domain module is used to calculate the visual embedding feature Z and the first-order codebook Each first-order code word in The first-order code word with the largest similarity is used as the quantitative visual embedding feature The combined feature extraction module is used to extract the combined feature representation ζ of the input image x and send it to the second-order quantization module; the combined feature extraction module includes an attention module, a feature fusion module, a linear layer and a feedforward neural network, wherein: The attention module is used to use the visual embedding feature Z as the query in the attention mechanism and the second-order codebook subset As the key and value in the attention mechanism, K represents the number of item types contained in the second-order codebook subset. The attention mechanism is used to extract the image to obtain the attention vector v of the input image and send it to the feature fusion module; The feature fusion module is used to superimpose the attention vector v onto the visual embedding feature Z to obtain the feature z=Z+v and send it to the linear layer; The linear layer is used to perform a linear transformation on the feature z to obtain the feature LN(z) and send it to the feedforward neural network; The feedforward neural network is used to process the feature LN(z) to obtain the combined feature representation ζ of the input image x; The second-order quantization cross-category module is used to calculate the visual embedding feature Z and the second-order codebook Each second-order code word E i The similarity of i = 1, 2, ..., N, and the code word E is determined according to the similarity. i The weight w i , the greater the similarity, the higher the weight, and then combine to get the quantitative combination feature S3: Randomly initialize N code words D with dimension L i ′, for N code words D i ′ is normalized to obtain the code word D i , constituting the initialization of the first-order codebook S4: Uniformly sample from different categories of the training sample set X to obtain the training sample subset X of the current batch B , each training sample is denoted as x i,j , i = 1, 2, ..., N, j = 1, 2, ..., M, M represents the number of samples in each category; S5: Subset the training samples to X B For each training sample x i,j Input image quantization model, the visual feature extraction module and the first-order quantization module extract the visual embedding feature Z i,j And the quantized visual embedding features are S6: Quantized visual embedding features using N×M training samples The second-order codebook Ω is calculated based on the first-order codebook Ω (2) , the second-order codebook Ω (2) Each second-order code word E i The calculation formula is: S7: For the training sample subset X B For each training sample x i,j , from the second-order codebook Ω (2) Filter the second-order codebook subset that does not contain its real item type Embed the visual feature Z i,j and the second-order codebook subset The subsequent combined feature extraction module in the input image quantization model obtains the combined feature ζ i,j , and then input into the second-order quantization module based on the second-order code book Ω (2) Get quantitative combination features S8: Calculate the loss function and then update the parameters of the image quantization model and the first-order codebook; S9: Determine whether the training end condition is met, if so, the training ends, otherwise return to step S4; S10: The entire second-order codebook Ω (2) As a second-order codebook subset Then each image in the image library is input into the image quantization model, based on the trained first-order codebook Ω and second-order codebook Ω (2) and the second-order codebook subset Get the combined features of each image; S11: The entire second-order codebook Ω (2) As a second-order codebook subset Then the image to be retrieved is input into the image quantization model, based on the trained first-order codebook Ω and second-order codebook Ω (2) and the second-order codebook subset Get its combined features, then calculate the similarity between the combined features and the combined features of each image in the image library, select the R images with the largest similarity as the retrieval results and return them. The value of R is set according to actual needs.

2. The general image retrieval method according to claim 1, characterized in that: The visual feature extraction module in step S2 includes a visual Transformer neural network, a fully connected layer and a normalization module, wherein: The visual Transformer neural network is used to extract the original features of the image and output them to the fully connected layer; The fully connected layer is used to process the original features to obtain the initial visual embedding features. And output to the normalization module; The normalization module is used to normalize the initial visual embedding feature Z′ to obtain the visual embedding feature 3. The general image retrieval method according to claim 1, characterized in that: The calculation method of the loss function in step S8 is: The cross-domain loss L of the embedded feature is calculated using the following formula: Z : Among them, o() represents contrastive learning loss, <> represents similarity, e represents a natural constant, and τ represents a temperature coefficient; The cross-category loss L of the combined features is calculated using the following formula: ζ : Then the loss function L is calculated using the following formula: L=L Z +λL ζ Among them, λ represents the weight.

4. The general image retrieval method according to claim 1, characterized in that: The second-order codebook is cut into Q blocks. During the operation of the image quantization model, the visual embedding feature Z extracted by the visual feature extraction module is also cut into Q blocks, which correspond one-to-one to the cut codebook; each sub-block of the visual embedding feature Z is independently quantized and learned in its corresponding sub-codebook, and finally the final quantized combined feature is obtained by concatenating the quantized combined features of multiple sub-blocks.

Citation Information

Patent Citations

  • Data retrieval method based on Hash learning of dimension analysis quantizer

    CN112241475A

  • Method for acquiring image retrieval model, image retrieval method, device and equipment

    CN114299306A

  • Method for training image quantization model and method and device for retrieving image

    CN116051917A

  • Retrieval model training method and apparatus, retrieval method and apparatus, device and medium

    WO2023020214A1

Cited By

  • Long-term visual positioning method based on association guidance network

    CN119850894A