A method for image feature retrieval based on hypersphere adaptive filtering

Abstract

The present invention relates to the problem of approximate nearest neighbor search in the field of image retrieval, and discloses an image feature retrieval method based on hypersphere adaptive filtering, which mainly consists of three stages: obtaining a candidate set, hypersphere filtering, and reordering, including: combining an inverted index constructed based on residual quantization, using a fully connected neural network to construct a hypersphere with a query feature vector as the sphere center in an image feature library, and using the hypersphere to adaptively obtain an inverted list to be retrieved to form a candidate set; according to the spatial position relationship between the hypersphere and the query feature vector, filtering out feature vectors whose spatial positions are outside the hypersphere from the candidate set to streamline the candidate set; in the result reordering stage, according to the similarity or Euclidean distance between the query feature vector and the candidate feature vector, sorting the feature vectors in the candidate set and taking the first feature vector with the highest similarity or the smallest Euclidean distance as the final retrieval result. The present invention combines deep learning to construct a hypersphere model, which effectively improves retrieval efficiency while ensuring retrieval accuracy.

Description

Technical Field

[0001] The present invention belongs to the technical field of image retrieval, and more specifically, relates to an image feature retrieval method based on hypersphere adaptive filtering. Technical Background

[0002] In recent years, with the increasing number of Internet users and more frequent information exchanges, multimedia information represented by images has been generated and disseminated in large quantities, and the scale of these image data is growing exponentially. Therefore, how to efficiently and accurately retrieve the required images from the massive amount of images has become one of the key technical research areas of academic and industry attention.

[0003] The nearest neighbor search method is a basic and indispensable method in many technical fields, such as image retrieval, pattern recognition, and computer vision. However, when processing high-dimensional data represented by image features, the exact nearest neighbor search method requires high computational cost and large storage overhead. In order to achieve a better balance between retrieval efficiency and retrieval accuracy, the approximate nearest neighbor search method was proposed.

[0004] There are two mainstream methods for approximate nearest neighbor search: hash-based and quantization-based. Among them, the hash-based method uses a hash function to convert feature vectors into compact binary codes while maintaining the similarity of the original feature vectors. For this type of method, the Hamming distance is used to measure the distance between binary codes. However, the limitation of this type of method is that the discrimination of the Hamming distance is limited by the length of the code used, and only a limited distance value can be used to represent the similarity relationship between feature vectors, and it cannot measure the complex distance relationship required for the similarity between feature vectors. The quantization-based method solves this problem better. It uses an asymmetric distance formed by the Euclidean distance from the query feature vector to the code book to measure the similarity between feature vectors. While ensuring the retrieval accuracy, it solves the problem of high overhead when calculating the exact Euclidean distance.

[0005] In the approximate nearest neighbor search method, compared with the complete search that needs to retrieve all the feature vectors in the data set, its retrieval efficiency is proportional to the size of the data set. By constructing an index structure, incomplete retrieval only needs to retrieve part of the feature vectors, which can improve the retrieval efficiency while ensuring the retrieval accuracy. At present, typical approximate nearest neighbor search methods based on incomplete retrieval usually use the construction of an inverted index structure to accelerate the retrieval process. The ADC-based inverted file system is the first incomplete retrieval method that uses the k-means algorithm and the product quantization method for inverted indexing. Since then, codebook-based inverted index methods have been proposed one after another, typically including: inverted multi-layer index structure, which uses inverted index and multi-layer index structure to quickly locate similar feature vectors and achieve efficient nearest neighbor search efficiency; probability-based learnable index, which learns the similarity probability between feature vectors through a neural network, and uses the similarity probability size to retrieve the neighbor vectors of the feature vector.

[0006] Usually, the approximate nearest neighbor search process based on the inverted index structure can be divided into two stages: obtaining candidate sets and re-ranking. For a query feature vector, its candidate set is composed of all feature vectors in the inverted list corresponding to the nearest W index codewords based on the distance relationship between the query feature vector and the index codeword; re-ranking requires calculating the similarity or distance between the feature vector in the candidate set and the query feature vector and returning the closest R feature vectors as the retrieval result.

[0007] Based on the above background, the present invention designs an image feature retrieval method based on hypersphere adaptive filtering, which consists of three stages: obtaining a candidate set, hypersphere filtering, and re-ranking. In the hypersphere filtering stage, after obtaining the candidate set, a hypersphere with the query feature vector as the center is constructed to filter out feature vectors in the candidate set that are not similar to the query feature vector to streamline the candidate set, thereby reducing the time overhead of calculating the Euclidean distance and the sorting time overhead in the re-ranking stage, thereby accelerating the retrieval process. Specifically, through a hypersphere with the query feature vector as the center, all feature vectors in the sub-inverted list corresponding to the sub-cluster center outside the hypersphere are filtered out from the candidate set, thereby reducing the number of times the distance between the query feature vector and the candidate feature vector needs to be calculated and the number of sorting in the re-ranking stage, effectively improving the retrieval efficiency while ensuring the retrieval accuracy. Summary of the invention

[0008] In view of this, the object of the present invention is to propose an image feature retrieval method based on hypersphere adaptive filtering, by adding a hypersphere filtering stage after the candidate set acquisition stage to reduce the number of distance calculations and sorting times between the query feature vector and the candidate feature vector, and reduce the sorting time overhead and calculation time overhead, thereby improving the retrieval efficiency and ensuring the retrieval accuracy.

[0009] To achieve the above purpose, the specific technical solutions implemented by the present invention include:

[0010] There are three stages: obtaining candidate sets, hypersphere filtering, and re-ranking.

[0011] The obtaining of the candidate set includes three stages: constructing an inverted index structure based on residual quantization, training a hypersphere model 1 using a fully connected neural network, and obtaining the candidate set. The specific steps include:

[0012] (1) Constructing an inverted index structure based on residual quantization

[0013] Step A1: Use the k-means algorithm to train two groups of codebooks C and C1, where the first group of codebooks C is trained in the database vector set X2 and contains M codewords, and the other group of codebooks is trained in the residual set E and contains M1 codewords, where the residual set E is composed of the residual vector between each feature vector in the database vector set X2 and its nearest codeword in C;

[0014] Step A2: Use the codebook C to construct the first-level inverted index structure, obtain M inverted lists, and each inverted list is identified by a corresponding cluster center;

[0015] Step A3: Using the M1 codewords in the codebook C1 and combining them with each codeword in C, it is equivalent to dividing each inverted list into M1 categories, forming M1 sub-inverted lists, which are identified by corresponding sub-cluster centers. Therefore, the database vector set X2 is divided into M×M1 sub-inverted lists in total;

[0016] For the feature vectors in the database vector set X2, the specific steps of inserting the inverted index structure are as follows:

[0017] Step A4: Calculate the nearest cluster center index id in C of the feature vector distance contained in the database vector set X2;

[0018] Step A5: Calculate the residual vector between the feature vector in the database vector set X2 and its nearest cluster center, and construct the residual set E corresponding to the database vector set X2;

[0019] Step A6: Calculate the nearest cluster center index id1 in C1 from the residual vector contained in the residual set E;

[0020] Step A7: The feature vectors in the database vector set X2 are inserted into the M×(id-1)+id1th sub-inverted list;

[0021] (2) Using fully connected neural network to train the hypersphere model 1

[0022] Step B1: Given the image feature training vector set X1 and the database vector set X2, use the k-means algorithm to generate K cluster centers in X1, each cluster center corresponds to a cluster;

[0023] Step B2: Select T training feature vectors from X1 using a sampling method based on binomial distribution. The randomly sampled training feature vectors are denoted as x t (t=1,2,...,T);

[0024] Step B3: Calculate x t The Euclidean distance between the K cluster centers and the input vector D of the hypersphere model is calculated based on the function F(*) t =F(x t );

[0025] Step B4: Get x in the database vector set X2 t The S nearest neighbor vectors of x t and the longest Euclidean distance between S nearest neighbor vectors. At the same time, calculate x t and the Euclidean distance between the nearest cluster center, taking the maximum value of the two to get Tr t ;

[0026] Step B5: Get training data pair {D t ,Tr t}, repeat steps B3 and B4 until T groups of training data pairs are obtained;

[0027] Step B6: Transform T sets of training data into {D t ,Tr t}(t=1,2,...,T) is input into the fully connected neural network, and after multiple iterations until convergence, a neural network model of a hypersphere 1 with the input feature vector as the center is obtained;

[0028] Based on the trained hypersphere model 1, the output value of the input feature vector is calculated and used as the radius value of the hypersphere, so that a hypersphere with the feature point corresponding to the feature vector as the sphere center can be constructed in the vector space.

[0029] (3) Obtaining candidate sets

[0030] In the mainstream method, the number of inverted lists W constituting the candidate set is set based on human experience. This approach makes the process of obtaining the candidate set lack flexibility and generalization. To solve the above problems, a hypersphere 1 with the query feature vector as the center is constructed to adaptively obtain the candidate set. Taking the feature vector q in the test vector set X3 as an example, the specific steps are as follows:

[0031] Step C1: construct an input vector D=F(q) and calculate the radius of the hypersphere 1 Pr=f(D) based on the hypersphere model f(*);

[0032] Step C2: Calculate the Euclidean distance between q and M cluster centers (codewords);

[0033] Step C3: Filter out the cluster centers whose spatial positions are outside the hypersphere 1, and form a candidate set A with all the feature vectors in the inverted list corresponding to the W cluster centers located inside the hypersphere 1;

[0034] The hypersphere filtering includes two stages: using a fully connected neural network to train a hypersphere model 2 and hypersphere filtering. The specific steps include:

[0035] (1) Using fully connected neural network to train the hypersphere model 2

[0036] Step D1: Given the image feature training vector set X1 and the database vector set X2, use the k-means algorithm to generate K cluster centers in X1, each cluster center corresponds to a cluster;

[0037] Step D2: Select T training feature vectors from X1 using a sampling method based on binomial distribution. The randomly sampled training feature vectors are denoted as x t (t=1,2,...,T);

[0038] Step D3: Calculate x t and the residual vector of K cluster centers;

[0039] Step D4: Calculate the Euclidean distance between K residual vectors and the corresponding cluster centers to construct the input vector D1 t =F1(x t );

[0040] Step D5: Get x in the database vector set X2 t The S1 nearest neighbor vector of x t and the longest Euclidean distance between S1 nearest neighbor vectors, and calculate x t The Euclidean distance between the nearest cluster center and the nearest cluster center is taken as the maximum value to obtain Tr1. t ;

[0041] Step D6: Get training data pair {D1 t ,Tr1 t}, repeat steps D3, D4 and D5 until T sets of training data pairs are obtained;

[0042] Step D7: T sets of training data {D1 t ,Tr1 t}(t=1,2,...,T) is input into the fully connected neural network, and after multiple iterations until convergence, a neural network model of a hypersphere 2 with the input feature vector as the center is obtained;

[0043] Based on the trained hypersphere model 2, the output value of the input feature vector is calculated and used as the radius value of the hypersphere, so that a hypersphere with the feature point corresponding to the feature vector as the sphere center can be constructed in the vector space.

[0044] (2) Hypersphere filtration

[0045] The candidate set A obtained at this time is composed of a series of inverted sublists. For a sublist, it is equivalent to a cluster. The eigenvectors contained in it are usually considered to be similar to the subcluster centers. Therefore, the subcluster centers can be used to represent all the eigenvectors therein. Therefore, the subcluster centers corresponding to the sublists in the candidate set A are used as filtering objects, and the sublists corresponding to the subcluster centers outside the hypersphere 2 are filtered out. Only the sublists corresponding to the subcluster centers inside the hypersphere 2 are retained and the eigenvectors therein are sorted. Taking the eigenvector q as an example, the specific steps are as follows:

[0046] Step E1: construct an input vector D1=F1(q) and calculate the radius Pr1=f1(D1) of the hypersphere 2 based on the second hypersphere model f1(*);

[0047] Step E2: Calculate the Euclidean distance between the residual vector corresponding to q and the W×M1 sub-cluster centers;

[0048] Step E3: Filter out the feature vectors in the sub-inverted list corresponding to the sub-cluster centers outside the hypersphere 2 from the candidate set A, thereby streamlining the candidate set A.

[0049] The reordering includes a reordering operation. Taking the feature vector q as an example, the specific steps include:

[0050] Step F1: Calculate the Euclidean distance between q and the feature vectors in the candidate set A and sort them;

[0051] Step F2: Return the first R feature vectors with the smallest Euclidean distance as the final retrieval result. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flow chart of the image feature retrieval method based on hypersphere adaptive filtering of the present invention.

[0053] Figure 2 It is a schematic diagram of a fully connected neural network for training the hypersphere model 1 of the present invention.

[0054] Figure 3It is a schematic diagram of a fully connected neural network for training the hypersphere model 2 of the present invention.

[0055] Figure 4 This is a flow chart of the present invention using a fully connected neural network to train a hypersphere model

[0056] Figure 5 It is a schematic diagram of a module for constructing an inverted index structure based on residual quantization of the present invention. Specific implementation methods

[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in combination with the technical solution, drawings and embodiments.

[0058] The present invention proposes an image feature retrieval method based on hypersphere adaptive filtering, which includes three stages: obtaining candidate sets, hypersphere filtering and re-ranking. The complete process is as follows Figure 1 As shown in the figure: First, in the candidate set acquisition stage, combined with the inverted index structure based on residual quantization, the hypersphere model 1 is trained by using a fully connected neural network to construct a hypersphere 1 with the query feature vector as the center, and the cluster centers inside the hypersphere 1 correspond to the feature vectors in the inverted list to form a candidate set A; then, in the hypersphere filtering stage, the hypersphere model 2 is trained by using a fully connected neural network and a hypersphere 2 with the query feature vector as the center is constructed, and the feature vectors in the sub-cluster centers outside the hypersphere 2 corresponding to the sub-inverted list are filtered out from the candidate set A, thereby streamlining the candidate set A; finally, in the re-ranking stage, the Euclidean distance between the query feature vector and all the feature vectors in the candidate set A is calculated and sorted, and the top R feature vectors with the smallest Euclidean distance are returned as the final retrieval result.

[0059] More specifically, the following Figure 1 , 2 , 3, 4, and 5 describe in detail the image feature retrieval method based on hypersphere adaptive filtering of the present invention.

[0060] (1) Fully connected neural network

[0061] The specific structure of the fully connected neural network is as follows Figure 2 and Figure 3 As shown in the figure, it consists of 1 input layer, 2 hidden layers and 1 output layer. The number of units in the input layer, hidden layer and output layer are set to K, 512, 512 and 1 respectively, and the ReLU activation function is used between layers. The training data D or D1 is input into the input layer, passes through 2 hidden layers, and finally outputs the learned hypersphere radius Pr or Pr1 in the output layer. The MSE loss function is used to minimize the training error during the training process.

[0062] (2) Training the Hypersphere Model

[0063] The specific process of training the hypersphere model is as follows Figure 4 As shown in the figure, T training feature vectors are selected from the given image feature training vector set X1 in a binomial distribution-based manner to obtain the training data set X = {x1, x2, ..., x T}, taking the tth sampling as an example:

[0064] Step G1: Use the k-menas algorithm to generate K cluster centers U = {u1,u2,...,u K};

[0065] Step G2: From the training dataset Randomly sample training feature vector x t ;

[0066] Step G3: Calculate x t The Euclidean distance between the K cluster centers is used to construct the input vector D t =(d t,1 ,...,d t,k ,...,d t,K ), where d t,k is x t and the kth cluster center u k The Euclidean distance of

[0067] Step G4: Calculate x t and the residual vector {e t,1 ,...,e t,k ,...,e t,K}, where e t,k =x t -u k ;

[0068] Step G5: Calculate the Euclidean distance between K residual vectors and the corresponding cluster centers to construct the input vector in is u k and the kth residual vector e t,k The Euclidean distance of

[0069] Step G6: Find x in the database vector set X2 t S or S1 nearest neighbor vectors G t = {g t,1 ,...,g t,s ,...,g t,S}or Calculate x t The maximum Euclidean distance Tr between S or S1 nearest neighbor vectors t or Tr1 t , where Trt and Tr1 t It can be expressed as:

[0070]

[0071] where ||.|| represents the Euclidean distance; then, calculate x t and the Euclidean distance r from the nearest cluster center, where r can be expressed as:

[0072]

[0073] Take the maximum value of the two and optimize Tr t or Tr1 t , where Tr t =MAX(Tr t ,r t ), Tr1 t =MAX(Tr1 t ,r t );

[0074] Step G7: Transform the training data into {D t ,Tr t} or {D1 t ,Tr1 t} is input into the fully connected neural network to learn the radius of the hypersphere Pr t or Pr1 t And calculate the training loss MIS or MIS1, the calculation formula is as follows:

[0075]

[0076] Step G8: Based on the obtained MIS or MIS1, the back propagation algorithm is used to update the parameters of each layer of the fully connected neural network, and multiple iterations are performed until the final convergence;

[0077] (3) Constructing an inverted index structure based on residual quantization

[0078] Construct an inverted index structure based on residual quantization in the database vector set X2. Figure 5 The detailed process of building the index structure is shown as follows:

[0079] Step H1: Generate a set of codebooks C = {c 1 ,...,c m ,...,c M}, these codeword indices are used as quantized codes of feature vectors, and the corresponding inverted list is represented as L = {l1,l2,...,l M};

[0080] Step H2: Each eigenvector {v1,...,v i ,...,v N} is inserted into the inverted list corresponding to the cluster center closest to it, where the mth inverted list l m The included eigenvectors can be expressed as:

[0081]

[0082] Step H3: Calculate each eigenvector v i (i=1,2,...,N) corresponding to the residual vector e i , construct the residual set E = {e1,...,e i ,...,e N}, where e i is the eigenvector v i The corresponding residual vector can be expressed as:

[0083]

[0084] Step H4: Generate a set of codebooks using the k-means algorithm in the residual set E

[0085] Step H5: Each inverted list l m (m=1,2,...,M), and then divided into M1 sub-posting lists, where represents the m1-th sub-posting list in the m-th posting list, and the feature vector it contains can be expressed by the following formula:

[0086]

[0087] The corresponding sub-cluster center is

[0088] (4) Image feature retrieval process based on hypersphere adaptive filtering

[0089] Figure 1 The detailed process of image feature retrieval based on hypersphere adaptive filtering is shown. Taking the feature vector q in the test vector set X3 as an example, the specific process is as follows:

[0090] ① Get candidate set

[0091] Step I1: construct a hypersphere 1 with q as the center and a radius of Pr;

[0092] Step I2: Calculate the Euclidean distance between q and M cluster centers, where the Euclidean distance between q and the mth cluster center can be expressed as ||qc m ||;

[0093] Step I3: Determine the similarity between q and M cluster centers B = {b1,...,b m ,...,b M}, where b m The calculation formula is as follows:

[0094]

[0095] Step I4: The feature vectors in the inverted list corresponding to W cluster centers with a similarity of 1 to q form a candidate set A;

[0096] ②Hypersphere filtration

[0097] Step J1: construct a hypersphere 2 with q as the center and a radius of Pr1;

[0098] Step J2: Calculate the Euclidean distance between q and the sub-cluster center corresponding to the W×M1 sub-inverted lists contained in the candidate set, where (w=1,2...,W;m1=1,2,...,M1) represents the Euclidean distance between q and the (w-1)×M1+m1th sub-cluster center;

[0099] Step J3: Determine the similarity between q and W×M1 cluster centers Among them, sb w×m1 The calculation formula is as follows:

[0100]

[0101] Step J4: Filter out the feature vectors in the sub-inverted list corresponding to the sub-cluster center with a similarity of 0 to q from the candidate set A, thereby streamlining the candidate set A;

[0102] ③ Reorder

[0103] Calculate the Euclidean distance between q and all feature vectors in the candidate set A and sort them, and return the first R feature vectors with the smallest Euclidean distance as the final retrieval result.

[0104] The specific embodiments described above further describe the purpose and technical solutions of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An image feature retrieval method based on hypersphere adaptive filtering, characterized in that: The retrieval process includes: obtaining candidate sets, hypersphere filtering and re-ranking; The obtaining of the candidate set comprises: combining an inverted index structure constructed based on residual quantization, using a fully connected neural network in the image feature library to construct a hypersphere 1 with the query feature vector as the sphere center, and using the hypersphere to retrieve an adaptive number of inverted lists to obtain the candidate set; The hypersphere filtering comprises: constructing a hypersphere 2 with the query feature vector as the center in the image feature library using a fully connected neural network, and filtering out the candidate feature vectors whose spatial positions are outside the hypersphere 2 from the candidate set according to the spatial position relationship between the hypersphere and the candidate feature vectors to simplify the candidate set; The re-ranking includes: calculating the similarity or Euclidean distance between all feature vectors in the simplified candidate set and the query feature vector and ranking them, and taking the first R feature vectors with the highest similarity or the smallest Euclidean distance as the final retrieval result; The process of training the hypersphere model 1 is as follows: First, given an image feature dataset X, which contains three subsets: training vector set X1, database vector set X2, K cluster centers are generated in the training vector set X1 through clustering technology. In addition, T training feature vectors are selected from the training vector set X1 according to the sampling method based on binomial distribution, thus forming a training dataset for training hypersphere model 1 Secondly, the learning process of the hypersphere model 1 is formalized as a function f: f(F(x)) = Tr. Using the transformation function F, the input vector D = F(x) that is independent of the dimension of the training feature vector is calculated based on the Euclidean distance from the training feature vector x to the K cluster centers. Then, calculate the S nearest neighbor vectors with the closest Euclidean distance to x in the database vector set X2, and take the farthest Euclidean distance from x to the above S nearest neighbor vectors as Tr. To ensure that at least one inverted list can be retrieved during the retrieval process, use the Euclidean distance between x and the nearest codeword to optimize Tr; Finally, the training data pair {D, Tr} is input into the fully connected neural network, and the training loss is calculated using the MSE loss function to obtain the neural network model of the hypersphere 1 with the feature vector x as the center of the sphere; The process of training the hypersphere model 2 is as follows: Calculate the residual vector between the training feature vector x and the K cluster centers to form the training data for training the hypersphere model 2 2. The image feature retrieval method based on hypersphere adaptive filtering according to claim 1, characterized in that: The process of obtaining the candidate set is as follows: Firstly, an inverted index structure based on residual quantization is constructed to divide the image feature library into several inverted lists. Each inverted list corresponds to a cluster center. The feature vectors contained in the inverted list are usually considered to be similar to the cluster center, so the cluster center can be used to represent all the feature vectors in it. Then, a fully connected neural network is used to construct a hypersphere 1 with the query feature vector as the center; Finally, the feature vectors in the inverted list corresponding to the cluster centers whose spatial positions are outside the hypersphere 1 are filtered out, and only the feature vectors in the inverted list corresponding to the cluster centers inside the hypersphere 1 are used to form the candidate set.

3. The image feature retrieval method based on hypersphere adaptive filtering according to claim 1, characterized in that: The process of the hypersphere filtering stage is: First, on the basis of the inverted index structure based on residual quantization, each inverted list is divided into several sub-inverted lists again. Therefore, the obtained candidate set consists of a series of sub-inverted lists, each of which corresponds to a sub-clustering center. Then, a fully connected neural network is used to construct a hypersphere 2 with the query feature vector as the center; Finally, the query feature vector and the feature vector in the sub-inverted list corresponding to the sub-cluster center located inside the hypersphere 2 are considered similar results, and all feature vectors in the sub-inverted list corresponding to the sub-cluster center located outside the hypersphere 2 are filtered out from the candidate set to streamline the candidate set.

4. The image feature retrieval method based on hypersphere adaptive filtering according to claim 3 is characterized in that: The process of training the hypersphere model 2 also includes: The learning process of the hypersphere model 2 is formalized as a function f1: f1(F1(x)) = Tr1. Using the transformation function F1, according to the training data The Euclidean distance to the corresponding cluster center is calculated as the input vector D1=F1(x) which is independent of the dimension of the training feature vector; Then, calculate the S1 nearest neighbor vectors with the closest Euclidean distance to x in the database vector set X2, and take the farthest Euclidean distance from x to the above S1 nearest neighbor vectors as Tr1. To ensure that at least one inverted list can be retrieved during the retrieval process, use the Euclidean distance between x and the nearest codeword to optimize Tr1; Finally, the training data pair {D1, Tr1} is input into the fully connected neural network, and the training loss is calculated using the MSE loss function to obtain the neural network model of the hypersphere 2 with the feature vector x as the center.

Images