An efficient occlusion-robust large-scale image search method

Abstract

The present invention discloses an efficient, occlusion-robust, large-scale image search method in the field of image processing technology, comprising the following steps: S1: extracting feature points and descriptors; S2: dividing feature points based on color information; S3; quantizing feature points using a corresponding category codebook; S4; aggregating quantized indexes into feature vectors; steps S1 and S2 are classified as HxW template images, and steps S3 and S4 are classified as KxD / K template feature vectors. The beneficial effects of the present invention are: by analyzing the color and spatial structure of the target image, feature points are generated, and the feature points are quantized, and the search is completed based on the quantization results, thereby improving the accuracy of the search; by using a search method different from the prior art, the search accuracy of the occluded image is improved, the search time is reduced, and the timeliness of the image search is improved; and by adopting a different search method, the image search method has a wider range of applications and stronger practicality.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an efficient occlusion-robust large-scale image search method. Background Art

[0002] Picture book and children's book reading has become a crucial learning tool in early childhood education. Introducing AI-based image search (image-based search) technology into educational electronics can quickly enable smart devices to identify the page a user (child) is currently reading, enabling richer user interactions.

[0003] Current electronic products use a two-step process for image search. The first step is feature extraction, which involves expressing the image as a high-dimensional feature vector. The second step is feature search, which involves searching a template feature library for the feature vector that is most similar to the feature vector corresponding to a particular image.

[0004] Based on the difference in feature extraction, image search technology is currently divided into two categories, which are briefly introduced as follows:

[0005] Methods based on traditional vision: first extract the feature points in the image and the feature descriptors corresponding to the feature points, and then aggregate all feature descriptors into a global feature vector of the image, such as the BOW bag-of-words model (Fei-Fei, L. and Perona, P., "A Bayesian Hei rarcical Model l for Learning Natural Scene Categories", Proc. CVPR, 2005.).

[0006] Method based on deep learning: First, a large-scale data set is used to train the neural network model, and then the image is input into the neural network model, and the output is the feature vector of the image.

[0007] Feature search can be viewed as a nearest neighbor search problem for high-dimensional features. For large-scale searches, efficiency is often prioritized, often resulting in only an approximate nearest neighbor result. The current mainstream approach to this problem is to model the feature database, using methods such as hierarchical trees (KD-TREEs) and graph structures (HNSWs), leveraging the data structure to support fast search.

[0008] Existing technologies have the following shortcomings: large-scale feature search operations take a long time and cannot meet the real-time requirements of scenarios such as smart desktops and human-computer interaction; in the feature extraction process, most of them only consider the global features of the image, and are not well adapted to occlusion or incomplete shooting, which often occurs in desktop scenarios. Summary of the Invention

[0009] The purpose of the present invention is to provide an efficient, occlusion-robust large-scale image search method. This method can establish corresponding feature points for the color and spatial structure of the image and quantify the feature points, thereby achieving accurate search and matching of the image. This image search method is fast and highly real-time. While increasing the speed, the search accuracy is not reduced. It is applicable to the occlusion phenomenon that often exists in desktop imaging scenarios, so as to solve the problem that the large-scale feature search proposed above takes a long time to calculate and cannot meet the real-time requirements of scenarios such as smart desktops and human-computer interaction. In the feature extraction process, most of the time, only the global features of the image are considered, which is not well adapted to occlusion or incomplete shooting, which often occurs in desktop scenarios.

[0010] To achieve the above objectives, the present invention provides the following technical solution: an efficient occlusion-robust large-scale image search method, comprising the following steps:

[0011] S1: Extract feature points and descriptors;

[0012] S2: Divide feature points based on color information;

[0013] S3; corresponding category codebook quantized feature points;

[0014] S4; aggregate the quantized index into a feature vector;

[0015] The steps S1 and S2 are classified as HxW template images, and the steps S3 and S4 are classified as KXD / K template feature vectors. The steps S1, S2, S3 and S4 also include the following process: determining the HxW image to be searched, dividing the image to be searched into MxN grids, comparing similar templates, extracting and aggregating features for each grid, performing feature vector analysis on the search image, extracting the D / K feature vector of the next grid, selecting the corresponding category database according to the feature vector, and requiring the matching similarity of each sample, sorting according to the similarity, extracting a most similar templates, using an image matching algorithm to select the most similar template from a samples, and rechecking the matching results.

[0016] As a further solution of the present invention: when rechecking the matching result, if the recheck result is a correct result, it is determined that the searched image template is the required template. If the recheck matching result is an incorrect result, it returns to the sub-step and re-extracts the D / K feature vector of the next grid until the matching reaches the template with the highest similarity.

[0017] As a further solution of the present invention: when the feature points and descriptors are extracted in step S1, the color and structural composition of the target object are simultaneously extracted to determine whether the target object is blocked, and then a search is performed based on the feature points, descriptors, colors and structures extracted from the target object.

[0018] As a further solution of the present invention: when step S2 performs feature point division according to the target color extracted in step S1, the target is divided into multiple color categories according to the color extracted from the target, and is divided into K vector codebooks in combination with the target layout. The search target is divided into multiple grid areas and feature points are obtained, and a synchronous search is performed based on the feature points of different grid areas.

[0019] As a further solution of the present invention: Step S3 completes the quantization of the feature points according to the category codebook generated in Step S2, and feeds the quantization result back to Step S4.

[0020] As a further solution of the present invention: Step S4 performs indexing based on the quantized feature points transmitted in Step S3, aggregates them into feature vectors, and searches for the target based on the aggregated feature vectors.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. In the present invention, the color and spatial structure of the target image are analyzed to generate feature points, which are then quantified. The search is performed based on the quantization results, thereby improving the accuracy of the search.

[0023] 2. In the present invention, a search method different from the prior art is used to improve the search accuracy of the blocked image, shorten the search time, and improve the timeliness of the image search.

[0024] 3. In the present invention, due to the use of different search methods, the image search method is more widely applicable and more practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of the method flow structure of the present invention;

[0026] Figure 2 Schematic diagram of the search process structure in the present invention. DETAILED DESCRIPTION

[0027] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0028] See also Figure 1 ,When performing image search, a single image is expressed as K groups of vectors.

[0029] The training process of this algorithm is similar to the BOW method based on traditional vision. First, the feature points and corresponding descriptors in the template image are extracted (this algorithm supports but is not limited to the use of methods such as S IFT, SURF, etc., and the descriptor is actually expressed as a d-dimensional vector). Then, unlike the BOW method, we first divide the feature points into K categories, and then select the codebook corresponding to the category according to the feature point category (that is, the set of feature point cluster centers, the cluster center actually expresses a d-dimensional vector) for quantization, and obtain the quantized index of the current feature point in the codebook (the cluster center index with the closest Euclidean distance to the feature point descriptor).

[0030] 1. Feature point definition:

[0031] Specifically, for an image I, for example, n feature points are extracted using the SIFT algorithm, and the i-th feature point is represented by {xi,yi,ci,Vi}, where xi and yi represent the coordinates of the feature points, Vi is a d-dimensional description vector, and ci is the category of the feature point, where ci is obtained by color classification of the feature points.

[0032] 2 Feature point classification:

[0033] A 5x5 patch can be formed around the point (xi, yi). Expanding the RGB pixel values of the patch can obtain a 75*1 color vector Ui. The kmeans algorithm can be used in advance to obtain K cluster centers {UC0, UC1...UCK}. The cluster center to which Ui is closer is considered to be that category.

[0034] ci=argmi n{||U i–UCj||, j=1,…K}

[0035] 3 Image features are quantized in the codebook:

[0036] The ci-th codebook is selected according to the feature point category. The codebook can actually be understood as the cluster center of the m-dimensional feature vector {VCc i0, VCc i 1...VCc im}. Quantization is to assign the index I dx(i) of the nearest cluster center to the feature point {xi,yi,ci,Vi}

[0037] I dx(i)=argmi n{||Vi–VCci s||, s=1,…m}

[0038] 4 Aggregate representation of quantized index:

[0039] The n feature points of an image are ultimately converted into n quantized indices in different codebooks. These n quantized indices are grouped according to the codebook, resulting in K groups. The indices of each group are pushed together, and the same indices are combined and accumulated to form a vector of dimension D / K. (Given that D is much larger than n, this vector is generally very sparse.) Thus, an image I is ultimately represented as K vectors J1, J2, ..., JK of dimension D / K.

[0040] Compared to the original BOW method, this algorithm maintains the same total codebook size D (search accuracy is directly related to codebook size, so a constant size means almost constant search accuracy). However, the codebook size for each category is reduced by 1 / K of the original. Therefore, after aggregating the quantized indices into feature vectors, the image is converted into a K x D / K two-dimensional feature vector. After integrating L template images, we obtain K L x D / K feature databases.

[0041] See also Figure 2 , the search process of this algorithm is as follows Figure 2 As shown in the figure, unlike the template processing process, in the feature extraction process, we first divide the image into MxN grids, and then extract feature points and descriptors for each grid image. At this time, the same method as template processing is used to obtain the feature vector expressing the grid image. Finally, an image to be searched is converted into KxMxN D / K-dimensional vectors. The K D / K-dimensional vectors in each network will constitute our search unit.

[0042] Image search process, where the template library has been expressed as K sets of vectors in offline training.

[0043] During the search phase, each search unit is scanned one by one and searched in the feature database of its corresponding category. Specifically, each template is first matched in the database to obtain the corresponding similarity. The t-th template image is defined as the vector group {J1(t), J2(t)...JK(t)}, and the current search unit is represented as the vector group {Qmn1, Qmn2...QmnK}. The similarity calculation formula is as follows:

[0044] S im_mn(t)=-Sum(||Jk(t)–Qmnk||)

[0045] Here, the subscripts m and n actually represent a local region in the image. The present invention scans all possible local regions one by one. A local region is then compared with all templates for similarity, and all similarities are finally ranked. The first a templates with the highest similarity are obtained. After obtaining a candidate templates, one-to-one image matching is then used to determine the final template. One-to-one image matching is the process of matching two images and calculating the transformation matrix between them. This method is relatively mature and is not the innovation of this paper. For details, see the SIFT method (Lowe, D.I. Stimulating image features from scale-invariant keypoints. International Journal of Computer Vision, 60, 2 (2004), pp. 91-110). Once similar templates are confirmed through one-to-one image matching (i.e., a valid transformation matrix exists between the two images), the search process is immediately terminated. Otherwise, another vector is extracted and the search is repeated, and this cycle continues until the end.

[0046] The entire search process can be understood as using only a local area of the image to be searched. When occlusion occurs in the actual scene, this local area can minimize the interference caused by the occlusion.

[0047] Complexity analysis:

[0048] Assuming P feature points are extracted from an image (the average number of feature points per image in the template is Q), the number of feature points corresponding to each search in this algorithm is P / (M x N). The main bottlenecks in the search process are feature point quantization and feature search. The following section analyzes the complexity of these two steps separately.

[0049] Feature point quantization: A codebook is used to quantize all feature points in the image. The quantization process typically requires calculating the Euclidean distance between the feature point descriptor and the cluster center. Before the improvement, when feature points were not classified, the complexity was O(P*D*d*d). After the improvement, the feature points are divided into K categories, with the complexity of one category being O(P / K*D / K*d*d), and the complexity of the K categories being O(P*D*d*d / K). This shows that in the feature point quantization phase, the complexity of this algorithm is 1 / K of the original complexity.

[0050] Feature Search: During the search phase, a D-dimensional vector (D / K after the improvement) needs to be searched. D is typically very large, and this vector is sparse. The complexity of quickly calculating the Euclidean distance between two sparse vectors is related to the number of nonzeros in the vector, not the dimensionality. Therefore, the complexity is proportional to the number of feature points to be searched. Therefore, before the improvement, the search complexity was L*P. After the improvement, the complexity of a single search is L*P / (M x N). In the worst case, when all search units are searched, the complexity remains the same as before the improvement. However, in practice, only about 1 / 4 of the search time is needed to find the final template, exiting the search process. This shows that the performance of this algorithm in the feature search phase is 1 / 4 of the performance of the original algorithm.

[0051] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. An efficient occlusion-robust large-scale image search method, comprising the following steps: S1: Extract feature points and descriptors; S2: Divide feature points based on color information; S3; corresponding category codebook quantized feature points; S4; aggregate the quantized index into a feature vector; Its characteristics are: The steps S1 and S2 are classified as HxW template images, and the steps S3 and S4 are classified as KXD / K template feature vectors. The steps S1, S2, S3 and S4 also include the following process: determining an HxW image to be searched, dividing the image to be searched into MxN grids, comparing similar template images, extracting and aggregating features for each grid, extracting feature vectors for the image to be searched, extracting the D / K feature vectors of the next grid, selecting a corresponding category database according to the feature vector, and requiring the matching similarity of each sample, sorting according to the similarity, extracting a most similar templates, using an image matching algorithm to select the most similar template from a samples, and rechecking the matching results, wherein the parameters "H, W" represent the height and width of the template image, "K" represents the number of color categories, "M, N" represent the number of grids into which the image to be searched is divided, and "D" represents the dimension of the feature vector.

2. The efficient occlusion-robust large-scale image search method according to claim 1, characterized in that: When rechecking the matching result, if the recheck result is a correct result, it is determined that the searched image template is the required template. If the recheck matching result is an incorrect result, it returns to the step of re-extracting the D / K feature vector of the next grid, and re-extracts the D / K feature vector of the next grid until the matching reaches the template with the highest similarity.

3. The efficient occlusion-robust large-scale image search method according to claim 1, characterized in that: When extracting feature points and descriptors in step S1, the color and structural composition of the target object are simultaneously extracted to determine whether the target object is blocked, thereby performing a search based on the extracted feature points, descriptors, colors and structures of the target object.

4. The efficient occlusion-robust large-scale image search method according to claim 1, characterized in that: When step S2 performs feature point division according to the target color extracted in step S1, the target is divided into multiple color categories according to the color extracted from the target, and is divided into K vector codebooks in combination with the target layout. The search target is divided into multiple grid areas and feature points are obtained, and a synchronous search is performed based on the feature points of different grid areas.

5. The efficient occlusion-robust large-scale image search method according to claim 1, characterized in that: The step S3 completes the quantization of the feature points according to the category codebook generated in the step S2, and feeds the quantization result back to the step S4.

6. The efficient occlusion-robust large-scale image search method according to claim 1, characterized in that: The step S4 performs indexing based on the quantized feature points transmitted in the step S3 and aggregates them into feature vectors, and performs target search based on the aggregated feature vectors.

Images