Image retrieval system and method based on color moment and deep learning features

Abstract

The invention discloses an image retrieval system and method based on color moments and deep learning features. The system comprises an image feature extraction module, an index construction module, a retrieval trap door generation module and an image retrieval module. The image feature extraction module extracts color moment features and deep learning features of images in the image data set to obtain image multi-level features; the index construction module is used for constructing an index tree from bottom to top for color moment clustering; the trap door generation module extracts hierarchical features of the query image to generate a retrieval image trap door; the image retrieval module traverses the index tree from top to bottom according to the retrieval trap door to obtain a similar image candidate set, calculates the similarity between the depth feature of the image in the candidate similar image set and the depth feature of the query image, and outputs top-k images similar to the query image; according to the method, the color moments of the images are extracted and clustered, and the index tree is constructed, so that the similarity between the comparison images and the query images does not need to be calculated one by one during retrieval, and the image retrieval efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the field of image retrieval, and specifically relates to an image retrieval method based on deep learning features and color moments. Background Art

[0002] With the development of the Internet and smart devices, the amount of image data is growing exponentially, and local computing and storage resources are gradually unable to meet user needs. Cloud servers have abundant storage and computing resources, which can meet the storage and retrieval of large-scale image data sets for resource-constrained users. Therefore, more and more users choose to outsource images to cloud platforms. The introduction of content-based image retrieval technology allows users to quickly retrieve the required outsourced images without downloading all images locally, reducing computing and communication overhead. However, there are still the following problems with image retrieval efficiency, computing overhead, and application scenarios.

[0003] Low retrieval efficiency. The image retrieval solution in the cloud environment can extract feature vectors locally and upload them to the cloud server. When retrieval is needed, the user sends a retrieval trap, and the cloud server will perform a similar image retrieval and return the result set to the user. However, most existing solutions have a linear time complexity in retrieving similar images. They need to calculate the similarity of the query image and the image feature vectors in the image dataset one by one, which is relatively inefficient.

[0004] The computational overhead is high. The existing image feature extraction schemes include global features and local features of the image. Global features refer to the shape, texture and color of the image. Local features include scale-invariant feature transform (SIFT), histogram of oriented gradients (HOG), local binary patterns (LBP), etc. These features are relatively simple and cannot fully represent the features of the image, resulting in low accuracy of image retrieval. Convolutional neural networks based on feature learning have replaced the state-of-the-art technology of traditional hand-designed feature representation. Deep learning is a hierarchical feature representation technology used to learn abstract features from data that are critical to the dataset and application. However, the feature dimension of deep learning is very large. Even if most schemes use principal component analysis (PCA) to reduce the dimension of deep learning features, in order to maintain a good effect, the vector dimension will be maintained at around 128 dimensions. For large-scale image datasets, the computational overhead of calculating the similarity of feature vectors one by one is huge. Summary of the invention

[0005] Data owners have a large number of images, but their computing and storage resources are limited. In order to provide data owners with powerful computing and storage resources, outsourcing to cloud services is the choice of most users. In order to ensure that users can retrieve the required images from the images stored in the cloud, an efficient content-based image retrieval scheme is proposed. Specifically, the data owner uploads the image data set to the cloud server. When there is a retrieval demand, the user generates a retrieval trap and sends it to the cloud server. The cloud server performs the retrieval operation and returns the top-k similar images. However, most retrieval schemes have low efficiency and retrieval accuracy. In order to solve the problems of the prior art, the present invention proposes an image retrieval method based on color moments and deep learning features. First, the color moments of the image are extracted, clustered and an index tree is constructed. When searching, there is no need to calculate the similarity between the comparison image and the query image one by one. The construction of the tree index improves the retrieval efficiency. Secondly, the VGG16 pre-trained model is used to extract deep learning features, and PCA is used to reduce the dimension of the deep feature vector, and the deep features are used to improve the accuracy of the retrieval method. Finally, during retrieval, the index tree constructed using color moments is first used to determine the candidate range of similar images. The similarity between the images in the candidate dataset and the query image is then calculated, and the top-k retrieval results are returned. Low-dimensional feature vectors are used to calculate large-scale image datasets, and deep features are calculated on the screened candidate datasets to reduce computational overhead.

[0006] In order to solve the problems of the prior art, the present invention adopts the following technical solutions:

[0007] An image retrieval system based on color moment and deep learning features, the system comprising: an image feature extraction module, an index construction module, a retrieval trap generation module and an image retrieval module; the image feature extraction module extracts the color moment features and deep learning features of images in an image data set to obtain multi-level features of the image;

[0008] The index building module builds an index tree from bottom to top by clustering color moments;

[0009] The trapdoor generation module extracts the hierarchical features of the query image and retrieves the image trapdoor according to the following formula;

[0010] TD={f q ,cf q}

[0011] Where: f q Represents the image depth feature vector, cf q Represents the image color moment vector;

[0012] The image retrieval module traverses the index tree from top to bottom according to the retrieval trap to obtain a candidate set of similar images; calculates the similarity between the deep features of the images in the candidate similar image set and the deep features of the query image, and outputs the top-k images similar to the query image.

[0013] Furthermore, the index building module constructs an index tree process from bottom to top by clustering color moments, including:

[0014] S201, clustering the color moments of the n images into k classes using the K-means clustering method, taking each class as a leaf node, and then the leaf nodes can be represented by the k clusters;

[0015] S202, initialize a set AC to represent the currently existing classes, then calculate the distance of each cluster in the set AC, merge the clusters with the closest distance, and then update the classes in AC;

[0016] S203, repeat the process of step 2 until only one class remains, generate an index tree constructed from bottom to top according to the construction process, and the index tree construction is completed.

[0017] Furthermore, the image output module calculates the similarity between the deep features of the images in the candidate similar image set and the deep features of the query image, and outputs the top-k images similar to the query image, including: a method of querying the index tree from top to bottom,

[0018] S301, in each layer, by comparing with the color moment vector cf q The vector with the smallest distance locates the node, thus finding the entry point of the next layer;

[0019] S302: After finding the entry point of the next layer, if it is not a leaf node, continue to iterate through these nodes; otherwise, obtain the candidate image data set and proceed to the next step;

[0020] S303, calculate the depth feature vector f of each image in the candidate image dataset i and the feature vector f of the query image q The inner product value between them is D i,q =f i ·f q , and then sort the calculation results to get the top-k similar results.

[0021] The present invention also adopts the following technical solution: an image retrieval method based on color moment and deep learning features, comprising the following steps: using a pre-trained VGG16 model to calculate the deep learning feature vector of each image mi in the image data M, and using PCA dimensionality reduction to obtain the deep learning feature vector fi corresponding to each image;

[0022] Calculate the color moment of each image mi in the image data M to obtain a nine-dimensional color moment vector cf i;

[0023] Cluster the color moments, take the obtained classes as leaf nodes, merge the nearest nodes and build the index tree from bottom to top;

[0024] Input the query image mq and calculate its corresponding depth feature vector fq and color moment cfq;

[0025] Traverse the index tree from top to bottom according to the color moment, select the child nodes that are closer to the color moment of the query image until the leaf node, and obtain the candidate similar image set;

[0026] The similarity between the deep features of the images in the candidate similar image set and the deep features of the query image is calculated to obtain the top-k images similar to the query image.

[0027] Beneficial Effects

[0028] Compared with the traditional technical solution, the beneficial effects brought by the present invention are:

[0029] (1) The present invention comprehensively considers image retrieval based on the combination of deep learning features and color moments, and uses image color features and semantic features to better describe the features of the image, thereby improving the accuracy of retrieval.

[0030] (2) The present invention constructs a tree index based on color moments, which further improves the retrieval efficiency of the image retrieval method compared with the method of calculating the similarity between the images in the image data set and the query images one by one.

[0031] (3) The present invention adopts color moment to filter the target data set, uses color moment to perform similarity matching on a large-scale data set to obtain a candidate image data set, and further uses deep learning features to perform similarity matching on a small-scale data set. Compared with the method of directly using deep features for calculation, the computational overhead of the image retrieval method is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 The flowchart of an image retrieval method based on color moment and deep learning features. DETAILED DESCRIPTION

[0033] The following is combined with Figure 1 The present invention is described as follows:

[0034] like Figure 1 As shown, the present invention provides an image retrieval method based on deep learning features and color moments, and the specific steps are as follows:

[0035] (1) Extract deep features from images.

[0036] For the stored image dataset, the VGG16 pre-trained model is used to extract the 1000-dimensional deep learning feature vector of each image in the collection, and the PCA dimensionality reduction is used to obtain the 128-dimensional deep learning feature vector f i .

[0037] (2) Build an index tree.

[0038] For the stored data set M, calculate the color moment of each image. Use K-means clustering method to cluster n images m1, m2, ..., m n The color moments cf1, cf2, …, cf n Cluster into K classes, and take each class as a leaf node, then the leaf node can be represented as C 0,1 ,C 0,2 ,…,C 0,K Initialize the set AC = {C 0,1 ,C 0,2 ,…,C 0,K} to represent the current class, then calculate the distance of each cluster in the set AC, merge the clusters with the closest distance and update the class in AC. The merge is as follows: calculate the node B closest to the current node A. If node B has a subset, then calculate the threshold between the current node A and node B. If DM A,B If the value of node A is greater than the set threshold Th, nodes A and B are merged into a new subset; otherwise, node A is merged into node B. The above steps are repeated until only one class is left, and the index tree I is constructed.

[0039] (3) Generate retrieval trap.

[0040] Given a query image m q , calculate the corresponding depth feature vector f q and the color moment vector cf q ; Get the search trap TD = {f q ,cf q}.

[0041] (4) Retrieve similar images.

[0042] After receiving the query request TD submitted by the image user, the matching nodes are searched from top to bottom in the index tree I. In each layer, by comparing with the vector cf q The vector with the smallest distance locates the node, thereby finding the entry point of the next layer. After finding the matching node, the cloud server will iteratively traverse these nodes until a matching leaf node is found. After finding the matching leaf node, the candidate search results are locked in the cluster represented by these nodes. Calculate the deep feature vector f of each image in the candidate image dataset i and the feature vector f of the query imageq The inner product value between them is D i,q =f i ·f q , and then sort the calculation results to get the top-k similar results.

[0043] Although the present invention has been described above, the present invention is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can make many modifications without departing from the purpose of the present invention, which are all within the protection of the present invention.

Claims

1. An image retrieval system based on color moment and deep learning features, characterized in that: The system comprises: an image feature extraction module, an index construction module, a retrieval trap generation module and an image retrieval module; The image feature extraction module extracts the color moment features and deep learning features of the image in the image data set to obtain multi-level features; the index construction module clusters the color moments to construct an index tree from bottom to top; the trapdoor generation module extracts the hierarchical features of the query image and retrieves the image trapdoor according to the following formula; TD={fq,cfq} Where: fq represents the image depth feature vector, cfq represents the image color moment vector; The image retrieval module traverses the index tree from top to bottom according to the retrieval trap to obtain a candidate set of similar images; calculates the similarity between the deep features of the images in the candidate similar image set and the deep features of the query image, and outputs the top-k images similar to the query image.

2. The image retrieval system based on color moment and deep learning features according to claim 1, characterized in that: The index building module constructs an index tree process from bottom to top by clustering color moments, including: S201, clustering the color moments of the n images into k classes using the K-means clustering method, taking each class as a leaf node, and then the leaf nodes can be represented by the k clusters; S202, initialize a set AC to represent the currently existing classes, then calculate the distance of each cluster in the set AC, merge the clusters with the closest distance, and then update the classes in AC; S203, repeat the process of step 2 until only one class remains, generate an index tree constructed from bottom to top according to the construction process, and the index tree construction is completed.

3. The image retrieval system based on color moment and deep learning features according to claim 1, characterized in that: The image retrieval module calculates the similarity between the deep features of the images in the candidate similar image set and the deep features of the query image, and outputs the top-k images similar to the query image, including: a method of querying the index tree from top to bottom, S301, in each layer, by comparing with the color moment vector cf q The vector with the smallest distance locates the node, thus finding the entry point of the next layer; S302: After finding the entry point of the next layer, if it is not a leaf node, continue to iterate through these nodes; otherwise, obtain the candidate image data set and proceed to the next step; S303, calculate the depth feature vector f of each image in the candidate image dataset i and the feature vector f of the query image q The inner product value between them is D i,q =f i ·f q , and then sort the calculation results to get the top-k similar results.

4. A method for image retrieval based on color moment and deep learning features using the system described in any one of claims 1 to 3, characterized in that: The steps include: Calculate the value of each image m in the image data M i The deep learning feature vector of each image is obtained by dimensionality reduction. i ; Calculate the color moment of each image mi in the image data M to obtain a nine-dimensional color moment vector cfi; Cluster the color moments, take the obtained classes as leaf nodes, merge the nearest nodes and build an index tree from bottom to top; Input the query image mq and calculate its corresponding depth feature vector fq and color moment cfq; Traverse the index tree from top to bottom according to the color moment, select the child nodes that are closer to the color moment of the query image until the leaf node, and obtain the candidate similar image set; The similarity between the deep features of the images in the candidate similar image set and the deep features of the query image is calculated to obtain the top-k images similar to the query image.

Images