Image retrieval method, device and computer readable storage medium

Abstract

The present invention relates to the field of computer vision, and provides an image retrieval method, device and computer-readable storage medium to accurately and efficiently retrieve images with complex backgrounds and / or small ROIs. The method comprises: performing dimensionality reduction operations on each convolutional layer feature of the image to be retrieved to obtain each dimensionality reduction feature; clustering based on each dimensionality reduction feature to obtain multiple clustering features; performing feature fusion on multiple clustering features to obtain global features; and retrieving the image to be retrieved from a database based on the global features. On the one hand, the present invention performs dimensionality reduction operations on each convolutional layer feature of the image to be retrieved, so that the region of interest (ROI) can be changed into any shape, thereby reducing the influence of noise on the image retrieval results; on the other hand, clustering operations are performed based on each dimensionality reduction feature to obtain multiple clustering region features, thereby minimizing the influence of factors such as illumination, angle, and occlusion on image features.

Description

[0001] The present invention is a divisional application of the patent application with application number 201710891875.9, application date September 27, 2017, and application name “An image retrieval method, device and computer-readable storage medium”. Technical Field

[0002] The present invention belongs to the field of computer vision, and in particular relates to an image retrieval method, device and computer-readable storage medium. Background Art

[0003] Image retrieval refers to the task of returning an image that is closest to the image to be retrieved by searching the database. Image retrieval is an important problem in computer vision and image analysis. Image retrieval has some important applications in many fields, such as product recommendation, security monitoring, and medical diagnosis.

[0004] Although the industry has invested a lot of energy in studying image retrieval, it is very regrettable that image retrieval is still a complex and difficult problem. The retrieval accuracy and efficiency of image retrieval are affected by many factors. This is because, on the one hand, the complex image background often has serious interference with the region of interest, affecting the accuracy of the retrieval; on the other hand, in the large-scale image retrieval process, the images often reach millions or even larger, and image retrieval has very high real-time requirements, so the image retrieval algorithm must be very efficient and fast. However, many existing image retrieval algorithms often cannot accurately and efficiently extract effective retrieval solutions for images with complex backgrounds or small regions of interest (ROI), and thus cannot extract good features for retrieval.

[0005] The above technical problems need to be solved urgently by the industry. Summary of the invention

[0006] The present invention provides an image retrieval method, an apparatus and a computer-readable storage medium to accurately and efficiently retrieve images with complex backgrounds and / or small ROIs.

[0007] A first aspect of the present invention provides an image retrieval method, the method comprising:

[0008] Input the image to be retrieved into the convolutional neural network;

[0009] Processing the image to be retrieved by using the convolution layer and the downsampling layer of the convolutional neural network to obtain the features of each convolution layer of the image to be retrieved;

[0010] Performing dimensionality reduction operations on each convolutional layer feature of the image to be retrieved to obtain each reduced dimensionality feature, wherein the dimension of the reduced dimensionality feature is smaller than the dimension of the convolutional layer feature;

[0011] Perform clustering based on the dimensionality reduction features to obtain multiple clustering features;

[0012] Performing feature fusion on the multiple cluster features to obtain a global feature;

[0013] Based on the global features, the image to be retrieved is retrieved from a database.

[0014] In some embodiments, before inputting the image to be retrieved into the convolutional neural network, the method further includes: obtaining a pre-learned neural network; and removing the fully connected layer of the neural network to obtain the convolutional neural network.

[0015] In some embodiments, the obtaining of the pre-learned neural network includes: learning the neural network to be learned through an image classification task or a distance learning task to obtain the learned neural network.

[0016] In some embodiments, the parameters of the convolutional layer include weights and offsets of the convolution kernel.

[0017] In some embodiments, the dimension of the dimensionality reduction feature is determined according to the retrieval accuracy; and / or the number of the clustering features is determined according to the retrieval accuracy.

[0018] In some embodiments, each of the convolutional layer features is used to characterize the features of the corresponding pixel area in the image.

[0019] In some embodiments, clustering is performed based on the dimensionality reduction features to obtain multiple clustering features, including: clustering the dimensionality reduction features based on the distance between the dimensionality reduction features to obtain multiple feature clustering centers; for each feature clustering center of the multiple feature clustering centers, maximum value sampling is performed on the corresponding convolutional layer features to obtain the clustering features of each feature clustering center.

[0020] A second aspect of the present invention provides an image retrieval device, the device comprising:

[0021] An input module, used for inputting the image to be retrieved into the convolutional neural network, processing the image to be retrieved by using the convolutional layer and downsampling layer of the convolutional neural network, and obtaining the features of each convolutional layer of the image to be retrieved;

[0022] A dimension reduction module, used for performing a dimension reduction operation on each convolutional layer feature of the image to be retrieved, to obtain each reduced dimension feature, wherein the dimension of the reduced dimension feature is smaller than the dimension of the convolutional layer feature;

[0023] A clustering module, used for performing clustering based on the dimensionality reduction features to obtain multiple clustering features;

[0024] A fusion module, used for fusing the multiple cluster features to obtain a global feature;

[0025] The retrieval module is used to retrieve the image to be retrieved from the database based on the global feature.

[0026] In some embodiments, the device further includes: a learning module for acquiring a pre-learned neural network; and an obtaining module for removing the fully connected layer of the neural network to obtain the convolutional neural network.

[0027] In some embodiments, the learning module is further used to: learn the neural network to be learned through an image classification task or a distance learning task to obtain the learned neural network.

[0028] In some embodiments, the parameters of the convolutional layer include weights and offsets of the convolution kernel.

[0029] In some embodiments, the dimension of the dimensionality reduction feature is determined according to the retrieval accuracy; and / or the number of the clustering features is determined according to the retrieval accuracy.

[0030] In some embodiments, each of the convolutional layer features is used to characterize the features of the corresponding pixel area in the image.

[0031] In some embodiments, the clustering module includes: a first processing unit, used to cluster the dimensionality reduction features based on the distance between the dimensionality reduction features to obtain multiple feature clustering centers; a sampling unit, used to perform maximum value sampling on the corresponding convolutional layer features for each feature clustering center of the multiple feature clustering centers to obtain the clustering features of each feature clustering center.

[0032] A third aspect of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0033] Input the image to be retrieved into the convolutional neural network;

[0034] Processing the image to be retrieved by using the convolution layer and the downsampling layer of the convolutional neural network to obtain the features of each convolution layer of the image to be retrieved;

[0035] Performing dimensionality reduction operations on each convolutional layer feature of the image to be retrieved to obtain each reduced dimensionality feature, wherein the dimension of the reduced dimensionality feature is smaller than the dimension of the convolutional layer feature;

[0036] Perform clustering based on the dimensionality reduction features to obtain multiple clustering features;

[0037] Performing feature fusion on the multiple cluster features to obtain a global feature;

[0038] Based on the global features, the image to be retrieved is retrieved from a database.

[0039] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0040] Input the image to be retrieved into the convolutional neural network;

[0041] Processing the image to be retrieved by using the convolution layer and the downsampling layer of the convolutional neural network to obtain the features of each convolution layer of the image to be retrieved;

[0042] Performing dimensionality reduction operations on each convolutional layer feature of the image to be retrieved to obtain each reduced dimensionality feature, wherein the dimension of the reduced dimensionality feature is smaller than the dimension of the convolutional layer feature;

[0043] Perform clustering based on the dimensionality reduction features to obtain multiple clustering features;

[0044] Performing feature fusion on the multiple cluster features to obtain a global feature;

[0045] Based on the global features, the image to be retrieved is retrieved from a database.

[0046] From the technical solution provided by the present invention, it can be seen that, on the one hand, by performing dimensionality reduction operations on the features of each convolutional layer of the image to be retrieved, the region of interest (ROI) can be changed into any shape, thereby reducing the influence of noise on the image retrieval results; on the other hand, by performing clustering operations based on each dimensionality reduction feature, multiple clustering region features are obtained, thereby minimizing the influence of factors such as lighting, angle, and occlusion on image features. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0048] Figure 1-a It is a schematic diagram of the implementation process of the image retrieval method provided by an embodiment of the present invention;

[0049] Figure 1-b is a schematic diagram of an implementation flow of an image retrieval method provided by another embodiment of the present invention;

[0050] Figure 1-c is a schematic diagram of an implementation flow of an image retrieval method provided by another embodiment of the present invention;

[0051] Figure 2 is a schematic diagram of the structure of an image retrieval device provided by an embodiment of the present invention;

[0052] Figure 3 is a schematic structural diagram of an image retrieval device provided by another embodiment of the present invention;

[0053] Figure 4 is a schematic structural diagram of an image retrieval device provided by another embodiment of the present invention;

[0054] Figure 5 is a schematic structural diagram of an image retrieval device provided by another embodiment of the present invention;

[0055] Figure 6 is a schematic structural diagram of an image retrieval device provided by another embodiment of the present invention;

[0056] Figure 7 is a schematic structural diagram of an image retrieval device provided by another embodiment of the present invention;

[0057] Figure 8 is a schematic structural diagram of an image retrieval device provided by another embodiment of the present invention;

[0058] Figure 9-a is a schematic structural diagram of an image retrieval device provided by another embodiment of the present invention;

[0059] Figure 9-b is a schematic structural diagram of an image retrieval device provided by another embodiment of the present invention;

[0060] Fig.10 It is a schematic diagram of the structure of a terminal device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0061] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0062] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0063] Attached Figure 1-a 1 is a schematic diagram of the implementation flow of the image retrieval method provided by an embodiment of the present invention, which mainly includes the following steps Sa101 to Sa104, which are described in detail below:

[0064] Sa101, performs dimensionality reduction operations on each convolutional layer feature of the image to be retrieved, and obtains each reduced dimensionality feature, wherein the dimension of the reduced dimensionality feature is smaller than the dimension of the convolutional layer feature.

[0065] The dimensionality reduction operation is performed on each convolutional layer feature of the image to be retrieved, based on the following facts: 1. Multicollinearity - the correlation between predictors (for example, multicollinearity can lead to instability in the solution space, which may lead to incoherent results); 2. The high-dimensional space itself is sparse (research shows that 68% of the values ​​of a one-dimensional normal distribution fall between the positive and negative standard deviations, while only 0.02% in a ten-dimensional space); 3) Too many variables will hinder the establishment of a search pattern; 4) Analyzing only at the variable level may ignore the potential connection between variables (for example, several predictors may fall into a group that only reflects one aspect of the data); 5) The most valuable information is concentrated in the front part of the high-dimensional vector, while the back part is often noise; 6) After dimensionality reduction, it is more conducive to clustering features quickly and efficiently and reducing the impact of noise. Therefore, the purpose of dimensionality reduction operation includes reducing the number of predictors, ensuring that these variables are independent of each other, and providing a framework to explain the results, etc.

[0066] Sa102, clustering is performed based on each dimensionality reduction feature to obtain multiple clustering features.

[0067] Sa103, performs feature fusion on multiple clustering features to obtain global features.

[0068] Sa104, based on global features, retrieves the image to be retrieved from the database.

[0069] From the above attached Figure 1-a From the example image retrieval method, we can see that, on the one hand, by performing dimensionality reduction operations on the features of each convolutional layer of the image to be retrieved, the region of interest (ROI) can be transformed into any shape, reducing the impact of noise on the image retrieval results; on the other hand, by performing clustering operations based on each dimensionality reduction feature, multiple clustering region features are obtained, which minimizes the impact of factors such as lighting, angle, and occlusion on image features.

[0070] Attached Figure 1-b It is a schematic diagram of the implementation process of the image retrieval method provided by an embodiment of the present invention, which mainly includes the following steps Sb101 to Sb105, which are described in detail below.

[0071] Sb101, performing dimensionality reduction operations on each convolutional layer feature of the image to be retrieved, respectively, to obtain each reduced dimensionality feature, wherein the dimension of the reduced dimensionality feature is smaller than the dimension of the convolutional layer feature.

[0072] Before performing dimensionality reduction operations on the convolutional layer features of the image to be retrieved, the image to be retrieved is input into a convolutional neural network to obtain the features of each convolutional layer. Specifically, the convolutional neural network is obtained by learning the neural network, that is, the weights of the neural network are pre-learned through image classification tasks or distance learning tasks, and then the fully connected layer of the neural network structure is discarded, and only the parameters of its convolutional layer, including the weight and offset of the convolution kernel, are retained to obtain the convolutional neural network; then, the image to be retrieved is input into the convolutional neural network, so that it outputs the features of each convolutional layer of the image to be retrieved, wherein the convolutional neural network includes a convolutional layer and a downsampling layer, and each convolutional layer feature here is used to characterize the features of the corresponding pixel area in the image to be retrieved. For example, if the pixel area is in 8*8 pixel units, the number of convolutional layer features is (the number of pixels contained in the height of the image to be retrieved / 8)*(the number of pixels contained in the width of the image to be retrieved / 8); for another example, if the pixel area is in 16*16 pixel units, the number of convolutional layer features is (the number of pixels contained in the height of the image to be retrieved / 16)*(the number of pixels contained in the width of the image to be retrieved / 16), and so on.

[0073] It should be noted that the dimension of each dimensionality reduction feature in the above embodiment can be determined according to the accuracy of the retrieval. Generally speaking, the dimension of each convolutional layer feature before dimensionality reduction is much larger than the dimension of the dimensionality reduction feature. For example, the dimension of each convolutional layer feature before dimensionality reduction is 256 or 512, while the dimension of the dimensionality reduction feature can be 5.

[0074] It can be seen from the embodiments that, on the one hand, by extracting the features of each convolutional layer and utilizing the powerful expression ability of the convolutional layer features of the convolutional neural network, images with similar semantics are made similar in feature space, thereby reducing the influence of factors such as lighting, angle, and occlusion on image features, thereby improving the accuracy or quality of image retrieval; on the other hand, by performing dimensionality reduction operations on the convolutional layer features of the image to be retrieved, the ROI of the image to be retrieved can be changed into any shape, thereby reducing the influence of noise on the image retrieval results.

[0075] In an embodiment of the present invention, each convolutional layer feature of the image to be retrieved is a set of D-dimensional features of the image to be retrieved at H*W positions in the image space. If dimensionality reduction operations are performed on each convolutional layer feature of the image to be retrieved respectively, each reduced dimensionality feature can be obtained by reducing the D-dimensional features of the image to be retrieved at H*W positions in the image space to the C-dimensional features of the image to be retrieved at H*W positions in the image space. Obviously, C here is smaller than D, and H and W are the height and width of the image to be retrieved, respectively.

[0076] Sb102, clustering the dimension reduction features based on the distance between each dimension reduction feature to obtain multiple feature clustering centers.

[0077] Specifically, the dimension reduction features with comparable distances between the features in the dimension reduction features of the image to be retrieved are clustered into a feature clustering center, and finally a plurality of feature clustering centers are formed.

[0078] Sb103, for each of the multiple feature clustering centers obtained in step Sb102, maximum value sampling is performed on the corresponding convolutional layer features to obtain clustering features of each feature clustering center.

[0079] It should be noted that in the above step Sb103, the corresponding convolutional layer features refer to the convolutional layer features corresponding to each dimensionality reduction feature contained in each feature clustering center, and the number of clustering features can be 32. It can also take other values ​​according to the accuracy of the retrieval. This is just an example.

[0080] By performing clustering operations based on the dimensionality reduction features, multiple clustering region features are obtained, thereby minimizing the impact of factors such as lighting, angle, and occlusion on image features.

[0081] Sb104, performing feature fusion on multiple clustering features to obtain global features.

[0082] Specifically, feature fusion of multiple clustering features to obtain a global feature can be: respectively performing regression processing on each clustering feature of the multiple clustering features to obtain each regression clustering feature, and adding each regression clustering feature to obtain a global feature, wherein respectively performing regression processing on each clustering feature of the multiple clustering features to obtain each regression clustering feature includes respectively performing homogenization processing, whitening processing, and then homogenization processing on each clustering feature of the multiple clustering features in turn, so as to obtain each regression clustering feature.

[0083] Sb105, based on the global features, retrieve the image to be retrieved from the database.

[0084] As an embodiment of the present invention, based on the global features, retrieving the image to be retrieved from the database may be: based on the global features, obtaining K images with the highest similarity to the image to be retrieved from the database, where K is an integer greater than or equal to 1. Specifically,

[0085] The similarities between the images in the database and the images to be retrieved can be sorted according to the distance between the global features and the features of the images in the database, and the K images with the highest similarity to the images to be retrieved can be obtained.

[0086] Since the smaller the distance between the features of the image in the database and the global features, the more similar the image is to the image to be retrieved or the higher the similarity is, the above embodiment follows this principle to sort the similarities between the images in the database and the images to be retrieved, and obtains K images with the highest similarity to the images to be retrieved.

[0087] Furthermore, based on the global features, obtaining K images with the highest similarity to the image to be retrieved from the database may include: calculating the distance between the global features and the features of each image in the database; based on the distance between the global features and the features of each image in the database, obtaining K images with the shortest distance to the global features from the database as the K images with the highest similarity to the image to be retrieved.

[0088] After obtaining K images with the highest similarity to the image to be retrieved from the database, the method further includes displaying the K images and their respective similarities to the image to be retrieved.

[0089] From the image retrieval method illustrated in the above-mentioned Figure 1, it can be seen that, on the one hand, by performing dimensionality reduction operations on the features of each convolutional layer of the image to be retrieved, the region of interest (ROI) can be changed into any shape, thereby reducing the influence of noise on the image retrieval results; on the other hand, by performing clustering operations based on each dimensionality reduction feature, multiple clustering region features are obtained, thereby minimizing the influence of factors such as lighting, angle, and occlusion on image features.

[0090] Attached Figure 1-c Schematic diagram of the implementation flow of an image retrieval method provided by another embodiment of the present invention, which mainly includes the following steps Sc101 to Sc107, which are described in detail below:

[0091] Sc101, input the image to be retrieved into the convolutional neural network to obtain the features of each convolutional layer.

[0092] Specifically, a convolutional neural network is obtained by learning a neural network, that is, the weights of the neural network are pre-learned through an image classification task or a distance learning task, and then the fully connected layer of the neural network structure is discarded, and only the parameters of its convolutional layer, including the weight and offset of the convolution kernel, are retained to obtain a convolutional neural network; then, the image to be retrieved is input into the convolutional neural network, so that it outputs the features of each convolutional layer of the image to be retrieved, wherein the convolutional neural network includes a convolutional layer and a downsampling layer, and each convolutional layer feature here is used to characterize the features of the corresponding pixel area in the image to be retrieved. For example, if the pixel area is 8*8 pixel units, the number of convolutional layer features is (the number of pixels contained in the height of the image to be retrieved / 8)*(the number of pixels contained in the width of the image to be retrieved / 8); for another example, if the pixel area is 16*16 pixel units, the number of convolutional layer features is (the number of pixels contained in the height of the image to be retrieved / 16)*(the number of pixels contained in the width of the image to be retrieved / 16), and so on.

[0093] Sc102, performing dimensionality reduction operations on each convolutional layer feature of the image to be retrieved to obtain each reduced dimensionality feature, wherein the dimension of the reduced dimensionality feature is smaller than the dimension of the convolutional layer feature.

[0094] As an embodiment of the present invention, each convolutional layer feature of the image to be retrieved is subjected to dimensionality reduction operations respectively, and each dimensionality reduction feature can be obtained by using a principal component analysis (PCA) method for dimensionality reduction. The PCA method specifically replaces the n features before dimensionality reduction with a smaller number of m features, and the new features are linear combinations of the old features. These linear combinations maximize the sample variance, and try to make the new m features uncorrelated with each other, while the mapping from the old features to the new features captures the inherent variability in the data.

[0095] It should be noted that the dimensionality reduction operation performed on each convolutional layer feature of the image to be retrieved in the present invention is not limited to the above-mentioned PCA method, and other methods may also be used, such as factor analysis or user-defined compounding, etc. Therefore, the above-mentioned PCA method is not used to limit the present invention.

[0096] In an embodiment of the present invention, each convolutional layer feature of the image to be retrieved is a set of D-dimensional features of the image to be retrieved at H*W positions in the image space. If dimensionality reduction operations are performed on each convolutional layer feature of the image to be retrieved respectively, each reduced dimensionality feature can be obtained by reducing the D-dimensional features of the image to be retrieved at H*W positions in the image space to the C-dimensional features of the image to be retrieved at H*W positions in the image space. Obviously, C here is smaller than D, and H and W are the height and width of the image to be retrieved, respectively.

[0097] Sc103, based on the distance between each dimensionality reduction feature, the dimensionality reduction features are clustered to obtain multiple feature clustering centers.

[0098] Specifically, the dimension reduction features with comparable distances between the features in the dimension reduction features of the image to be retrieved are clustered into a feature clustering center, and finally a plurality of feature clustering centers are formed.

[0099] Sc104, for each of the multiple feature clustering centers obtained in step Sc103, perform maximum sampling on the corresponding convolutional layer features to obtain clustering features of each feature clustering center.

[0100] It should be noted that in the above step S c104, the corresponding convolutional layer features refer to the convolutional layer features corresponding to each dimensionality reduction feature in each dimensionality reduction feature contained in each feature cluster center, and the number of cluster features can be 32, and it can also take other values ​​according to the accuracy of the retrieval. This is just an example.

[0101] By performing clustering operations based on the dimensionality reduction features, multiple clustering region features are obtained, thereby minimizing the impact of factors such as lighting, angle, and occlusion on image features.

[0102] Sc105, performing regression processing on each of the multiple cluster features respectively to obtain each regression cluster feature.

[0103] Specifically, performing regression processing on each cluster feature in the multiple cluster features to obtain each regression cluster feature includes performing homogenization processing, whitening processing, and re-homogenization processing on each cluster feature in the multiple cluster features in sequence to obtain each regression cluster feature.

[0104] Sc106, add the features of each regression cluster to obtain the global feature.

[0105] Sc107, based on global features, retrieve the image to be retrieved from the database.

[0106] As an embodiment of the present invention, based on the global features, retrieving the image to be retrieved from the database may be: based on the global features, obtaining K images with the highest similarity to the image to be retrieved from the database, where K is an integer greater than or equal to 1. Specifically,

[0107] The similarities between the images in the database and the images to be retrieved can be sorted according to the distance between the global features and the features of the images in the database, and the K images with the highest similarity to the images to be retrieved can be obtained.

[0108] Since the smaller the distance between the features of the image in the database and the global features, the more similar the image is to the image to be retrieved or the higher the similarity is, the above embodiment follows this principle to sort the similarities between the images in the database and the images to be retrieved, and obtains K images with the highest similarity to the images to be retrieved.

[0109] Furthermore, based on the global features, obtaining K images with the highest similarity to the image to be retrieved from the database may include: calculating the distance between the global features and the features of each image in the database; based on the distance between the global features and the features of each image in the database, obtaining K images with the shortest distance to the global features from the database as the K images with the highest similarity to the image to be retrieved.

[0110] After obtaining K images with the highest similarity to the image to be retrieved from the database, the method further includes displaying the K images and their respective similarities to the image to be retrieved.

[0111] Figure 2 20 is a schematic diagram of an image retrieval device provided by an embodiment of the present invention, which mainly includes a dimension reduction module 201, a clustering module 202, a fusion module 203 and a retrieval module 204, which are described in detail as follows:

[0112] A dimension reduction module 201 is used to perform a dimension reduction operation on each convolutional layer feature of the image to be retrieved, to obtain each reduced dimension feature, wherein the dimension of the reduced dimension feature is smaller than the dimension of the convolutional layer feature, and the dimension of the reduced dimension feature may be 5;

[0113] A clustering module 202 is used to perform clustering based on each dimension reduction feature to obtain a plurality of clustering features, where the number of clustering features may be 32;

[0114] A fusion module 203 is used to fuse multiple cluster features to obtain a global feature;

[0115] The retrieval module 204 is used to retrieve the image to be retrieved from the database based on the global features.

[0116] It should be noted that the device provided in the embodiment of the present invention is based on the same concept as the embodiment of the method of the present invention, and the technical effect it brings is the same as the embodiment of the method of the present invention. The specific content can be found in the description in the embodiment of the method of the present invention, and will not be repeated here.

[0117] Attached Figure 2 The exemplary image retrieval device may further include an input module 301, as shown in the attached Figure 3 An exemplary image retrieval device: The input module 301 is used to input the image to be retrieved into the convolutional neural network to obtain the features of each convolutional layer, wherein each convolutional layer feature is used to represent the features of the corresponding pixel area in the image.

[0118] Attached Figure 2 The exemplary clustering module 201 may include a first processing unit 401 and a sampling unit 402, as shown in the attached Figure 4 An exemplary image retrieval apparatus, wherein:

[0119] The first processing unit 401 is used to cluster the dimension reduction features based on the distance between each dimension reduction feature to obtain multiple feature clustering centers;

[0120] The sampling unit 402 is used to perform maximum sampling on the corresponding convolution layer features for each feature clustering center of the multiple feature clustering centers, so as to obtain the clustering features of each feature clustering center.

[0121] Attached Figure 2 The exemplary fusion module 203 may include a regression processing unit 501 and an addition unit 502, as shown in the attached figure. Figure 5 An exemplary image retrieval apparatus, wherein:

[0122] The regression processing unit 501 is used to perform regression processing on each cluster feature of the multiple cluster features to obtain each regression cluster feature;

[0123] The adding unit 502 is used to add each regression clustering feature to obtain a global feature.

[0124] Attached Figure 5 The exemplary regression processing unit 501 may include a second processing unit 601, as shown in the attached Figure 6 The second processing unit 601 is used to perform a homogenization process, a whitening process, and a re-homogenization process on each of the plurality of cluster features in order to obtain each regression cluster feature.

[0125] Attached Figure 2 The example retrieval module 204 may include a first acquisition unit 701, as shown in the attached Figure 7 The first acquisition unit 701 of the exemplary image retrieval device is used to acquire K images having the highest similarity with the image to be retrieved from the database based on the global features, where K is an integer greater than or equal to 1.

[0126] Attached Figure 7 The first acquisition unit 701 of the example may include a calculation unit 801 and a second acquisition unit 802, as shown in the attached Figure 8 An exemplary image retrieval apparatus, wherein:

[0127] A calculation unit 801 is used to calculate the distance between the global feature and each image feature in the database;

[0128] The second acquisition unit 802 is used to acquire K images with the smallest distance to the global feature from the database based on the distance between the global feature and each image feature in the database, as the K images with the highest similarity to the image to be retrieved.

[0129] Attached Figure 7 or attached Figure 8 The exemplary image retrieval device may further include a display module 901, as shown in the attached Figure 9-a or attached Figure 9-b Example of an image retrieval device. The display module 901 is used to display the K images with the highest similarity to the image to be retrieved and their respective similarities to the image to be retrieved.

[0130] Fig.10 FIG. 1 is a schematic diagram of the structure of a terminal device provided by an embodiment of the present invention. Fig.10 As shown, the terminal device 10 of this embodiment includes: a processor 100, a memory 101, and a computer program 102 stored in the memory 101 and executable on the processor 100, such as a program of the image retrieval method. When the processor 100 executes the computer program 102, the steps in the above-mentioned image retrieval method embodiment are implemented, such as Figure 1-a Alternatively, when the processor 100 executes the computer program 102, the functions of each module / unit in the above-mentioned device embodiments are implemented, for example Figure 2 The functions of the dimension reduction module 201, the clustering module 202, the fusion module 203 and the retrieval module 204 are shown.

[0131] Exemplarily, the computer program 102 of the image retrieval method mainly includes: performing dimensionality reduction operations on each convolutional layer feature of the image to be retrieved, respectively, to obtain each dimensionality reduction feature, the dimension of the dimensionality reduction feature is smaller than the dimension of the convolutional layer feature; clustering based on the dimensionality reduction features to obtain multiple clustering features; performing feature fusion on the multiple clustering features to obtain global features; based on the global features, retrieving the image to be retrieved from the database. The computer program 102 can be divided into one or more modules / units, one or more modules / units are stored in the memory 101 and executed by the processor 100 to complete the present invention. One or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program 102 in the computing device 10. For example, the computer program 102 can be divided into the functions of a dimensionality reduction module 201, a clustering module 202, a fusion module 203 and a retrieval module 204 (modules in the virtual device), and the specific functions of each module are as follows: the dimensionality reduction module 201 is used to perform dimensionality reduction operations on each convolutional layer feature of the image to be retrieved, so as to obtain each reduced dimensionality feature, and the dimension of the reduced dimensionality feature is smaller than the dimension of the convolutional layer feature; the clustering module 202 is used to perform clustering based on the each reduced dimensionality feature to obtain multiple clustering features; the fusion module 203 is used to perform feature fusion on the multiple clustering features to obtain global features; the retrieval module 204 is used to retrieve the image to be retrieved from the database based on the global features.

[0132] The terminal device 10 may include but is not limited to a processor 100 and a memory 101. Those skilled in the art will appreciate that Fig.10 It is only an example of the terminal device 10 and does not constitute a limitation of the terminal device 10. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0133] The processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0134] The memory 101 may be an internal storage unit of the terminal device 10, such as a hard disk or memory of the terminal device 10. The memory 101 may also be an external storage device of the terminal device 10, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 10. Further, the memory 101 may also include both an internal storage unit of the terminal device 10 and an external storage device. The memory 101 is used to store computer programs and other programs and data required by the terminal device. The memory 101 may also be used to temporarily store data that has been output or is to be output.

[0135] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0136] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0137] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0138] In the embodiments provided by the present invention, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0139] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0140] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0141] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program of the image retrieval method can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments, that is, perform dimensionality reduction operations on each convolutional layer feature of the image to be retrieved, respectively, to obtain each dimensionality reduction feature, and the dimension of the dimensionality reduction feature is smaller than the dimension of the convolutional layer feature; clustering is performed based on the dimensionality reduction features to obtain multiple clustering features; feature fusion is performed on the multiple clustering features to obtain global features; based on the global features, the image to be retrieved is retrieved from the database. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form. Computer-readable media may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memories, read-only memories (ROM), random access memories (RAM), electric carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals. The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. An image retrieval method, characterized in that: The method comprises: Input the image to be retrieved into the convolutional neural network; Processing the image to be retrieved by using the convolution layer and the downsampling layer of the convolutional neural network to obtain the features of each convolution layer of the image to be retrieved; Performing dimensionality reduction operations on each convolutional layer feature of the image to be retrieved to obtain each reduced dimensionality feature, wherein the dimension of the reduced dimensionality feature is smaller than the dimension of the convolutional layer feature; Perform clustering based on the dimensionality reduction features to obtain multiple clustering features; Performing feature fusion on the multiple cluster features to obtain a global feature; Based on the global features, the image to be retrieved is retrieved from a database.

2. The method according to claim 1, characterized in that Before inputting the image to be retrieved into the convolutional neural network, the method further includes: Get a pre-learned neural network; The fully connected layer of the neural network is removed to obtain the convolutional neural network.

3. The method according to claim 2, characterized in that The step of obtaining a pre-learned neural network comprises: The neural network to be learned is learned through an image classification task or a distance learning task to obtain the learned neural network.

4. The method according to any one of claims 1 to 3, characterized in that The parameters of the convolution layer include the weight and offset of the convolution kernel.

5. The method according to any one of claims 1 to 3, characterized in that The dimension of the dimension reduction feature is determined according to the retrieval accuracy; And / or, the number of clustering features is determined according to retrieval accuracy.

6. The method according to any one of claims 1 to 3, characterized in that Each of the convolutional layer features is used to characterize the features of the corresponding pixel area in the image.

7. The method according to any one of claims 1 to 3, characterized in that The clustering is performed based on the dimensionality reduction features to obtain multiple clustering features, including: Based on the distances between the dimensionality reduction features, the dimensionality reduction features are clustered to obtain a plurality of feature clustering centers; For each of the multiple feature clustering centers, maximum value sampling is performed on the corresponding convolutional layer features to obtain the clustering features of each feature clustering center.

8. An image retrieval device, characterized in that: The device comprises: An input module, used for inputting the image to be retrieved into the convolutional neural network, processing the image to be retrieved by using the convolutional layer and downsampling layer of the convolutional neural network, and obtaining the features of each convolutional layer of the image to be retrieved; A dimension reduction module, used for performing a dimension reduction operation on each convolutional layer feature of the image to be retrieved, to obtain each reduced dimension feature, wherein the dimension of the reduced dimension feature is smaller than the dimension of the convolutional layer feature; A clustering module, used for performing clustering based on the dimensionality reduction features to obtain multiple clustering features; A fusion module, used for fusing the multiple cluster features to obtain a global feature; The retrieval module is used to retrieve the image to be retrieved from the database based on the global feature.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images