An image retrieval method based on the combination of a deep convolutional neural network and a locality sensitive hashing algorithm

Abstract

The present application relates to the technical field of image retrieval, and particularly to an image retrieval method based on a combination of a deep convolutional neural network and a local sensitive hashing algorithm, which has the following steps: step S1: training set and validation set in an open source data set of image retrieval; step S2: input of the model during training; step S3: test retrieval ranking; step S4: the loss function of image retrieval adopts a contrast loss function, and in addition to mAP, the model evaluation index also newly adds mP@k; the present method is based on a combination of a deep convolutional neural network and a local sensitive hashing algorithm, the algorithm extracts image features of a gallery library and a query library in a deep convolutional manner, carries out LSH hash coding, greatly improves the retrieval performance, and uses contrast learning in a twin network, thereby greatly improving the retrieval precision.

Description

Technical Field

[0001] This invention relates to the field of image retrieval technology, specifically to an image retrieval method based on a combination of deep convolutional neural networks and locality-sensitive hashing algorithm. Background Technology

[0002] Image search, also known as image retrieval, is a process that takes an input image (multiple images can be stored in a query database) and uses computer vision and other technologies to extract features from the input image and match them with those features in the database to quickly and accurately find similar or identical images. Currently, there are two main types of image content-based retrieval technologies that are truly applicable: one is based on image features (such as color and texture) to distinguish different images, converting images into color histograms, and using similarity metrics (such as Euclidean distance and cosine similarity) to calculate the similarity between images. This method is simple to implement, but for complex images, it may not be able to capture higher-level semantic features, thus limiting its retrieval effectiveness. The other type is deep learning-based retrieval technology: using deep learning models such as convolutional neural networks (CNNs) for feature extraction and matching. This method can effectively extract high-level semantic features and can achieve better performance through end-to-end training on large-scale datasets. However, it requires a large amount of data and computing resources and performs poorly on small datasets. The former has limited feature extraction capabilities and can only be used for simple image comparison, making it unsuitable for real-world applications. The latter relies on a large amount of data, and its accuracy cannot be guaranteed.

[0003] In summary, this invention addresses the problem of simultaneously failing to guarantee both speed and accuracy by designing an image retrieval method based on a combination of deep convolutional neural networks and locality-sensitive hashing. Summary of the Invention

[0004] To overcome the problems of retrieving complex images and accuracy, the present invention aims to provide an image retrieval method based on a combination of deep convolutional neural networks and locality-sensitive hashing algorithm. This method removes redundant features and optimizes the loss function, thereby improving both accuracy and speed, and solving the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] An image retrieval method based on a combination of deep convolutional neural networks and locality-sensitive hashing algorithm, the specific steps of which are as follows:

[0007] Step S1: The training and validation sets of the open-source image retrieval dataset use retrieval-SfM-120k, while the test sets use Oxford5k, Paris6k, ROxford5k, and RParis6k. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above datasets. After preprocessing the data, the image input size is 1024*1024*3. Use a deep convolutional neural network ResNet101 to extract RGB image features. Features need to be extracted from images involving both the gallery and query libraries. For subsequent retrieval comparison, the difference between extracting features from the gallery and query libraries is whether to return LSH encoding. The Locality Sensitive Hash algorithm (LSH) is used to encode each image feature map with 0 and 1 binary values ​​to speed up the matching speed between the query image and the image search library.

[0008] Step S2: During training, the model's input consists of images in the training set transformed into feature vectors by the model. From these vectors, a QSIZE (QP pairs) of tuples is selected. Each tuple contains 1+1+NNUM feature vectors, representing the query object Q, the positive class P, and NNUM negative classes N1, N2, ... The query and positive classes are directly provided by the QP pairs; the negative classes are determined by the current model's query results in the image pool, selecting NNUM images from top to bottom that are in different clusters from Q, and these NNUM images are also in different clusters. The model's output is a matrix composed of the vector features of each tuple processed by the model.

[0009] During testing, the model's input consists of images from the image library in the test set and images of the query objects. The model's output consists of the feature matrix of the query objects (i.e., a matrix composed of the feature vectors of all query objects) and the feature matrix of the image library (i.e., a matrix composed of the feature vectors of all image library images).

[0010] Step S3: Test retrieval ranking: The dot product of the feature matrix of the image library and the feature matrix of the query object yields the scores matrix, which has the dimension of: number of images in the image library * number of queries. The i-th row and j-th column represent the similarity score between the i-th image in the image pool and the j-th query object. The ranks are the column-sorted index values ​​of the scores, meaning that the index of the image with the higher score is ranked first. This is the final retrieval result.

[0011] Step S4: The loss function for image retrieval adopts the contrastive loss function. In addition to mAP, a new evaluation metric, mP@k, is added, which is the accuracy metric of the top-k retrieval results in the result list, reflecting the quality of the image search engine. The higher the ranking of the matched images, the higher the score, and the lower the ranking of the unmatched images, the higher the score.

[0012] As a preferred embodiment of this method, the RGB image dimension in S1 is W*H*3, where H and W are the height and width of the feature map, respectively, and 3 is the number of channels.

[0013] In the preferred embodiment of this method, the image input in S2 is 1024*1024*3. The image passes through a convolutional layer, i.e., a convolutional neural network, such as ResNet101 with the last layer removed (i.e., a fully connected layer), followed by a GeM pooling layer and L2 normalization (vector normalization), ultimately forming a fixed-dimensional vector representation of the image, i.e., high-dimensional features. Features need to be extracted from images in both the gallery and query libraries. For subsequent retrieval and comparison, the difference between extracting features from the gallery and query libraries is whether LSH encoding is returned. The Locality Sensitive Hash (LSH) algorithm is used to encode each image feature map using 0s and 1s, aiming to accelerate the matching speed between the query image and the image search library. GeM pooling, or generalized average pooling, can improve the quality of the output descriptor.

[0014] GeM pooling:

[0015] In Formula 2, when P k →∞ Formula (2) is max pooling, when P k =1 Formula (2) is average pooling, combined with attention mechanism.

[0016] As a preferred embodiment of this method, the specific steps of the retrieval process are as follows:

[0017] First, the images in the image pool of the retrieval database are converted into column vector features. Multiple column vector features are then concatenated to form a matrix. A Locality Sensitive Hash (LSH) algorithm is then introduced to encode the images in the retrieval database, aiming to improve retrieval efficiency. The LSH algorithm is shown below:

[0018] S1 - Initialization: Randomly generate k random vectors as the initial hash table;

[0019] S2 - Calculate the hash: For each query vector x, hash it into a hash value h(x) and map h(x) to the corresponding slot in the hash table;

[0020] S3 - Expand Slots: If the number of elements in the current slot exceeds m, slot expansion is required. Specifically, p elements are randomly selected from all elements and added to the current slot.

[0021] S4 - Calculate Similarity: For each vector y to be searched, calculate its similarity to the elements in all slots. Commonly used similarity measures include Euclidean distance and cosine similarity.

[0022] S5 - Matching Results: Based on the similarity ranking of each slot, the top k slots are selected as possible matching positions; then for each position, the hash value of the corresponding object is calculated and compared with the query vector to find the k closest objects;

[0023] In the hash calculation process in S2, numbering each image feature map with 0s and 1s means converting the value of each pixel, usually a grayscale value, into a binary number (0 or 1) and using it as input to the hash function. The goal is to represent each pixel in the image as a fixed-length binary vector, so that objects with similar attributes are mapped to adjacent positions in the hash space. This method can improve the efficiency and accuracy of the hash algorithm because it allows for more compact encoding of the image and can find the image most similar to the query image in a relatively short time.

[0024] Secondly, the query objects are converted into column vector features. If multiple query objects are queried at the same time, their column vector features are concatenated into a matrix.

[0025] Finally, the feature matrix of the image pool is transposed and compared with the vector features of the query object to obtain the cosine similarity. In this result, the element in the i-th row and j-th column represents the similarity between the image in the i-th image pool and the j-th query object.

[0026] As a preferred embodiment of this method, the contrastive loss function in S4 is as follows:

[0027]

[0028] Where d represents the distance between two vectors, typically Euclidean distance; y represents whether two inputs are similar, with 1 if similar and 0 if dissimilar; margin is a predefined threshold, indicating that the two samples are dissimilar when the vector distance between them exceeds a certain value. From formula (1), if the two inputs are similar, i.e., y = 1, then only d remains in the formula. 2 This aligns with conventional understanding: if two inputs are similar, the greater the distance between their vectors, the greater the loss. If the two inputs are dissimilar, i.e., y = 0, then only max(margin - d, 0) remains in the equation. 2 That is, when two inputs are dissimilar, if the distance between the vectors is greater than the margin, the loss is 0; if the distance between the vectors is less than the margin, the smaller the distance, the greater the loss.

[0029] Compared with the prior art, the beneficial effects of the present invention are:

[0030] In this invention, an image retrieval method based on a combination of deep convolutional neural networks and locality-sensitive hashing (LSH) is designed. This algorithm uses deep convolution to extract image features from the gallery and query libraries and performs LSH hashing, which greatly improves retrieval performance. Furthermore, the use of contrastive learning in Siamese networks significantly improves retrieval accuracy. Attached Figure Description

[0031] Figure 1 This is a network architecture diagram of the present invention. Detailed Implementation

[0032] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0033] Please see Figure 1 The present invention provides a technical solution:

[0034] An image retrieval method based on a combination of deep convolutional neural networks and locality-sensitive hashing algorithm includes the following steps:

[0035] Step S1: The training and validation sets in the open-source image retrieval dataset are retrieval-SfM-120k, while the test sets are Oxford5k, Paris6k, ROxford5k, and RParis6k. If you need to train on your own dataset, you need to prepare the data format to be consistent with the above datasets. After preprocessing the data, the image input size is 1024*1024*3, and the RGB image dimension is W*H*3, where H and W are the height and width of the feature map, respectively, and 3 is the number of channels. The deep convolutional neural network ResNet101 is used to extract RGB image features. The image passes through a convolutional layer (i.e., a convolutional neural network, such as ResNet101 with the last fully connected layer removed), then through a GeM pooling layer with an added attention mechanism and L2 normalization (i.e., vector normalization), finally forming a fixed-dimensional vector representation of the image, i.e., high-dimensional features. Images from both the gallery and query libraries require feature extraction. For subsequent retrieval and comparison, the difference between feature extraction from the gallery and query libraries lies in whether LSH encoding is returned. The Locality Sensitive Hash (LSH) algorithm is used to encode each image's feature map using 0s and 1s, aiming to accelerate the matching speed between the query image and the image search library. GeM pooling, or Generalized Average Pooling, can improve the quality of the output descriptors.

[0036] GeM pooling:

[0037] In Formula 2, when P k →∞ Formula (2) is max pooling, when P k =1 Formula (2) is average pooling. Combined with the attention mechanism, the effect is as follows: it focuses more on salient features:

[0038] like Figure 1 Network architecture diagram: Add 3 attention units to ResNet-101.

[0039] The Att1 unit consists of four convolutional layers with kernel sizes of 3*3, 3*1, 1*1, and 1*1. The first layer has a stride of 2, and the others have a stride of 1. The output channels are 1024, 512, 512, and 2048, respectively. In addition, each convolutional layer is followed by BN and ReLU activation. Except for the last layer which uses the sigmoid function, Att2 and Att3 each contain only one convolutional layer with a kernel size of 1*1 and a stride of 1. The output channel size is the same as the input channel size. Then, there is sigmoid activation, followed by GeM pooling and L2 normalization, finally forming a 2048-dimensional vector as the descriptor of the input image.

[0040] Step S2: During training, the model's input consists of images from the training set transformed into feature vectors. From these, QSIZE (the number of QP pairs) tuples are selected. Each tuple contains (1+1+NNUM) feature vectors, representing the query object Q, the positive class P, and NNUM negative classes N1, N2, ... The query and positive classes are directly provided by the QP pairs. The negative classes Q are selected from the current model's query results in the image pool, sequentially from top to bottom, choosing NNUM images that are in different clusters from q, and these NNUM images are also in different clusters. The model's output is a matrix composed of the vector features of each tuple processed by the model. During testing, the model's input consists of images from the image library in the test set and images of the query object. The model's output is the feature matrix of the query object (a matrix composed of all feature vectors of the query object) and the feature matrix of the image library (a matrix composed of all feature vectors of the image library).

[0041] Step S3: Test retrieval ranking: The dot product of the image feature matrix and the query object feature matrix yields the scores matrix (dimension: number of images in the image library * number of queries), where the i-th row and j-th column represent the similarity score between the i-th image in the image pool and the j-th query object. ranks is the column-sorted index value of scores, meaning images with higher scores are ranked first, representing the final retrieval result.

[0042] The specific steps of the retrieval process are as follows:

[0043] First, the images in the image pool of the retrieval database are converted into column vector features. Multiple column vector features are then concatenated to form a matrix. A Locality Sensitive Hash (LSH) algorithm is then introduced to encode the images in the retrieval database, aiming to improve retrieval efficiency. The LSH algorithm is shown below:

[0044] S1 - Initialization: Randomly generate k random vectors as the initial hash table.

[0045] S2 - Calculate the hash: For each query vector x, hash it into a hash value h(x) and map h(x) to the corresponding slot in the hash table.

[0046] S3 - Expand Slots: If the number of elements in the current slot exceeds m, slot expansion is required. Specifically, p elements are randomly selected from all elements and added to the current slot.

[0047] S4 - Calculate Similarity: For each vector y to be searched, calculate its similarity to the elements in all slots. Commonly used similarity measures include Euclidean distance and cosine similarity.

[0048] S5 - Matching Results: Based on the similarity ranking of each slot, the top k slots are selected as potential matching positions. Then, for each position, the hash value of the corresponding object is calculated and compared with the query vector to find the k closest objects.

[0049] In the hash calculation process in S2, numbering each image feature map with 0s and 1s means converting the value of each pixel (usually a grayscale value) into a binary number (0 or 1) and using it as input to the hash function. The goal is to represent each pixel in the image as a fixed-length binary vector, allowing objects with similar attributes to be mapped to adjacent positions in the hash space. This method improves the efficiency and accuracy of the hash algorithm because it allows for more compact encoding of the image and can find the most similar image to the query image in a relatively short time.

[0050] Secondly, the query objects are converted into column vector features. If multiple query objects are queried at the same time, their column vector features are concatenated into a matrix.

[0051] Finally, the feature matrix of the image pool is transposed and compared with the vector features of the query object (i.e., cosine similarity is calculated) to obtain the similarity result. In this result, the element in the i-th row and j-th column represents the similarity between the image in the i-th image pool and the j-th query object.

[0052] Step S4: The image retrieval loss function adopts the contrastive loss function. In addition to mAP, a new evaluation metric, mP@k, is added, which is the accuracy metric for the top-k retrieval results in the result list, reflecting the quality of the image search engine. Matching images ranked higher receive higher scores, while non-matching images ranked lower receive higher scores. The contrastive loss function is as follows:

[0053]

[0054] Where d represents the distance between two vectors, typically Euclidean distance; y represents whether two inputs are similar, with 1 if similar and 0 if dissimilar; margin is a predefined threshold, indicating that the two samples are dissimilar when the vector distance between them exceeds a certain value. From formula (1), if the two inputs are similar (i.e., y = 1), only d remains in the formula. 2 This aligns with conventional understanding: if two inputs are similar, the greater the distance between their vectors, the greater the loss. If two inputs are dissimilar (i.e., y = 0), then only max(margin - d, 0) remains in the equation. 2 That is, when two inputs are dissimilar, if the distance between the vectors is greater than the margin, the loss is 0; if the distance between the vectors is less than the margin, the smaller the distance, the greater the loss.

[0055] Example:

[0056] The specific steps of the algorithm are as follows:

[0057] The input image is 1024*1024*3. It passes through a convolutional layer (i.e., a convolutional neural network, such as ResNet with the last fully connected layer removed), then through a GeM pooling layer and L2 normalization (vector normalization), ultimately forming a fixed-dimensional vector representation of the image. Using the LSH algorithm, the feature maps of each image are numbered with 0s and 1s, and all images are feature-encoded. Since the size of each image can be different, the ResNet101 network outputs a 1*2048 vector through a fully connected layer. Therefore, each feature map is encoded as a 1*2048 vector. Simultaneously, the retrieved image is feature-encoded and then used for image retrieval. The feature matrix of the gallery is transposed and compared with the vector features of the retrieved object (i.e., cosine similarity is calculated) to obtain the similarity result. In this result, the element in the i-th row and j-th column represents the similarity between the image in the i-th image pool and the j-th query object.

[0058] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An image retrieval method based on a combination of deep convolutional neural networks and locality-sensitive hashing algorithm, the specific steps of which are as follows: Step S1: The training and validation sets for the open-source image retrieval dataset are retrieval-SfM-120k, while the test sets are Oxford5k, Paris6k, ROxford5k, and RParis6k. If training is required on the dataset, the data format should be prepared to be consistent with the above datasets. After preprocessing the data, the image input size is 1024*1024*3. The deep convolutional neural network ResNet101 is used to extract RGB image features. Features need to be extracted from images involving both the image library and the query library. The difference between extracting features from the image library and the query library is whether to return LSH encoding. The Locality Sensitive Hash algorithm (LSH) is used to encode each image feature map in binary format (0 and 1). The image library is the gallery library, and the query library is the query library. Step S2: During training, the model input is the images in the training set, which are transformed into feature vectors by the model. QSIZE, which is the number of QP pairs, is selected from these vectors. Each tuple contains 1+1+NNUM feature vectors, which are the query object Q, the positive class P, and NNUM negative classes N1, N2, ... The query and positive classes are directly given by the QP pairs. The negative class Q is the result of the current model's query in the image pool. NNUM images that are in different clusters from q are selected sequentially from top to bottom according to the query order. The model output is a matrix composed of the vector features of each tuple. During testing, the model's input consists of images from the image library in the test set and images of the query object. The output of the test model is: the feature matrix of the query object, which is a matrix composed of the feature vectors of all query objects, and the feature matrix of the image library, which is a matrix composed of the feature vectors of all images in the image library. Step S3: Test retrieval ranking: The dot product of the image feature matrix and the query object feature matrix yields the scores matrix, which has the dimension of: number of images in the image library * number of queries. The i-th row and j-th column represent the similarity score between the i-th image in the image pool and the j-th query object. The ranks are the column-sorted index values ​​of the scores, meaning that images with higher scores are ranked first, which is the final retrieval result. Step S4: The loss function for image retrieval is the contrastive loss function. The model evaluation metrics include mAP and mP@k, where the accuracy of the top-k retrieval results in the result list reflects the quality of the image search engine. The higher the ranking of a matching image, the higher its score; the lower the ranking of a non-matching image, the higher its score. The contrastive loss function in S4 is as follows: Where d represents the distance between two vectors; y represents whether two inputs are similar, with 1 if they are similar and 0 if they are not; margin is a threshold, which indicates that the two samples are not similar when the vector distance between them exceeds a certain threshold. From formula (1), if the two inputs are similar, i.e., y = 1, then only d remains in the formula. 2 ; If two inputs are similar, the greater the distance between their vectors, the greater the loss; if two inputs are dissimilar, i.e., y = 0, then the loss is max(margin - d, 0). 2 That is, when two inputs are dissimilar, if the distance between the vectors is greater than the threshold, the loss is 0; if the distance between the vectors is less than the threshold, the smaller the distance, the greater the loss.

2. The image retrieval method based on a combination of deep convolutional neural networks and locality-sensitive hashing algorithm as described in claim 1, characterized in that, The RGB image dimension in S1 is W*H*3, where H and W are the height and width of the feature map, respectively, and 3 is the number of channels.

3. The image retrieval method based on a combination of deep convolutional neural networks and locality-sensitive hashing algorithm as described in claim 1, characterized in that, In S2, the image input is 1024*1024*3, and the image passes through a convolutional layer, i.e., a convolutional neural network, specifically: ResNet101, after removing the last layer (fully connected layer), then passing through GeM pooling layers and L2 normalization (vector normalization), ultimately forms a fixed-dimensional vector representation of the image, i.e., high-dimensional features; GeM pooling... M-pooling, or generalized average pooling, improves the quality of the output descriptor. GeM pooling: In formula (2), when P k →∞ Formula (2) is max pooling, when P k =1 Formula (2) is average pooling, combined with attention mechanism.

4. The image retrieval method based on a combination of deep convolutional neural networks and locality-sensitive hashing algorithm as described in claim 1, characterized in that, The specific steps of the retrieval process are as follows: First, the images in the image pool of the retrieval database are converted into column vector features. Multiple column vector features are then concatenated to form a matrix. A Locality Sensitive Hash (LSH) algorithm is then introduced to encode the images in the retrieval database. The LSH algorithm is shown below: S1 - Initialization: Randomly generate k random vectors as the initial hash table; S2 - Calculate the hash: For each query vector x, hash it into a hash value h(x) and map h(x) to the corresponding slot in the hash table; S3 - Expand Slots: If the number of elements in the current slot exceeds m, then slot expansion is required. Specifically, p elements are randomly selected from all elements and added to the current slot. S4 - Calculate similarity: For each vector y to be searched, calculate its similarity with the elements in all slots. Similarity measurement methods include Euclidean distance and cosine similarity. S5 - Matching Results: Based on the similarity ranking of each slot, the top k slots are selected as the matching positions; Then, for each location, the hash value of the corresponding object is calculated and compared with the query vector to obtain the k closest objects; In the hash calculation process in S2, numbering each image feature map with 0 and 1 means converting the value of each pixel, i.e. the gray value, into a binary number 0 or 1, and using it as the input of the hash function. Specifically, each pixel in the image is represented as a fixed-length binary vector, so that objects with similar attributes are mapped to adjacent positions in the hash space. Secondly, the query objects are converted into column vector features. If multiple query objects are queried at the same time, their column vector features are concatenated into a matrix. Finally, the feature matrix of the image pool is transposed and compared with the vector features of the query object to obtain the cosine similarity result. In the result, the element in the i-th row and j-th column represents the similarity between the image in the i-th image pool and the j-th query object.

Images