Image retrieval method, storage medium and product based on layered hash floating point features

Abstract

The present invention discloses an image retrieval method, storage medium, and product based on hierarchical hash floating-point features, which are applied to the field of image retrieval, including: constructing and training a deep floating-point feature network model; constructing a deep hierarchical hash-floating-point feature network model, and using a knowledge distillation method to train a deep hierarchical hash-floating-point feature network model using the trained deep floating-point feature network model; using the deep hierarchical hash-floating-point feature network model to extract N layers of deep hash features and deep floating-point features and establish a tree index; and in the image retrieval process, narrowing the retrieval range layer by layer according to a coarse-to-fine hierarchical retrieval from low-dimensional deep hash features, high-dimensional deep hash features, and then to deep floating-point features, and outputting the retrieval results. By constructing hierarchical deep hash-floating-point features and combining them with a coarse-to-fine hierarchical retrieval, the present invention effectively improves the efficiency of image retrieval while ensuring retrieval accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image retrieval technology, and more specifically to an image retrieval method, storage medium, and product based on knowledge distillation layered deep hash-floating point features. Background Art

[0002] Video images often contain a wealth of objective information. Content-based image retrieval allows users to input a single image and search for other images with the same or similar content. However, as the size of image and video libraries continues to grow, content-based image retrieval methods are becoming increasingly slow, significantly hindering the application of image retrieval technology.

[0003] Content-based image retrieval is a key topic in information retrieval and computer vision. When building an image database, a content-based image retrieval system analyzes and classifies input images into a unified model. It then extracts image features based on various image models and stores them in a feature library. This library is then indexed to improve search efficiency. The feature extractor must ensure that features of images with similar content have close feature vectors. Distance metrics can include Euclidean distance, Hamming distance, cosine distance, and other distance metrics.

[0004] When searching for similar vectors, the features extracted from images are typically high-dimensional. Directly measuring similarity is slow due to the use of distance metrics. Therefore, heuristic search algorithms, or approximate nearest neighbor (ANN) search algorithms, are often used to find similar vectors. These include hashing, quantization, and graph search. There are three main approaches for extracting feature vectors: 1) Local descriptor-based methods, such as SIFT descriptors. However, due to the variable number of SIFT descriptors generated for a single image, these methods are less commonly used for retrieval. 2) Global descriptor-based methods, such as Fisher and Gist, fuse local descriptors or directly extract global descriptors. These methods were commonly used before breakthroughs in neural network-based algorithms. 3) Convolutional neural network-based methods can extract high-level descriptors that contain more semantic information than lower-level features. They are also invariant to scale, illumination, and rotation, resulting in greater robustness and, therefore, better performance in practice.

[0005] Because feature vectors are high-dimensional and sparse, directly searching for their nearest neighbor vectors is inefficient. Therefore, approximate nearest neighbor search is often used to obtain similar feature vectors. Approximate nearest neighbor methods can be categorized into three main types: 1) data-independent algorithms, such as Locality Sensitive Hashing (LSH); 2) unsupervised data-dependent algorithms, such as Productive Quantization (PQ); and 3) supervised data-dependent algorithms, such as Deep Hashing (CNN). Deep hashing, which currently offers outstanding retrieval accuracy, uses convolutional neural networks to extract feature vectors from images and simultaneously generate image indexes. Feature vectors extracted by CNNs provide richer high-level semantic information than other low-level descriptors, allowing them to perform similarity indexing at a wider range of semantic levels than other methods. By adding hidden layers to a CNN and training it with a Fine-Tune network, feature vectors are extracted from images and binary hash codes are directly generated for image retrieval. This combines the two sub-problems of vector extraction and retrieval. To effectively utilize the large amount of manually extracted annotation information, the deep hashing algorithm can use supervised learning. Based on the image label information, the deep hashing algorithm calculates the hash value and the distance between similar and dissimilar images input to the network, and adds the hash value to the loss function. Through backpropagation of the network weights, the deep hashing algorithm can learn the information of similarity or dissimilarity of images through supervision.

[0006] However, current deep hashing algorithms employ a single-layer structure and linear indexing and search methods, making them inefficient for retrieval of large-scale image datasets. Deep hashing algorithms also use a deep convolutional neural network to extract image features. These extracted feature vectors are then compressed, reduced in dimension, and binarized to produce a binary vector. This compression and dimensionality reduction process inevitably results in information loss, leading to a reduction in retrieval accuracy. Each time a query is performed for similar images, each feature vector must be compared. If the image database size is N, the algorithmic complexity for querying a single image is O(N), resulting in performance that degrades as the dataset grows. For larger databases, such as those containing one million images, image retrieval currently takes thousands of times longer than the time required to process and generate the hash value. Consequently, existing deep hashing retrieval algorithms struggle to achieve satisfactory search speed and accuracy.

[0007] Therefore, how to improve the accuracy and efficiency of image search is an urgent problem that those skilled in the art need to solve. Summary of the Invention

[0008] In view of this, the present invention provides an image retrieval method, storage medium and product based on layered hash floating-point features, which can improve the image retrieval speed while ensuring that the retrieval accuracy is not reduced, and meet the real-time requirements of tasks such as pedestrian re-identification in large-scale spatiotemporal scenes.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] The image retrieval method based on hierarchical hash floating point features includes the following steps:

[0011] Step 1: Obtain a training sample image, build a deep floating-point feature network model, and use the training sample to train the deep floating-point feature network model to obtain the trained deep floating-point feature network model and the deep floating-point features of the training sample image;

[0012] Step 2: Build a deep layered hash-floating-point feature network model based on the trained deep floating-point feature network model, and use the deep floating-point features of the training sample images for training, to obtain the trained deep layered hash-floating-point feature network model and N layers of deep hash features of the training sample images, N ≥ 2, and each layer of deep hash features has different dimensions;

[0013] Step 3: Input the images in the image library to be retrieved into the deep hierarchical hash-floating point feature network model to obtain the image features of each image, including N layers of deep hash features and deep floating point features, and construct a hierarchical tree index of the image features of the image library to be retrieved;

[0014] Step 4: Input the image to be retrieved into the deep hierarchical hash-floating point feature network model to obtain its N-layer deep hash features and deep floating point features, and perform a level-by-level search in the hierarchical tree index of image features in the image library to output the retrieval results.

[0015] The dimensions of multiple deep hash features are different. For example, the first layer is a 128-dimensional deep hash feature, and the second layer is a 256-dimensional deep hash feature. Both layers of deep hash features can be used to uniquely represent images. During the retrieval process, the first layer of deep hash features are first used for comparison and sorting. Similar images are retained and then the second layer of deep hash features are used for comparison and sorting, and so on.

[0016] Preferably, the deep floating-point feature network model can be selected from various deep learning models such as HRNet, ResNet, and ViT.

[0017] Preferably, the specific implementation process of step 1 is:

[0018] Step 11: Annotate the training sample images with similarity information. Based on the similarity information of the training sample images and the back propagation algorithm, use the classification loss and triplet loss to supervise and complete the training of the deep floating-point feature network model.

[0019] Step 12: Input the training sample images into the trained deep floating-point feature network model to extract the deep floating-point features of all training sample images.

[0020] Preferably, the specific implementation process of step 2 is:

[0021] Step 21: Add a layered deep hash network model to the trained deep floating-point feature network model to construct a deep layered hash-floating-point feature network model; the layered deep hash network model includes N layers of output layers consisting of fully connected layers, activation functions, and sign functions;

[0022] The layered deep hash network model is a multi-dimensional deep hash network model. The number of layers and dimensions of the N-layer deep hash features output by the deep layered hash-floating point feature network model are preset. The number of layers used by the model and the dimension of each layer of hash features are hyperparameters that can be set by the user based on their own database conditions.

[0023] Step 22: Train the layered deep hash network model based on the similarity information and deep floating-point features of the training sample images, and update the parameters of the layered deep hash network model; input each training sample image into the deep layered hash-floating-point feature network model to obtain N-layer deep hash feature prediction results. If the difference between the dimension of the deep floating-point feature corresponding to each training sample image and the dimension of the first-layer deep hash feature prediction result is less than the set threshold, proceed to step 23, otherwise proceed to step 24;

[0024] The similarity guides the training of network model parameters so that the image features generated by the network continue to maintain the similarity; the lowest-dimensional deep hash network model is trained based on the similarity information and deep floating-point features of the training sample images to obtain the lowest-dimensional deep hash features;

[0025] Step 23: Using the knowledge distillation method, the model loss is calculated based on the deep hash feature prediction results and similarity information of each layer, and the KL divergence is calculated based on the deep hash feature prediction results and deep floating point features of each layer as the knowledge distillation loss. The model loss and knowledge distillation loss of each layer are weighted averaged, and the parameters of the hierarchical deep hash network model are updated through back propagation.

[0026] Step 24: Use the knowledge distillation method and teaching assistant mechanism to update the parameters of the hierarchical deep hashing network model;

[0027] Step 241: Using the knowledge distillation method, the model loss is calculated based on the N-layer hash feature prediction results and similarity information, and the KL divergence is calculated based on the N-layer deep hash feature prediction results and deep floating point features as the knowledge distillation loss;

[0028] Step 242: The remaining N-1 layers of deep hash feature prediction results are respectively used with the similarity information to calculate the model loss, and the KL divergence is calculated based on the current layer deep hash feature prediction result and the previous layer deep hash feature prediction result as knowledge distillation;

[0029] Step 243: Update the parameters of the layered deep hashing network model by back propagation after taking the weighted average of the model loss and knowledge distillation loss of each layer;

[0030] Step 25: The deep floating-point feature network model and the updated hierarchical deep hash network constitute the trained deep hierarchical hash-floating-point feature network model.

[0031] The knowledge distillation method and teaching assistant mechanism are used to update the parameters of the deep hierarchical hash-floating feature network model. The knowledge distillation method is used to calculate the prediction results of the hash features and the model loss of the true labels. For the highest-dimensional deep hash features, the KL divergence of the deep hash feature prediction results and the deep floating-point feature prediction results is calculated as the knowledge distillation loss. For non-highest-dimensional deep hash features, the KL divergence of the current layer deep hash feature prediction results and the previous layer high-dimensional deep hash feature prediction results is calculated as the knowledge distillation loss. The model loss and the knowledge distillation loss are weighted averaged and then the parameters of the deep hierarchical hash-floating feature network model are updated through back propagation.

[0032] Preferably, the specific implementation process of step 3 is:

[0033] Step 31: Input the images in the image library to be retrieved into the deep layered hash-floating point feature network model to obtain N layers of deep hash features and deep floating point features for each image;

[0034] Step 32: Establish a tree index according to the obtained N-layer deep hash features and deep floating-point features;

[0035] Select an image, calculate the similarity between the first-layer deep hash features of the current image and other images, and classify them according to the set similarity threshold. Images with a similarity greater than the threshold and other images are divided into two groups of advanced hash index classes respectively;

[0036] In each group of high-level hash index classes, the similarity between the second-level deep hash features of the current image and other images is calculated, and classification is performed according to the set similarity threshold. Images with a similarity greater than the similarity threshold and other images are divided into two groups of sub-high-level hash index classes respectively; each level of hash index class is divided in turn according to the next level of deep hash features. The last level of hash index class contains several images, and the corresponding deep floating-point features are stored as a list. At the same time, the file path or ID of the corresponding image is recorded to form a tuple, and a hierarchical tree index is obtained.

[0037] First, use high-level (low-dimensional) hash index for classification. Each category contains several levels of low-level (high-dimensional) hash indexes. Several low-level (high-dimensional) indexes are repeatedly classified according to the highest-level (lowest-dimensional) hash index until only the last-level (highest-dimensional) hash index remains. The last-level hash index may contain several floating-point feature indexes, which are stored as a list. Each item in the list is a two-tuple that records the image file path or ID and the corresponding floating-point feature index, obtaining a hierarchical tree index.

[0038] First, the deep floating-point feature network model is trained based on the image similarity information to obtain the deep floating-point features. Then, the lowest-dimensional hierarchical deep hash network model is trained based on the deep floating-point features to obtain the lowest-dimensional N-layer deep hash features. Then, the high-dimensional N-layer deep hash features are trained based on the low-dimensional N-layer deep hash features. Finally, the highest-level (lowest-dimensional) deep hash features are obtained. The deep hash features of all different levels (dimensions) constitute the multi-dimensional hierarchical deep hash features. Together with the deep floating-point features obtained first, the hierarchical deep hash-floating-point features are obtained.

[0039] Preferably, the specific implementation process of step 4 is:

[0040] Step 41: Input the image to be retrieved into the deep layered hash-floating point feature network model to obtain its N-layer deep hash features and deep floating point features;

[0041] Step 42: Perform hierarchical retrieval in the order from the root node to the leaf node of the hierarchical tree index; starting from the first level, calculate the distance between the N-layer deep hash features of the image to be retrieved and the N-layer deep hash features of the nodes in the hierarchical tree index of the image library to be retrieved, as well as the distance between the depth floating-point features of the image to be retrieved and the deep hash features of the nodes, and sort them according to the distance. In each level of retrieval, filter some images corresponding to the N-layer deep hash features according to the set ratio based on the sorting results. The last level of retrieval is sorted according to the distance between the depth floating-point features of the filtered image and the depth floating-point features of the leaf nodes, select the image in the image library to be retrieved with the smallest distance as the retrieval result, and output the retrieval result.

[0042] According to the order from the root to the leaf node of the hierarchical tree index, that is, from low-dimensional deep hash features, high-dimensional deep hash features to deep floating-point features, the distance between the N-layer deep hash features and deep floating-point features of the image to be retrieved and the corresponding dimensional features in the hierarchical tree index of the image features of the image library is calculated step by step. The first-layer deep hash features of the image to be retrieved are only calculated with the first-layer deep hash features of the images in the image library to be retrieved. The distance between the N-th layer deep hash features of the image to be retrieved is only calculated with the N-th layer deep hash features of the images in the image library. The deep hash feature values ​​that exceed the given threshold distance range are filtered according to a certain set ratio. For example, under the condition of retaining a ratio of 0.2, the image to be retrieved and 1,000 image library images are sorted using the distance of the first-layer deep hash features. The top 200 images with the closest distance to the image to be retrieved are retained to enter the next layer of retrieval, and finally the retrieval results are output based on the distance sorting of the last layer of deep floating-point features. By sorting each layer of features, a portion of the images with the highest ranking are selected to enter the next layer of retrieval. Therefore, the hash feature actually plays the role of filtering images. Each layer of hash features will reduce the number of images that need to be compared. In the end, only a small number of images will be sorted using the last layer of floating-point features to output the retrieval results.

[0043] As can be seen from the above technical solution, compared with the existing technology, the present invention provides an image retrieval method, storage medium, and product based on layered hash floating-point features. This method uses knowledge distillation layered deep hash-floating-point features to efficiently retrieve images, effectively improving the accuracy and efficiency of image search. This method can be applied to computer vision tasks such as pedestrian / vehicle re-identification and object classification, as well as natural language processing tasks such as knowledge graph relationship classification and link prediction, which also require feature similarity calculation, to achieve efficient retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 The accompanying drawing is a schematic flow chart of the image retrieval method based on layered hash floating-point features provided by the present invention;

[0046] Figure 2 The accompanying drawing is a schematic diagram of the structure of the hierarchical hash feature part and floating-point feature part of the convolutional neural network model provided by the present invention;

[0047] Figure 3The accompanying drawing is an example diagram of the retrieval process provided by the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0049] The embodiment of the present invention specifically discloses an image retrieval method based on layered hash floating-point features, comprising the following steps:

[0050] S1: Obtain training sample images and annotate them with similarity information; construct a deep floating-point feature network model, and use the training sample images and their corresponding similarity information to train the deep floating-point feature network model using the back-propagation algorithm to obtain the trained deep floating-point feature network model and the deep floating-point features of all training sample images;

[0051] S11: Deep floating-point feature network models can use various deep learning models such as HRNet, ResNet, and ViT;

[0052] S12: For the network model selected in S11, supervise and complete the training of the deep floating-point feature network model based on the similarity information of the training sample image annotations and the back-propagation algorithm;

[0053] S13: Input all training sample images into the trained deep floating-point feature network model to extract deep floating-point features of all training sample images;

[0054] S2: Based on the trained deep floating-point feature network model and the deep floating-point features of all training sample images, a knowledge distillation method is used to construct and train a deep hierarchical hash-floating-point feature network model to obtain hierarchical deep hash features of all training sample images;

[0055] S21: Add multiple layers of fully connected layers, activation functions, and sign functions as output layers of different dimensions after the trained deep floating-point feature network model to construct a hierarchical hash-floating-point feature network model;

[0056] S22: During the training process, the parameters of the deep floating-point feature network model constructed in S1 are fixed, and only the parameters of the newly added layered hash-floating-point feature network model in S21 are trained and optimized;

[0057] S23: During the training of the deep hierarchical hash network model, the knowledge distillation method is used. That is, in addition to calculating the loss of the hash feature prediction results and the true label, the KL divergence of the deep hash feature prediction results and the deep floating point feature prediction results is also calculated as the knowledge distillation loss. The two losses are weighted averaged and the parameters of the newly added hierarchical hash-floating point feature network model are updated through backpropagation.

[0058] S24: When the dimension of the deep floating-point feature is significantly different from that of the deep hash feature, a teaching assistant mechanism can be added on the basis of S23, that is, the deep floating-point feature network model (i.e., teacher) guides the training of the high-dimensional hash feature model (i.e., teaching assistant), and then the high-dimensional hash feature model (i.e., teaching assistant) guides the low-dimensional hash feature model (i.e., student), thereby achieving layer-by-layer distillation and accelerated convergence; specifically, for the highest-dimensional deep hash feature, the network is trained according to S23; the KL divergence of the prediction results of the deep hash feature and the prediction results of the deep floating-point feature is calculated as the knowledge distillation, and the loss guides the training; for non-highest-dimensional deep hash features, the KL divergence of the prediction results of the current layer deep hash feature and the prediction results of the previous layer high-dimensional deep hash feature is calculated as the loss of knowledge distillation, and the loss of the prediction results of the current layer hash feature and the true label is calculated. After the two parts of the loss are weighted averaged, the parameters of the newly added hierarchical hash-floating feature network model are updated through back propagation;

[0059] S3: Input the images in the image library into the deep hierarchical hash-floating point feature network model to obtain hierarchical deep hash features and deep floating point features of different dimensions for each image, and construct a hierarchical tree index of the image features of the image library;

[0060] S31: Input the images in the image library into the deep hierarchical hash-floating point feature network model to obtain hierarchical deep hash features and deep floating point features of different dimensions for each image;

[0061] S32: A tree-like index is established based on the obtained hierarchical hash-floating point features. First, a high-level (low-dimensional) hash index is used for classification. Each category contains several levels of low-level (high-dimensional) hash indexes. Several low-level (high-dimensional) indexes are repeatedly classified according to the highest-level (lowest-dimensional) hash index among them, until only the last-level (highest-dimensional) hash index remains.

[0062] S33: The last-level hash index obtained in S32 may contain several floating-point feature indexes, which are stored as a list. Each item in the list is a two-tuple that records the image file path or ID and the corresponding floating-point feature index.

[0063] S4: Input the image to be retrieved into the deep hierarchical hash-floating point feature network model to obtain its hierarchical hash features and deep floating point features, and perform a level-by-level search in the hierarchical tree index of image features in the image library. Output the retrieval results based on the distance sorting of the last layer of deep floating point features.

[0064] S41: Input the image to be retrieved into the deep hierarchical hash-floating point feature network model to obtain its hierarchical deep hash features and deep floating point features;

[0065] S42: According to the order from the root of the hierarchical hash index to the leaf node, that is, according to the coarse to fine order of low-dimensional deep hash features, high-dimensional deep hash features to deep floating-point features, the distance between the hierarchical deep hash features and deep floating-point features of the image to be retrieved and the corresponding dimensional features in the hierarchical tree index of the image features of the image library is calculated step by step, and the deep hash feature values ​​that exceed the given threshold distance range are filtered out according to a certain ratio, and finally the retrieval results are output based on the distance sorting of the last layer of deep floating-point features.

[0066] In a specific embodiment, a method for image retrieval based on knowledge distillation layered deep hash-floating point features is disclosed. To address the problem of slow speed and low precision of deep hash image retrieval methods, a floating point feature network is used to fit the trained image library, and the obtained floating point features are used to extract layered hash values ​​through the knowledge distillation method. By establishing a tree index for the layered hash-floating point values, the search process can use the comparison of high-level hash values ​​to cut off some branches that do not need to be searched further, thereby greatly accelerating the search process. In the last layer, accurate floating point features are used for local sorting, thereby improving the accuracy of the final retrieval results. The accuracy and efficiency of content-based image search problems are guaranteed.

[0067] The overall process is as follows Figure 1 , including the training of floating-point feature networks, the training of hierarchical hash neural networks, hash extraction of image libraries and establishment of tree-structured hash indexes, and retrieval of query images. The specific steps include:

[0068] S1: Use the labeled image library to be retrieved to train the floating-point feature network;

[0069] Specifically, a deep convolutional neural network is first trained using a labeled image library to extract floating-point features from the images. The deep convolutional neural network used can be a mainstream model for image classification, such as ResNet, HRnet, ViT, etc. During the training process, classification loss and triplet loss are used to supervise the training of the deep floating-point feature network model, obtaining the trained deep floating-point feature network model and the deep floating-point features of all training sample images.

[0070] S2: Use the labeled image library to be retrieved and the deep floating-point features generated in S1 to train a hierarchical hash neural network to obtain a deep hierarchical hash-floating-point feature network model;

[0071] The layered hash neural network is constructed by passing the deep floating-point features generated by the deep floating-point feature network model through multiple fully connected layers, followed by activation functions and sign functions. The activation function after the fully connected layer causes the value of each dimension in the feature to be distributed near 0, reducing information loss after binarization by the sign function. The number of layers in the layered hash neural network and the length of the hash features in each layer are specified by the user. Larger image libraries can use deeper layers, and applications with higher retrieval accuracy requirements can use longer hash layers.

[0072] The connection method of the deep layered hash-floating point feature network model is as follows Figure 2 As shown, it is an N-level hierarchical hash-floating point network using knowledge distillation in this specific embodiment; during training, the floating point feature network is first trained using the labeled image library Training Set as the training set to obtain floating point features. After the floating point feature network reaches the optimal effect, the parameters of the floating point feature network, that is, the parameters of the deep floating point feature network model, are fixed; the obtained floating point features are respectively passed through N hash layers to generate N hash features of different lengths. In the process of training the hash layer, as shown in FIG. Figure 2 As shown, if only the knowledge distillation method is used, as shown by the dotted line connection relationship in the figure, in addition to calculating the loss of the prediction result of the hash feature and the true label, the KL divergence of the prediction result of the deep hash feature and the prediction result of the deep floating-point feature is also calculated as the loss of knowledge distillation. The two parts of the loss are weighted averaged and the parameters of the hierarchical hash network model are updated through back propagation; when the dimension of the deep floating-point feature is significantly different from that of the deep hash feature, an assistant teaching mechanism can be added on the basis of the knowledge distillation method, as shown by the short dotted line in the figure, the N-th level hash feature calculates the loss of the prediction result of the deep hash feature and the prediction result of the deep floating-point feature through KL divergence to guide the training (loss), the first level hash feature calculates the loss of the prediction result of the deep hash feature and the prediction result of the second level feature through KL divergence to guide the training, and the second level hash feature also calculates the loss of the prediction result of the deep hash feature and the prediction result of the third level feature through KL divergence to guide the training, and so on to achieve the effect of layer-by-layer distillation and accelerated convergence;

[0073] S3: extracting feature values ​​from the image library to be searched and building a tree-like hash index for the extracted hash-floating point values;

[0074] Specifically, all images that need to be stored are input into the trained deep layered hash-floating point feature network model, which outputs the hash feature values ​​of each hash layer and the final floating point feature value;

[0075] For the hierarchical hash-float algorithm, we can first classify according to the first-level hash features and store images with similar first-level hash feature distances in similar locations;

[0076] If there are only two levels of indexes, after the high-level indexing, there is only one level of index left, which can be stored as a list. Each item in the list is a two-tuple that records the image file path or ID and the low-level index at the same time. If the index has three or more levels, then after the above steps, there are two or more levels of indexes under each high-level hash value. It is necessary to continue to classify according to the higher-level hash values ​​until only the first level of index is left. Finally, each item stored in a list is a two-tuple that records the image file path or ID and the low-level index at the same time.

[0077] S4: Search for relevant images through hierarchical retrieval;

[0078] like Figure 3 , is an example of a hierarchical retrieval process. When querying, the image to be retrieved is first input into the generated deep hierarchical hash-floating point feature network model to obtain the corresponding hierarchical hash-floating point features (hash-floating point values). The Hamming distance between the first-level hash features of the image to be retrieved and the first-level hash features of all images in the database is calculated. After sorting, a certain proportion of the top-ranked images are retained to enter the next level of retrieval. The specific retention ratio can be specified by the user, and its size will affect the speed and accuracy of the retrieval process, such as Figure 3 As shown in the figure, after the first-level hash feature sorting, a portion of the images are sorted using the second-level hash feature, and a certain proportion of the images enter the next level of retrieval. This process is repeated until only a small number of images are sorted locally using higher-dimensional and more accurate deep floating-point features, and the results are used as the final results. If only the top 20% of each level of index is retained and entered into the next level, then the number of images retrieved using floating-point features will only be the same as the previous number. , where N is the number of layers of hierarchical hashing, which greatly reduces the amount of calculation and significantly improves the retrieval efficiency.

[0079] For segmented search algorithms, you can adjust the number of segments (N) and the segment length (K) to fine-tune the algorithm. A larger N value significantly accelerates search performance, but may also reduce accuracy. Choosing appropriate parameters is crucial for practical applications. By controlling the ratio of layered filtering at different levels, you can trade off search speed and accuracy.

[0080] In addition, the principles of the present invention can also be used for the efficient calculation of feature similarity in computer vision tasks such as pedestrian / vehicle re-identification and target classification and recognition, and natural language processing tasks such as knowledge graph relationship classification and link prediction.

[0081] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0082] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image retrieval method based on hierarchical hash floating point features, characterized in that: The following steps are involved: Step 1: Obtain a training sample image, build a deep floating-point feature network model, and use the training sample image to train the deep floating-point feature network model to obtain the trained deep floating-point feature network model and the deep floating-point features of the training sample image; Step 2: Build a deep layered hash-floating-point feature network model based on the trained deep floating-point feature network model, and use the deep floating-point features of the training sample images for training, to obtain the trained deep layered hash-floating-point feature network model and N layers of deep hash features of the training sample images, N ≥ 2; each layer of deep hash features has different dimensions; Step 3: Input the images in the image library to be retrieved into the deep hierarchical hash-floating point feature network model to obtain the image features of each image, including N layers of deep hash features and deep floating point features, and construct a hierarchical tree index of the image features of the image library to be retrieved; Step 4: Input the image to be retrieved into the deep hierarchical hash-floating point feature network model to obtain its N-layer deep hash features and deep floating point features, and perform a level-by-level search in the hierarchical tree index to output the retrieval results; The specific implementation process of step 2 is: Step 21: Add a layered deep hash network model to the trained deep floating-point feature network model to construct a deep layered hash-floating-point feature network model; The layered deep hashing network model includes N layers of output layers consisting of fully connected layers, activation functions, and sign functions; Step 22: Train the layered deep hash network model based on the similarity information and deep floating-point features of the training sample images, and update the parameters of the layered deep hash network model; input each training sample image into the deep layered hash-floating-point feature network model to obtain N-layer deep hash feature prediction results. If the difference between the dimension of the deep floating-point feature corresponding to each training sample image and the dimension of the first-layer deep hash feature prediction result is less than the set threshold, proceed to step 23, otherwise proceed to step 24; Step 23: Using the knowledge distillation method, the model loss is calculated based on the deep hash feature prediction results and similarity information of each layer, and the KL divergence is calculated based on the deep hash feature prediction results and deep floating point features of each layer as the knowledge distillation loss. The model loss and knowledge distillation loss of each layer are weighted averaged, and the parameters of the hierarchical deep hash network model are updated through back propagation. Step 24: Use the knowledge distillation method and teaching assistant mechanism to update the parameters of the hierarchical deep hashing network model; Step 241: Using the knowledge distillation method, the model loss is calculated based on the N-layer hash feature prediction results and similarity information, and the KL divergence is calculated based on the N-layer deep hash feature prediction results and deep floating point features as the knowledge distillation loss; Step 242: The remaining N-1 layers of deep hash feature prediction results are respectively used with the similarity information to calculate the model loss, and the KL divergence is calculated based on the current layer deep hash feature prediction result and the previous layer deep hash feature prediction result as knowledge distillation; Step 243: Update the parameters of the layered deep hashing network model by back propagation after taking the weighted average of the model loss and knowledge distillation loss of each layer; Step 25: The deep floating-point feature network model and the updated hierarchical deep hash network constitute the trained deep hierarchical hash-floating-point feature network model.

2. The image retrieval method based on layered hash floating point features according to claim 1, characterized in that: The specific implementation process of step 1 is: Step 11: Annotate the training sample images with similarity information. Based on the similarity information of the training sample images and the back propagation algorithm, supervise and complete the training of the deep floating-point feature network model. Step 12: Input the training sample image into the trained deep floating-point feature network model to extract the deep floating-point features of the training sample image.

3. The image retrieval method based on layered hash floating point features according to claim 1, characterized in that: The specific implementation process of step 3 is: Step 31: Input the images in the image library to be retrieved into the deep layered hash-floating point feature network model to obtain N layers of deep hash features and deep floating point features for each image; Step 32: Establish a tree index according to the obtained N-layer deep hash features and deep floating-point features; Select an image, calculate the similarity between the first-layer deep hash features of the current image and other images, and classify them according to the set similarity threshold. Images with a similarity greater than the threshold and other images are divided into two groups of advanced hash index classes respectively; In each group of high-level hash index classes, the similarity between the second-level deep hash features of the current image and other images is calculated, and classification is performed according to the set similarity threshold. Images with a similarity greater than the similarity threshold and other images are divided into two groups of sub-high-level hash index classes respectively; each level of hash index class is divided in turn according to the next level of deep hash features. The last level of hash index class contains several images, and the corresponding deep floating-point features are stored as a list. At the same time, the file path or ID of the corresponding image is recorded to form a tuple, and a hierarchical tree index is obtained.

4. The image retrieval method based on layered hash floating point features according to claim 1, characterized in that: The specific implementation process of step 4 is as follows: Step 41: Input the image to be retrieved into the deep layered hash-floating point feature network model to obtain its N-layer deep hash features and deep floating point features; Step 42: Perform hierarchical retrieval in the order from the root node to the leaf node of the hierarchical tree index; starting from the first level, calculate the distance between the N-layer deep hash features of the image to be retrieved and the N-layer deep hash features of the nodes in the hierarchical tree index of the image library to be retrieved, as well as the distance between the depth floating-point features of the image to be retrieved and the depth floating-point features of the nodes, and sort them according to the distance. In each level of retrieval, filter the images corresponding to some N-layer deep hash features according to the set ratio based on the sorting results. The last level of retrieval is sorted according to the distance between the depth floating-point features of the filtered image and the depth floating-point features of the leaf node, select the image in the image library to be retrieved with the smallest distance as the retrieval result, and output the retrieval result.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images