Image retrieval method, medium and device based on local feature enhancement and re-ranking

Abstract

The present invention discloses an image retrieval method, medium and device based on local feature enhancement and reordering. The present invention constructs a feature extraction model, and by introducing a local feature enhancement encoder, extracts the local geometric structure information of the region of interest in the feature map, so that the features in the region of interest are given a higher weight, and the interference of non-important areas on the global features is reduced, so that the encoder can more accurately extract the important content in the image. The present invention introduces a feature fusion module, and obtains a fused feature map that takes into account both local and global features through feature fusion. The addition of reordering technology allows the model to update the feature vectors of the query image and the image library image according to the information of the image library, so that the query image feature vector is more adapted to the image library feature vector, thereby improving the accuracy of the retrieval results. The present invention is trained and evaluated on the ROxford5K dataset, the RParis6K dataset and the Revisitop1m dataset. Compared with previous methods, the present invention has better results.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to an image retrieval method based on local feature enhancement and reordering, which is used to output an image in an image library that is closest to the information expressed by a given image. Background Art

[0002] Image retrieval involves extracting the desired images from input information, thereby retrieving images that meet the requirements from an image retrieval database. Image retrieval can be applied to fields such as search engines, e-commerce, and healthcare. Therefore, image retrieval technology holds significant research value.

[0003] Research in image retrieval technology can be categorized into text-based and content-based techniques, depending on how the image's content is described. Text-based image retrieval utilizes textual descriptions to retrieve images, while content-based image retrieval utilizes information such as color, texture, objects contained in the image, and categories. While content-based image retrieval requires more complex input information, it is more intuitive and convenient to apply. Therefore, research in content-based image retrieval has room for further development and has become increasingly popular in recent years.

[0004] The core of content-based image retrieval research lies in extracting the most effective feature vectors from images. There are three methods for extracting feature vectors: local features, global features, and a combination of local and global features. Each method uses a feature map generated by passing the image through a neural network as the initial processing step. The global feature method uses pooling and other methods to generate a feature vector from the feature map, while the local feature method selects the vectors of the most important regions in the feature map as feature vectors. The combined local and global feature method performs both local and global feature processing on the feature map, then combines the local and global vectors or uses them separately for image retrieval. In addition to extracting feature vectors, reranking the image library and feature vectors during image retrieval can further improve image retrieval accuracy. Reranking involves combining the feature vectors of the top-ranked images in the image library based on the initial search results, replacing the query feature vector, and then performing further retrieval.

[0005] Current research methods for content-based image retrieval still face many challenges. Global feature methods often lack local information, which results in interference from irrelevant information in the image during the pooling of feature maps, limiting their accuracy. Local feature processing, which utilizes multiple features, often requires significant computing resources. Combining local and global features makes it difficult to develop an effective vector merging method or to leverage the combined effects of both. Furthermore, models often have poor generalization capabilities, and efficiency often drops significantly after updating the image library. Summary of the Invention

[0006] In current image retrieval tasks, the search target in a query image is easily confused with similar images in the image library, hindering image retrieval. Therefore, solving the problem of distinguishing similar but different images and improving recognition accuracy is a pressing technical problem. The present invention aims to address these existing technical issues and propose an image retrieval method based on local feature enhancement and re-ranking.

[0007] In order to achieve the above-mentioned purpose of the invention, the technical solutions specifically adopted by the present invention are as follows:

[0008] In a first aspect, the present invention provides an image retrieval method based on local feature enhancement and re-ranking, comprising the following steps:

[0009] S1. Obtain images of uniform resolution and size that have been grouped by category as training data for the neural network;

[0010] S2. The feature extraction model is cascaded with the classifier, and the feature extraction model is trained with the training data for the image classification task; the feature extraction model is composed of a backbone network, a local feature enhancement encoder, a feature fusion module and a global pooling module. The input image of the model is first subjected to the backbone network to extract the initial feature map, and then the local feature enhancement encoder is used to expand the features of each pixel in the initial feature map into a local feature map corresponding to the area centered on the pixel. All local feature maps are compressed into a feature vector and then recombined into an enhanced local feature map according to the original spatial position relationship of the central pixel. The initial feature map and the enhanced local feature map are then linearly fused by the feature fusion module, and the obtained fused feature map is passed through the global pooling module to obtain the global feature;

[0011] S3. Using the trained feature extraction model, perform feature extraction on the query image and each image in the image index library, and extract corresponding global features from each image.

[0012] S4. The query image and each image library image in the image index library are respectively used as the retrieved images. The image retrieval model calculates the similarity between the images based on the global features of the images, and retrieves several image library images with the highest similarity from the image index library to form a similar image set of the retrieved image; then a reranking method based on query expansion and database enhancement is used to fuse and update the global features of all images in each similar image set into the global features of the corresponding retrieved image itself; finally, the image retrieval model re-retrieves similar images of the query image in the image index library based on the fused and updated global features to obtain the final retrieval result.

[0013] As a preferred embodiment of the first aspect, the images in the training data, the query images, and the images in the image index library are all landmark images.

[0014] As a preferred embodiment of the first aspect, the local feature enhancement encoder is formed by cascading a local feature region cutting module and a local feature enhancement encoding module;

[0015] The local feature region cutting module includes a filling layer and a sliding window cutting layer. The filling layer first fills the periphery of the pixel area of ​​the initial feature map with a width of 2, and then the sliding window cutting layer uses a 5*5 window to slide on the filled area with a step size of 1, extracting the local feature map corresponding to the 5*5 area centered on each original pixel;

[0016] The local feature enhancement coding module includes two local feature coding layers connected in series, each of which is composed of a 3*3 convolutional layer, a normalization layer and a linear activation layer. After passing through the two local feature coding layers, each local feature map is compressed and converted into a feature vector corresponding to the central pixel of the local feature map. All feature vectors obtained by compression conversion are recombined according to the position of the central pixel in the initial feature map to form an enhanced local feature map.

[0017] As a preferred embodiment of the above-mentioned first aspect, in the feature fusion module, the enhanced local feature map is normalized and then spliced ​​with the initial feature map, and then the obtained spliced ​​features are sequentially passed through a convolution layer, a linear activation layer, and then through another convolution layer to realize linear fusion of spatial local information and global information, and output a fused feature map.

[0018] As a preferred embodiment of the first aspect, when using the training data to train a feature extraction model for an image classification task, an ArcFace loss function is used as a loss function.

[0019] As a preferred embodiment of the first aspect, in S4, the process of obtaining the final search results by using the re-ranking method based on query expansion and database enhancement is as follows:

[0020] S41, inputting the query image into the image retrieval model, calculating the similarity between images based on the global features of the images, and retrieving several image library images with the highest similarity from the image index library to form a first similar image set to the query image;

[0021] S42: Input each image library image in the image index library into the image retrieval model, calculate the similarity between images based on the global features of the images, and retrieve several image library images with the highest similarity, including the image library image itself, from the image index library to form a second similar image set for each image library image;

[0022] S43. For all similar images in the first similar image set, using the normalized similarity between each image and the query image as a weight, performing weighted fusion on the global features of all images and superimposing the fused features onto the global features of the query image, thereby updating the global features of the query image.

[0023] S44. For each image library image in the image index library, perform weighted fusion on the global features of all similar images in the respective second similar image set, where the weighted value is the normalized similarity between each similar image and the current image library image, and use the fused features as the updated global features of the current image library image;

[0024] S45. The image retrieval model recalculates the similarity between the updated global features of the query image and the updated global features of each image in the image index library, and returns several image library images with the highest similarity as the final retrieval results.

[0025] As a preferred embodiment of the first aspect above, the backbone network adopts a convolutional neural network.

[0026] In a second aspect, the present invention provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, can implement the image retrieval method based on local feature enhancement and reordering as described in any one of the solutions in the first aspect above.

[0027] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image retrieval method based on local feature enhancement and reordering as described in any one of the schemes in the first aspect above can be implemented.

[0028] In a fourth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0029] The memory is used to store computer programs;

[0030] The processor is configured to implement the image retrieval method based on local feature enhancement and reordering as described in any one of the solutions of the first aspect when executing the computer program.

[0031] Compared with the prior art, the present invention adopts the above technical solution, and has the following beneficial effects:

[0032] 1) This paper designs a local feature enhancement encoder. Compared with ordinary encoders, this local feature enhancement encoder integrates the surrounding feature information of each feature into the feature. During the global feature extraction process, global features are more easily affected by features in unimportant areas. The local feature enhancement encoder extracts the local geometric structure information in the region of interest of the feature map, giving the features of the region of interest more weight, reducing the interference of unimportant areas on the global features, and enabling the encoder to more accurately extract the important content of the image.

[0033] 2) The present invention designs a feature fusion module. The feature map generated after the local feature enhancement encoder can enhance spatial locality but cannot be used as a single retrieval criterion. The feature map that has not passed the local feature enhancement encoder often ignores spatial locality. The feature fusion module can retain the advantages of both while eliminating the impact of their disadvantages.

[0034] 3) The present invention introduces re-ranking technology, which enables the model to update the feature vectors of the query image and the image library image according to the information of the image library, making the query image feature vector more adapted to the image library feature vector, and also improving the generalization ability of the model and the accuracy of the retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a structural diagram of the feature extraction model of the present invention.

[0036] Figure 2 This is a structural diagram of the local feature enhancement coding module of the present invention.

[0037] Figure 3 This is a structural architecture diagram of the feature fusion module of the present invention.

[0038] Figure 4 This is a structural diagram of the reordering method of the present invention.

[0039] Figure 5 This is an example of the model image retrieval effect of the present invention. DETAILED DESCRIPTION

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0041] The terms "including," "having," and any variations thereof, as used in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to the process, method, product, or apparatus.

[0042] In view of the problem that the retrieval target in the query image in the current image retrieval task is easily confused with similar images in the image library, which is not conducive to image retrieval, the present invention introduces a local feature enhancement encoder that strengthens the spatial correlation of features in the feature map, strengthens the association between each feature in the feature map and its surrounding features, so that each vector in the feature map has wider spatial information; at the same time, by adding a feature fusion module, the feature map after local feature enhancement and the initial global feature map are merged according to learnable weights, and the local feature enhancement feature map can enhance spatial locality but cannot be fully utilized as a single retrieval criterion, so that it and the unprocessed feature map are fused into an image retrieval feature map that embodies spatial locality. In addition, the present invention also introduces a re-ranking technology to improve the accuracy of retrieval without participating in model training, so that the generalization ability of retrieval is stronger and can better adapt to different types of images. The specific implementation of the method in the present invention is described in detail below.

[0043] In the present invention, the specific implementation steps of the image retrieval method based on local feature enhancement and re-ranking are as follows:

[0044] S1. Obtain images with uniform resolution and grouped by category as training data for the neural network.

[0045] In the embodiment of the present invention, the images in the training data, the query images, and the images in the image index library can all be landmark images, i.e., iconic architectural landmarks in different regions. Of course, the present invention is not limited to searching for landmark images, and can also be applied to any other images.

[0046] S2. Cascade the feature extraction model and the classifier, and use the training data to train the feature extraction model with the image classification task; the feature extraction model consists of a backbone network, a local feature enhancement encoder, a feature fusion module and a global pooling module. The input image of the model first passes through the backbone network to extract the initial feature map, and then passes through the local feature enhancement encoder to expand the features of each pixel in the initial feature map into a local feature map corresponding to the area centered on the pixel. All local feature maps are each compressed into a feature vector and then recombined into an enhanced local feature map according to the original spatial position relationship of the center pixel. The initial feature map and the enhanced local feature map are then linearly fused by the feature fusion module, and the obtained fused feature map is passed through the global pooling module to obtain the global feature.

[0047] In an embodiment of the present invention, the structure of the feature extraction model is as follows: Figure 1 As shown in the figure, the backbone network can be implemented by a convolutional neural network; the local feature enhancement encoder is composed of a cascade of a local feature region cutting module and a local feature enhancement encoding module. The local feature region cutting module is responsible for expanding each local feature into a local feature map of the area centered on the local feature; the local feature enhancement feature encoding module is responsible for compressing the information in the local feature map of each area into a local vector, so as to achieve the purpose of local feature enhancement.

[0048] The local feature region cutting module includes a padding layer and a sliding window cutting layer. The padding layer first fills the pixel area of ​​the initial feature map with a width of 2. The sliding window cutting layer then slides a 5*5 window over the padded area with a step size of 1, extracting the local feature map corresponding to a 5*5 area centered on each original pixel. The function of the local feature region cutting module is to convert the feature map dimension to a dimension that is suitable for the local feature enhancement coding module, and to convert the feature vector of each pixel in the feature map output by the backbone network into a feature vector map of a region.

[0049] And as Figure 2 As shown in the figure, the local feature enhancement coding module includes two local feature coding layers connected in series. Each local feature coding layer is composed of a 3*3 convolutional layer, a normalization layer and a linear activation layer (ReLU layer) cascaded together. After passing through the two local feature coding layers, each local feature map is compressed and converted into a feature vector corresponding to the central pixel of the local feature map. All feature vectors obtained by compression conversion are recombined according to the position of the central pixel in the initial feature map to form an enhanced local feature map.

[0050] The calculation of each local feature encoding layer in the local feature enhancement encoding module can be expressed as follows:

[0051] s i=max(0,Batchnorm(Conv(s i+1 )))

[0052] where s i Represents the output of the i-th local feature encoding layer. In this process, the feature vector of each pixel is converted into a feature vector map of a region and then converted back into a feature vector, which reduces the amount of subsequent calculations and has more spatial information for each pixel compared to the original features output from the backbone network.

[0053] like Figure 3 As shown in the figure, a structural diagram of the feature fusion module is shown. In this feature fusion module, the enhanced local feature map is first normalized and then spliced ​​with the initial feature map. The spliced ​​features are then passed through a convolutional layer, a linear activation layer (ReLU layer), and then through another convolutional layer to achieve linear fusion of spatial local information and global information, and a fused feature map is output.

[0054] In an embodiment of the present invention, if the input of the feature fusion module is the initial feature map output from the backbone network and the output of the local feature enhancement encoder Then, the output of the local feature enhancement encoder is linearly fused with the initial feature map features to obtain both excellent spatial local information and global information. This process can be expressed as:

[0055] f6=max(0,(Concat(f3,f0)×W1)+B1)×W2+B2

[0056] Where: W i and B i are the weight and bias of the i-th convolutional layer respectively, and Concat represents the concatenation operation.

[0057] In this feature fusion module, all pixel locations are processed individually to reduce the overall computational load.

[0058] In an embodiment of the present invention, the feature extraction model needs to be trained in advance using the training data obtained in S1 to meet the extraction accuracy requirements. When training the feature extraction model for an image classification task, the loss function may be an ArcFace loss function.

[0059] S3. Use the trained feature extraction model to extract features from the query image and each image in the image index library, and extract corresponding global features from each image.

[0060] S4. The query image and each image library image in the image index library are respectively used as the retrieved images. The image retrieval model calculates the similarity between the images based on the global features of the images, and retrieves several image library images with the highest similarity from the image index library to form a similar image set of the retrieved image; then a reranking method based on query expansion and database enhancement is used to fuse and update the global features of all images in each similar image set into the global features of the corresponding retrieved image itself; finally, the image retrieval model re-retrieves similar images of the query image in the image index library based on the fused and updated global features to obtain the final retrieval result.

[0061] The reranking method of the present invention incorporates query expansion and database enhancement. Its core approach is to find the most similar photos to the query image and each photo in the image library based on the results of the initial search. The updated query feature vector and image library feature matrix are then obtained by linearly adding the feature vectors of the most similar photos for each photo. Finally, the search is repeated using the updated features. This method of updating the feature vector of an image using the feature vector of the image in the database can significantly improve the accuracy of search results.

[0062] In an embodiment of the present invention, the process of obtaining the final search results using the re-ranking method based on query expansion and database enhancement is as follows:

[0063] S41. Input the query image into the image retrieval model, calculate the similarity between images based on the global features of the images, retrieve several image library images with the highest similarity from the image index library, and form a first similar image set of the query image.

[0064] S42. Input each image library image in the image index library into the image retrieval model, calculate the similarity between images based on the global features of the images, and retrieve several image library images with the highest similarity, including the image library image itself, from the image index library to form a second similar image set for each image library image.

[0065] S43. For all similar images in the first similar image set, using the normalized similarity between each image and the query image as a weight, perform weighted fusion on the global features of all images and superimpose the fused features onto the global features of the query image, thereby updating the global features of the query image.

[0066] The essence of step S43 is to implement query expansion, that is, to update the global feature vector of the query image. This is done by first finding the feature vectors of the n images most similar to the query image from the feature matrix of the image library by sorting the similarity scores, and then linearly adding the feature vectors of these n images to obtain the updated feature vector. The calculation process is as follows:

[0067]

[0068] Where: V′ q and V q are the updated global features and the global features before and after updating corresponding to the query image; α is the exponential coefficient, score qi is the similarity score between the query image and the image in the image library that is similar to the query image. During this process, the query image's feature vector is updated using the feature vectors of the image library images, with feature vectors that are more similar to the query image being given higher weights, using the similarity scores as weights. Generally speaking, a first search can largely filter out images in the image library that share the same category as the query image. However, query expansion allows the query image to better represent the category of images rather than just a single image, significantly improving the accuracy of retrieval results.

[0069] S44. For each image library image in the image index library, perform weighted fusion on the global features of all similar images in the respective second similar image set, where the weighted value is the normalized similarity between each similar image and the current image library image, and the fused features are used as the updated global features of the current image library image.

[0070] The essence of the above step S44 is to achieve database enhancement, that is, to update the feature vectors of the images in the image library. Similar to the above query expansion, this database enhancement also obtains the most similar images through similarity scores, and then updates the feature vectors using the feature vectors of the most similar images. The calculation formula can be expressed as:

[0071]

[0072] Where: V′ i and V i are the updated global features and the global features before and after updating corresponding to the image in the i-th image index library, respectively.

[0073] Similar to query expansion, database enhancement can aggregate the feature vectors of a single image into feature vectors similar to images of this type, thereby greatly enhancing retrieval accuracy.

[0074] S45. The image retrieval model recalculates the similarity between the updated global features of the query image and the updated global features of each image in the image index library, and returns several image library images with the highest similarity as the final retrieval results.

[0075] In an embodiment of the present invention, similarity between images is calculated by using the distance values ​​of their respective global features. The similarity can be calculated by multiplying global vectors or can be represented by cosine distance or other methods.

[0076] It should be noted that the number of images in the image library with the highest similarity returned by the image retrieval model in the present invention can be optimized and adjusted according to actual needs and is not limited.

[0077] It should be noted that the above method steps S1 to S4 can essentially be implemented in the form of a computer program.

[0078] Similarly, based on the same inventive concept, the present invention also provides a computer electronic device corresponding to the image retrieval method based on local feature enhancement and re-ranking provided in the above embodiment, which includes a memory and a processor;

[0079] The memory is used to store computer programs;

[0080] The processor is configured to implement the aforementioned image retrieval method based on local feature enhancement and re-ranking when executing the computer program;

[0081] Furthermore, the logic instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention.

[0082] Therefore, based on the same inventive concept, the present invention provides a computer-readable storage medium corresponding to an image retrieval method based on local feature enhancement and reordering, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it can implement the image retrieval method based on local feature enhancement and reordering as described above.

[0083] Therefore, based on the same inventive concept, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the image retrieval method based on local feature enhancement and re-ranking as described above.

[0084] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by the processor to perform the above steps S1 to S4.

[0085] It is understood that the storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Furthermore, the storage medium may be any medium capable of storing program code, such as a USB flash drive, a mobile hard drive, a magnetic disk, or an optical disk.

[0086] It is understandable that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0087] It should also be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0088] The image retrieval method based on local feature enhancement and re-ranking shown in S1 to S4 in the above embodiments is applied to a specific example to demonstrate its technical effect.

[0089] Example

[0090] In this embodiment, the steps of image retrieval based on local feature enhancement and re-ranking are the same as those in S1 to S4 above. The specific process and parameter definitions can also be found in the previous text. The following specifically shows the specific implementation process in the example. The detailed method is as follows:

[0091] Step (1): Obtain training data for neural network training.

[0092] 1.1 In this embodiment, the training data is a fixed-resolution landmark image after preprocessing of grouped landmark images. Landmark images and their corresponding annotations can be obtained from the existing public dataset GLDv2-clean.

[0093] GLDv2-clean is an image retrieval dataset. GLDv2 contains over 5 million images of over 200,000 different landmarks collected by photographers around the world. GLDv2-clean is a subset of GLDv2, containing 1.5 million images and 81,313 classes. Each image is annotated with the landmark it corresponds to. Training a model with such a massive dataset significantly enhances its generalization.

[0094] In order to verify the generalization of the model, the present invention will also use the ROxford5K dataset, RParis6K dataset and Revisitop1m dataset as auxiliary datasets for evaluation. The original Oxford and Paris datasets consist of 5063 and 6392 high-resolution images, respectively. Each dataset contains 55 queries, and each landmark contains 5 queries from a total of 11 landmarks. ROxford5K and RParis6K are re-annotated on the Oxford5K and Paris6K data, fixing the previous labeling issues; increasing the number of queries from 55 to 70; redistributing labels; and proposing three types of evaluation methods: Easy\Medium\Hard. The R1M dataset includes 1,001,001 landmark images. In order to further verify the robustness of the model, the present invention uses the R1M dataset as interference and mixes it with the ROxford5K dataset and RParis6K dataset to test the robustness of the model under the influence of so many interfering images.

[0095] 1.2 Perform data preprocessing on image retrieval images and transform the resolution to a fixed size to adapt to the input scale of the feature extraction model.

[0096] In the above data preprocessing operations, the resolution of the color image needs to be converted to a fixed size to adapt to the input of the network model, and operations such as adding noise and rotating the image need to be performed. Implementing data augmentation in this way can effectively enhance the robustness of the trained model.

[0097] Step (2), using the above training data to train a feature extraction model;

[0098] This embodiment constructs Figure 1The feature extraction model shown in the figure is cascaded with the classifier, and the feature extraction model is trained with the image classification task using the training data; the feature extraction model consists of a backbone network, a local feature enhancement encoder, a feature fusion module and a global pooling module

[0099] like Figure 1 As shown in the figure, the feature extraction model includes a backbone network, a local feature enhancement encoder, a feature fusion module, and a global pooling module. Each module is introduced below.

[0100] 1) The backbone network uses a convolutional neural network, and ResNet-101 pre-trained on ImageNet is selected; its input is the landmark image, and the output is the initial feature map

[0101] 2) The encoder includes a local feature region cutting module and a local feature enhancement encoding module, which further encodes the output of the backbone network to achieve local feature enhancement; the local feature region cutting module is responsible for expanding each local feature into a local feature map of the area centered on the local feature; the local feature enhancement feature encoding module is responsible for compressing the information in the local feature map of each area into a local vector, so as to achieve the purpose of local feature enhancement.

[0102] 2-1) The local feature region cutting module includes a filling layer and a sliding window cutting layer connected in series. After the vector is input, it first needs to pass through the padding layer. In the padding layer, the feature map is filled with 2 rows or 2 columns of zero data to generate the padded feature map. Then, the layer is cut through the sliding window. The selected sliding window size is 5 and the stride is 1. Converted into cutting feature distribution map The cutting feature distribution diagram It is a series of local feature maps corresponding to 5×5 regions centered on each pixel in the original pixel domain The total number of local feature maps s0 is H×W. In this process, the padding layer ensures that the number of regional feature maps obtained after the sliding window cutting layer is consistent with the number of local feature vectors in the initial feature map. Then, through the sliding window cutting, each local feature vector in the initial feature map is mapped to a corresponding regional feature map, replacing the local features with regional feature maps with broader spatial information, thereby strengthening each local feature.

[0103] 2-2) If Figure 2The local feature enhancement coding module includes two local feature coding layers connected in series, each of which consists of a 3*3 convolutional layer, a normalization layer, and a linear activation layer. The input of the second local feature coding layer is the output of the first local coding layer. Each feature map in the cut feature distribution map After the first local encoding layer, its dimension becomes The information is further extracted and concentrated. The calculation results of this process are as follows:

[0104] s1=max(0,Batchnorm(Conv(s0)))

[0105] The calculation method of the second layer local feature enhancement module is the same as the first layer. After passing through the second layer, each feature map in the feature distribution map is cut The dimension becomes Distribution diagram of the entire cutting feature It becomes an enhanced local feature map with more compact information distribution At this point, the local feature enhancement encoding of the entire initial feature map is completed, and the initial feature map is converted into a local enhanced feature map in which each local feature contains information about a region centered on the vector.

[0106] 3) The feature fusion module is used to linearly fuse the output of the local feature enhancement encoder with the initial feature map features to obtain a local feature map that has both enhanced local feature maps. Excellent spatial local information and initial feature map Fusion feature map of global information Its input consists of two parts: the first part is the updated enhanced local feature map f3 of the encoder, which is used to guide the information focus of the initial feature map f0; the second part is the initial feature map Used to dominate the generation of fusion feature map f4.

[0107] like Figure 3 As shown in the figure, before feature fusion, the enhanced local feature map f3 needs to be normalized to make the initial weights of the two feature maps consistent, and then the channel dimension of the two features is spliced ​​to generate a feature map The calculation process is as follows:

[0108] f4=Concat(f3,f0)

[0109] The generated feature map is then subjected to convolution, ReLU activation, and reconvolution. The first convolution compresses the 0th dimension of feature map f4 to half, generating a feature map with the same dimensions as the enhanced local feature map f3 and the initial feature map f0. The second convolution operation simply performs a linear operation on each eigenvector in the feature map, ultimately generating the fused feature map f5. The calculation results are shown below:

[0110] f5=max(0, (f4×W1)+B1)×W2+B2

[0111] W i and B i are the weights and biases of the i-th convolutional layer, respectively. In this module, all pixel positions are processed individually to reduce the overall computational load;

[0112] 4) The global pooling module is implemented through generalized average pooling, which combines the characteristics of maximum pooling and average pooling, extracting the information in the fusion feature map f5 to generate a global feature The calculation results are as follows:

[0113]

[0114] Among them, f (g) is the output global feature vector, is the eigenvalue of the kth layer, X k is the feature matrix of the kth layer of the input feature map, p k is the exponential constant.

[0115] 2-3) Define the loss function;

[0116] The present invention uses the ArcFace loss function introduced from the face recognition task to train the feature extraction model. The feature extraction model needs to be cascaded with a classifier and then trained according to the image classification task on the training data. ArcFace improves the normalization of the classifier weight vector W and the interval m of the additional angle, thereby enhancing the separability between classes and the compactness within classes. By introducing a smaller intra-class variance, it can show good results for global descriptor learning. Its calculation formula is as follows:

[0117]

[0118] where ω i is the i-th row of the weight vector W, f g is the above eigenvector f (g)The result after normalization. y is the one-hot label vector. For example, if the actual category of the query image is k, the value of the k-th bit of the corresponding one-hot label vector y is 1. γ is the scaling factor. AF represents the adjusted cosine similarity, and its calculation formula is:

[0119] AF(s,c)=(1-c)×s+c×cos(acos(s)+m)

[0120] Where s is the cosine similarity, m is the ArcFace margin, and c is a binary value indicating whether it is the actual category.

[0121] Step (4): Obtain the final retrieval results using a re-ranking method based on query expansion and database enhancement

[0122] The present invention adopts a reordering method that updates the query image feature vector and the image library image feature vector simultaneously. Figure 4 As shown in the figure, after the query image and the image index library pass through the feature extraction model, a global feature corresponding to the query image is obtained and recorded as V q At the same time, each image in the image index library also obtains the corresponding global feature, which is recorded as V i , the global features of all the images in the image index library can be represented in matrix form as an image library feature matrix. This allows the retrieval of similar images based on the image retrieval model. In this embodiment, by multiplying the query image feature vector with the image library feature matrix, the similarity score between the query image and each image in the image index library can be obtained. qi At the same time, by multiplying the image library feature matrix by itself, the similarity scores between images in the image index library can be obtained. qi For example, the calculation formula can be expressed as follows:

[0123] score qi =V q ×V i

[0124] Where V q is the global feature vector of the query image, V i is the global feature vector corresponding to the i-th image in the image index library.

[0125] After obtaining the similarity score, the feature vector of the query image is first updated to achieve query expansion. The specific method is to first update the similarity score qi Sort and find the global feature vector V of the n images that are most similar to the query image from the image library feature matrix (n is a parameter that can be adjusted according to actual conditions in the image retrieval model) iThen the feature vectors of these n images are linearly added to obtain the updated global feature vector V′ of the query image. q , the calculation process is as follows:

[0126]

[0127] Where V q is the initial global feature vector of the query image, α is the exponential coefficient, score qi is the similarity score between the i-th image and the query image among n similar images, V i is the global feature vector of the i-th image among the n similar images to the query image. During this process, the query image's feature vector is updated using the feature vectors of the images in the image library. The feature vectors that are more similar to the query image are weighted, using the similarity score as a weight. Generally speaking, the first search can largely filter out images in the image index library that share the same category as the query image. Reranking can make the query image better represent the category of images rather than just a single image, significantly improving the accuracy of retrieval results.

[0128] In addition to the query image, the global feature vector V of the image in the image index library i It will also be updated. Similar to the above method, for each image P in the image index library, the most similar n images are retrieved from the image index library by similarity score. It should be noted that these n images must include the retrieved image itself. Therefore, in the process of updating its own global feature vector by the global feature vector of its most similar image, it is no longer necessary to use the same method as V′ q The calculation of is the same as that of , and the global feature vector of is added separately. Therefore, for any image P in the image index library, the calculation formula for updating the global feature vector is as follows:

[0129]

[0130] Where: V i is the global feature vector of the i-th image before updating in the n similar images of the image P in the image library, V′ i V i The corresponding global feature vector is updated. By aggregating the feature vectors of a single image into feature vectors similar to those of similar images, retrieval accuracy is greatly enhanced. Once the global feature vectors of all images in the image index are updated, the database enhancement is complete.

[0131] Finally, after query expansion and database enhancement are completed, the image retrieval model can be used again to calculate the updated global feature vector V′ of the query image. q The global feature vector V′ after updating each image P in the image index libraryi The similarity between them is calculated, and then several image library images with the largest similarity are selected and returned as the final retrieval results.

[0132] Based on this, in the image retrieval method based on local feature enhancement and reordering provided by this embodiment, on the one hand, the feature extraction model introduces a local feature enhancement encoder so that each vector in the feature map contains information about its surrounding vectors, thereby enhancing the spatial locality of the feature map. At the same time, the weight of important areas will be enhanced during processing, reducing the interference caused by irrelevant areas. On the other hand, the Angmu feature fusion module fuses the feature map that extracts the local geometric structure information in the area of ​​interest of the feature map with the initial global feature map, allowing the feature map with more spatial local information to guide the query feature map learning, so that the final query feature map focuses more on the information of important areas. Finally, by introducing reordering, the retrieval accuracy is improved without occupying model training resources, and the generalization ability of the model is enhanced.

[0133] The present invention is trained and evaluated on the GLDv2-clean dataset, ROxford5K dataset, RParis6K dataset and Revisitop1m dataset. The detailed methods of training and evaluation are shown in detail below. This embodiment is implemented using the Pytorch deep learning framework. The dataset uses the GLDv2-clean dataset for image enhancement, and 362×362 pixel crops are cut from randomly resized images, and then random color jitter is performed. This embodiment uses a batch size of 32 to train the training model for 30 epochs on a single NVIDIA TitanX 24G GPU. The initial learning rate used in training is 1e -2 , the learning rate decay coefficient is 1e -4 , an SGD optimizer with momentum of 0.9. A linear decay scheduler is used, which gradually decays the learning rate to 0 when the required number of steps is reached. The size d of the global feature is set to 1024. For the ArcFace edge loss, the margin m is empirically set to 0.2 and the scale γ is set to 32.0.

[0134] The trained model in this example will be evaluated on the ROxford5K and RParis6K datasets. The test set images will undergo the same data preprocessing as the training set images before being fed into the model to generate test results. The mAP metric will be used for evaluation.

[0135] mAP is a commonly used metric for evaluating the accuracy of image retrieval algorithms. As the name suggests, mAP is the average of AP, so we need to first calculate AP and then average it. In information retrieval, AP refers to the average of the accuracy rates at different recall rates. However, some current retrieval systems no longer consider recall when calculating AP in order to return results quickly. The calculation results are shown below:

[0136]

[0137] Where X is the set of all images related to the query image in the dataset, x is the ranking of the image in X after image retrieval, and i is the ranking of the image represented by x in X.

[0138] Table 1 Evaluation results on ROxford5K dataset and RParis6K dataset

[0139]

[0140]

[0141] Table 1 shows the evaluation results of various methods on the ROxford5K and RParis6K datasets. It can be seen that the present invention achieves higher overall accuracy than previous methods, including models such as DELG (ECCV'2020), DOLG (IEEE'2021), and FIRe-ASMK (ICLR'2022). From a quantitative analysis perspective, the present invention and FIRe-ASMK achieve similar accuracy on the Medium level of the ROxford5K dataset, but the present invention significantly improves over FIRe-ASMK in other evaluations. The present invention's Hard accuracy on the ROxford5K dataset increased by 4.49, the Medium accuracy on the RParis6K dataset increased by 4.00, and the Hard accuracy on the RParis6K dataset increased by 8.33. Furthermore, on DOLG, except for the Medium metric on the ROxford5K dataset, all other metrics show significant improvements, with improvements of 6.87 and 0.63 on the Hard levels of the ROxford5K and RParis6K, respectively. The present invention is significantly superior to previous methods in terms of the Hard index of the two data sets, indicating that the present invention can better distinguish the differences between two similar images and solve the problem of similar images being difficult to identify.

[0142] Table 2 Evaluation results of adding interference dataset R1M

[0143]

[0144]

[0145] Table 2 shows the evaluation results of the present invention after adding 1,001,001 interference images to the ROxford5K dataset and the RParis6K dataset. Compared with models such as DELG (ECCV'2020) and FIRe-ASMK (ICLR'2022), the present invention still has a higher accuracy, indicating that the present invention has stronger robustness.

[0146] in addition, Figure 5 Some specific search ranking results of the present invention are shown. It can be seen that the present invention can provide more accurate search results and can better solve the problem that the search target in the query image is easily confused with similar images in the image library.

[0147] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. An image retrieval method based on local feature enhancement and re-ranking, characterized in that: include: S1. Obtain images of uniform resolution and size that have been grouped by category as training data for the neural network; S2. cascading the feature extraction model and the classifier, and using the training data to train the feature extraction model with an image classification task; The feature extraction model consists of a backbone network, a local feature enhancement encoder, a feature fusion module and a global pooling module. The input image of the model first passes through the backbone network to extract the initial feature map, and then passes through the local feature enhancement encoder to expand the features of each pixel in the initial feature map into a local feature map corresponding to the area centered on the pixel. All local feature maps are compressed into a feature vector and then recombined into an enhanced local feature map according to the original spatial position relationship of the central pixel. The initial feature map and the enhanced local feature map are then linearly fused by the feature fusion module. The obtained fused feature map is passed through the global pooling module to obtain the global feature. S3. Using the trained feature extraction model, perform feature extraction on the query image and each image in the image index library, and extract corresponding global features from each image. S4. The query image and each image in the image index library are respectively used as the searched images. The image retrieval model calculates the similarity between the images based on the global features of the images, and retrieves several image library images with the highest similarity from the image index library to form a similar image set of the searched image. Then, a re-ranking method based on query expansion and database enhancement is used to fuse and update the global features of all images in each similar image set into the global features of the corresponding searched image itself. Finally, the image retrieval model re-searches similar images of the query image in the image index library based on the fused and updated global features to obtain the final search result. In S4, the process of obtaining the final search results by using the re-ranking method based on query expansion and database enhancement is as follows: S41, inputting the query image into the image retrieval model, calculating the similarity between images based on the global features of the images, and retrieving several image library images with the highest similarity from the image index library to form a first similar image set to the query image; S42: Input each image library image in the image index library into the image retrieval model, calculate the similarity between images based on the global features of the images, and retrieve several image library images with the highest similarity, including the image library image itself, from the image index library to form a second similar image set for each image library image; S43. For all similar images in the first similar image set, using the normalized similarity between each image and the query image as a weight, performing weighted fusion on the global features of all images and superimposing the fused features onto the global features of the query image, thereby updating the global features of the query image. S44. For each image library image in the image index library, perform weighted fusion on the global features of all similar images in the respective second similar image set, where the weighted value is the normalized similarity between each similar image and the current image library image, and use the fused features as the updated global features of the current image library image; S45. The image retrieval model recalculates the similarity between the updated global features of the query image and the updated global features of each image in the image index library, and returns several image library images with the highest similarity as the final retrieval results.

2. The image retrieval method based on local feature enhancement and re-ranking according to claim 1, characterized in that: The images in the training data, the query images, and the images in the image index library are all landmark images.

3. The image retrieval method based on local feature enhancement and re-ranking according to claim 1, characterized in that: The local feature enhancement encoder is formed by cascading a local feature region cutting module and a local feature enhancement encoding module; The local feature region cutting module includes a filling layer and a sliding window cutting layer. The filling layer first fills the periphery of the pixel area of ​​the initial feature map with a width of 2, and then the sliding window cutting layer uses a 5*5 window to slide on the filled area with a step size of 1, extracting the local feature map corresponding to the 5*5 area centered on each original pixel; The local feature enhancement coding module includes two local feature coding layers connected in series, each of which is composed of a 3*3 convolutional layer, a normalization layer and a linear activation layer. After passing through the two local feature coding layers, each local feature map is compressed and converted into a feature vector corresponding to the central pixel of the local feature map. All feature vectors obtained by compression conversion are recombined according to the position of the central pixel in the initial feature map to form an enhanced local feature map.

4. The image retrieval method based on local feature enhancement and re-ranking according to claim 1, characterized in that: In the feature fusion module, the enhanced local feature map is normalized and then spliced ​​with the initial feature map. The obtained spliced ​​features are then passed through a convolution layer, a linear activation layer, and then another convolution layer in sequence to achieve linear fusion of spatial local information and global information, and output a fused feature map.

5. The image retrieval method based on local feature enhancement and re-ranking according to claim 1, characterized in that: When the training data is used to train a feature extraction model for an image classification task, the loss function adopts the ArcFace loss function.

6. The image retrieval method based on local feature enhancement and re-ranking according to claim 1, characterized in that: The backbone network adopts convolutional neural network.

7. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, it can implement the image retrieval method based on local feature enhancement and re-ranking as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by the processor, the image retrieval method based on local feature enhancement and reordering as described in any one of claims 1 to 6 is implemented.

9. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the image retrieval method based on local feature enhancement and reordering as described in any one of claims 1 to 6 when executing the computer program.

Images