Image retrieval method based on local feature enhancement and reordering, medium and equipment
The local feature enhancement encoder and feature fusion module extract the local features of the image, combined with global pooling and reordering technology, the problem of lack of local information and poor generalization ability of feature extraction in the existing technology is solved, and the accuracy of image retrieval is improved Sex and efficiency.
Patent Information
- Application Number
- CN202410979619.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-22
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-07-22
AI Technical Summary
Existing content-based image retrieval technology lacks local information when extracting feature vectors, is susceptible to interference from unrelated information, and has poor generalization capabilities of the model, resulting in a decrease in retrieval accuracy and efficiency.
Using image retrieval methods based on local feature enhancement and reordering, the local feature map is extracted through the local feature enhancement encoder, and linearly fused with the initial feature map, combining with the global pooling module to obtain global features, and simultaneously introducing query extension and database enhancement Reordering technology, updating feature vectors to improve retrieval accuracy.
It improves the accuracy of image retrieval and model generalization ability, reduces the interference of non-important areas on global features, enhances the spatial locality of feature maps, and improves the ability to distinguish similar images.
Smart Images

Figure CN119027694A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision technology, and specifically relates to an image retrieval method based on local feature enhancement and reordering, which is used to output an image in an image library that is closest to information expressed by a given image. Background Art
[0002] Image retrieval is a technology that extracts the required images from the image retrieval database by inputting information. Image retrieval can be applied to search engines, e-commerce, medical and other fields. Therefore, image retrieval technology has great research value.
[0003] The research on image retrieval technology can be divided into text-based image retrieval technology and content-based image retrieval technology according to the different ways of describing the image content. Text-based image retrieval technology uses text descriptions to retrieve images; content-based image retrieval technology uses information such as the color, texture, objects, and categories contained in the image to retrieve images. The input information of content-based image retrieval is more difficult to extract, but it is more intuitive and convenient in application. Therefore, the research on content-based image retrieval has more room for progress, and its research has become more popular in recent years.
[0004] The core of the research on content-based image retrieval is how to extract the best feature vector from the image. The methods for extracting feature vectors are divided into three methods: local features, global features, and combined local features and global features. No matter which method is used, the feature map obtained after the image passes through the neural network will be used as the preliminary processing. The global feature method is to obtain a feature vector by pooling the vector set in the feature map, while the local feature is to select the vectors of the most important areas in the feature map as feature vectors. The combined local and global feature method is to process the feature map with local features and global features at the same time, and then merge the local vector and the global vector or use them separately for image retrieval. In addition to extracting feature vectors, reordering the image library and feature vectors in the process of retrieving images with feature vectors can further improve the accuracy of image retrieval. Reordering is a method that combines the feature vectors of the top-ranked images in the image library based on the results of the first retrieval to replace the query feature vector, and then further performs retrieval.
[0005] There are still many problems to be overcome in the current research methods of content-based image retrieval. Global feature methods often lack local information, so they will be disturbed by irrelevant information in the image during the process of pooling feature maps, which limits the improvement of their accuracy. The processing of local features often requires a lot of computing resources because multiple features are used. It is difficult to propose an effective vector merging method by combining local features and global features, and it is difficult to exert the effects of local features and global features at the same time. Moreover, the model often has poor generalization ability, and its efficiency often drops significantly after the image library is updated. Summary of the invention
[0006] In current image retrieval tasks, the retrieval target in the query image is easily confused with similar images in the image library, which is not conducive to image retrieval. Therefore, how to solve the problem of distinguishing similar but different images and improve the recognition accuracy is a technical problem that needs to be solved urgently. The purpose of the present invention is to solve the technical problems existing in the prior art and propose an image retrieval method based on local feature enhancement and re-ranking.
[0007] In order to achieve the above-mentioned invention object, the technical solution specifically adopted by the present invention is as follows:
[0008] In a first aspect, the present invention provides an image retrieval method based on local feature enhancement and re-ranking, comprising the following steps:
[0009] S1, obtain images with uniform resolution and grouped by category as training data for the neural network;
[0010] S2, cascading the feature extraction model and the classifier, and using the training data to train the feature extraction model with an image classification task; the feature extraction model is composed of a backbone network, a local feature enhancement encoder, a feature fusion module and a global pooling module, the input image of the model first passes through the backbone network to extract the initial feature map, and then passes through the local feature enhancement encoder to expand the features of each pixel in the initial feature map into a local feature map corresponding to the area centered on the pixel, all local feature maps are compressed into a feature vector and then recombined into an enhanced local feature map according to the original spatial position relationship of the central pixel, and then the initial feature map and the enhanced local feature map are linearly fused by the feature fusion module, and the obtained fused feature map is passed through the global pooling module to obtain the global feature;
[0011] S3, using the trained feature extraction model to perform feature extraction on the query image and each image in the image index library, respectively, and extracting corresponding global features from each image;
[0012] S4. The query image and each image library image in the image index library are respectively used as the retrieved images. The image retrieval model calculates the similarity between the images based on the global features of the images, and retrieves several image library images with the highest similarity from the image index library to form a similar image set of the retrieved image; then a reranking method based on query expansion and database enhancement is used to fuse and update the global features of all images in each similar image set into the global features of the corresponding retrieved image itself; finally, the image retrieval model re-retrieves similar images of the query image in the image index library based on the fused and updated global features to obtain the final retrieval result.
[0013] As a preferred embodiment of the first aspect, the images in the training data, the query images and the images in the image index library are all landmark images.
[0014] As a preferred embodiment of the first aspect, the local feature enhancement encoder is formed by cascading a local feature region cutting module and a local feature enhancement encoding module;
[0015] The local feature region cutting module includes a filling layer and a sliding window cutting layer. The filling layer first fills the periphery of the pixel region of the initial feature map with a width of 2, and then the sliding window cutting layer uses a 5*5 window to slide on the filled region with a step length of 1, and extracts the local feature map corresponding to the 5*5 size region centered on each original pixel;
[0016] The local feature enhancement coding module includes two local feature coding layers connected in series, each of which is composed of a 3*3 convolutional layer, a normalization layer and a linear activation layer cascaded; after each local feature map passes through the two local feature coding layers, it is compressed and converted into a feature vector corresponding to the central pixel of the local feature map; all feature vectors obtained by compression conversion are recombined according to the position of the central pixel in the initial feature map to form an enhanced local feature map.
[0017] As a preferred embodiment of the above-mentioned first aspect, in the feature fusion module, the enhanced local feature map is normalized and then spliced with the initial feature map, and then the obtained spliced features are sequentially passed through a convolutional layer, a linear activation layer and then another convolutional layer to achieve linear fusion of spatial local information and global information, and output a fused feature map.
[0018] As a preferred embodiment of the first aspect, when the feature extraction model is trained using the training data for an image classification task, an ArcFace loss function is used as a loss function.
[0019] As a preferred embodiment of the first aspect, in S4, the process of obtaining the final search result by using the re-ranking method based on query expansion and database enhancement is as follows:
[0020] S41, inputting the query image into the image retrieval model, calculating the similarity between images based on the global features of the images, and retrieving several image library images with the highest similarity from the image index library to form a first similar image set of the query image;
[0021] S42, inputting each image library image in the image index library into the image retrieval model, calculating the similarity between images based on the global features of the images, and retrieving several image library images including the image library image with the highest similarity from the image index library to form a second similar image set for each image library image;
[0022] S43, for all similar images in the first similar image set, using the normalized similarity between each image and the query image as a weight, weightedly fused the global features of all images and superimposed the fused features onto the global features of the query image, thereby updating the global features of the query image;
[0023] S44, for each image library image in the image index library, weighted fusion is performed on the global features of all similar images in the respective second similar image set, where the weighted value is the normalized similarity between each similar image and the current image library image, and the fused features are used as the updated global features of the current image library image;
[0024] S45. The image retrieval model recalculates the similarity between the updated global features of the query image and the updated global features of each image library image in the image index library, and returns several image library images with the highest similarity as the final retrieval result.
[0025] As a preferred embodiment of the first aspect above, the backbone network adopts a convolutional neural network.
[0026] In a second aspect, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the image retrieval method based on local feature enhancement and reordering as described in any one of the schemes of the first aspect above.
[0027] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image retrieval method based on local feature enhancement and reordering as described in any one of the schemes of the first aspect above can be implemented.
[0028] In a fourth aspect, the present invention provides a computer electronic device comprising a memory and a processor;
[0029] The memory is used to store computer programs;
[0030] The processor is used to implement the image retrieval method based on local feature enhancement and reordering as described in any one of the solutions of the first aspect when executing the computer program.
[0031] Compared with the prior art, the present invention adopts the above technical solution, and has the following beneficial effects:
[0032] 1) The present invention designs a local feature enhancement encoder, which integrates the surrounding feature information of each feature into the feature compared to the ordinary encoder. In the process of global feature extraction, the global feature is more easily affected by the features of unimportant regions, while the local feature enhancement encoder extracts the local geometric structure information in the region of interest of the feature map, so that the features of the region of interest are given more weight, reducing the interference of unimportant regions on the global features, so that the encoder can more accurately extract the important content in the image.
[0033] 2) The present invention designs a feature fusion module. The feature map generated after the local feature enhancement encoder can enhance the spatial locality but cannot be used as a single retrieval criterion. The feature map that has not passed through the local feature enhancement encoder often ignores the spatial locality. The feature fusion module can retain the advantages of both while eliminating the impact of their disadvantages.
[0034] 3) The present invention introduces a re-ranking technology, which enables the model to update the feature vectors of the query image and the image library image according to the information of the image library, making the query image feature vector more adapted to the image library feature vector, and also improving the generalization ability of the model and the accuracy of the retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 This is a structural architecture diagram of the feature extraction model of the present invention.
[0036] Figure 2 It is a structural architecture diagram of the local feature enhancement encoding module of the present invention.
[0037] Figure 3 This is a structural architecture diagram of the feature fusion module of the present invention.
[0038] Figure 4 It is a structural architecture diagram of the reordering method of the present invention.
[0039] Figure 5 This is an example of the model image retrieval effect of the present invention. DETAILED DESCRIPTION
[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0041] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.
[0042] Aiming at the problem that the retrieval target in the query image in the current image retrieval task is easily confused with similar images in the image library, which is not conducive to image retrieval, the present invention introduces a local feature enhancement encoder that strengthens the spatial correlation of features in the feature map, strengthens the association between each feature in the feature map and its surrounding features, so that each vector in the feature map has wider spatial information; at the same time, by adding a feature fusion module, the feature map after local feature enhancement and the initial global feature map are merged according to learnable weights, and the local feature enhancement feature map can enhance the spatial locality but cannot be used as a single retrieval standard. The feature map is fully utilized, and the feature map is fused with the unprocessed feature map into an image retrieval feature map that embodies the spatial locality. In addition, the present invention also introduces a reordering technology to improve the accuracy of retrieval without participating in model training, so that the generalization ability of retrieval is stronger and can better adapt to different types of images. The specific implementation of the method in the present invention is described in detail below.
[0043] In the present invention, the specific implementation steps of the image retrieval method based on local feature enhancement and re-ranking are as follows:
[0044] S1. Obtain images with uniform resolution and grouped by category as training data for the neural network.
[0045] In the embodiment of the present invention, the images in the training data, the query images and the images in the image index library can all be landmark images, that is, iconic architectural landmarks in different regions. Of course, the present invention is not limited to searching for landmark images, and can also be applied to any other images.
[0046] S2. Cascade the feature extraction model and the classifier, and use the training data to train the feature extraction model with an image classification task; the feature extraction model is composed of a backbone network, a local feature enhancement encoder, a feature fusion module and a global pooling module. The input image of the model first passes through the backbone network to extract the initial feature map, and then passes through the local feature enhancement encoder to expand the features of each pixel in the initial feature map into a local feature map corresponding to the area centered on the pixel. All local feature maps are each compressed into a feature vector and then recombined into an enhanced local feature map according to the original spatial position relationship of the central pixel. The initial feature map and the enhanced local feature map are then linearly fused by the feature fusion module, and the obtained fused feature map is passed through the global pooling module to obtain the global feature.
[0047] In an embodiment of the present invention, the structure of the feature extraction model is as follows: Figure 1 As shown, the backbone network can be implemented by a convolutional neural network; the local feature enhancement encoder is cascaded by a local feature region cutting module and a local feature enhancement encoding module, and the local feature region cutting module is responsible for expanding each local feature into a local feature map of a region centered on the local feature; the local feature enhancement feature encoding module is responsible for compressing the information in the local feature map of each region into a local vector, so as to achieve the purpose of local feature enhancement.
[0048] The local feature region cutting module includes a filling layer and a sliding window cutting layer. The filling layer first fills the periphery of the pixel area of the initial feature map with a width of 2, and then the sliding window cutting layer uses a 5*5 window to slide on the filled area with a step length of 1, and extracts the local feature map corresponding to the 5*5 area centered on each original pixel. The function of the local feature region cutting module is to convert the feature map dimension into a dimension that adapts to the local feature enhancement coding module, and convert the feature vector of each pixel in the feature map output by the backbone network into a feature vector map of a region.
[0049] And as Figure 2 As shown in the figure, the local feature enhancement coding module includes two local feature coding layers connected in series, each of which is composed of a 3*3 convolutional layer, a normalization layer and a linear activation layer (ReLU layer) cascaded; after each local feature map passes through two local feature coding layers, it is compressed and converted into a feature vector corresponding to the central pixel of the local feature map; all feature vectors obtained by compression conversion are recombined according to the position of the central pixel in the initial feature map to form an enhanced local feature map.
[0050] The calculation of each local feature encoding layer in the local feature enhancement encoding module can be expressed as follows:
[0051] s i=max(0,Batchnorm(Conv(s i+1 )))
[0052] where s i Represents the output of the i-th local feature encoding layer. In this process, the feature vector of each pixel is converted into a feature vector map of a region and then converted back into a feature vector, which reduces the amount of subsequent calculations and has more spatial information for each pixel compared to the original features output from the backbone network.
[0053] like Figure 3 As shown, a structural schematic diagram of the feature fusion module is shown. In this feature fusion module, the enhanced local feature map is first normalized and then spliced with the initial feature map. The spliced features are then passed through a convolutional layer, a linear activation layer (ReLU layer) and then a convolutional layer to achieve linear fusion of spatial local information and global information, and a fused feature map is output.
[0054] In an embodiment of the present invention, if the input of the feature fusion module is the initial feature map output from the backbone network And the output of the local feature enhancement encoder Then the output of the local feature enhancement encoder is linearly fused with the initial feature map features to obtain both excellent spatial local information and global information. This process can be expressed by the formula:
[0055] f 6 =max(0,(Concat(f 3 ,f 0 )×W 1 )+B 1 )×W 2 +B 2
[0056] Where: W i and B i are the weight and bias of the i-th convolutional layer respectively, and Concat represents the concatenation operation.
[0057] In this feature fusion module, all pixel locations are processed individually to reduce the overall computational load.
[0058] In an embodiment of the present invention, the feature extraction model needs to be trained in advance using the training data obtained in S1 to meet the extraction accuracy requirement. When training the feature extraction model with an image classification task, the loss function may use the ArcFace loss function.
[0059] S3. Use the trained feature extraction model to extract features from the query image and each image in the image index library, and extract corresponding global features from each image.
[0060] S4. The query image and each image library image in the image index library are respectively used as the retrieved images. The image retrieval model calculates the similarity between the images based on the global features of the images, and retrieves several image library images with the highest similarity from the image index library to form a similar image set of the retrieved image; then a reranking method based on query expansion and database enhancement is used to fuse and update the global features of all images in each similar image set into the global features of the corresponding retrieved image itself; finally, the image retrieval model re-retrieves similar images of the query image in the image index library based on the fused and updated global features to obtain the final retrieval result.
[0061] The above re-ranking method of the present invention introduces query expansion and database enhancement. The core is to find the most similar photos of the query image and each photo in the image library based on the results of the first search, and linearly add the feature vectors of the most similar photos of each photo to obtain the updated query feature vector and image library feature matrix, and finally use the updated features to re-search. This method of using the feature vector of the image in the database to update the feature vector of the image can greatly improve the accuracy of the search results.
[0062] In an embodiment of the present invention, the process of obtaining the final search result by using the re-ranking method based on query expansion and database enhancement is as follows:
[0063] S41, inputting the query image into the image retrieval model, calculating the similarity between images based on the global features of the images, and retrieving several image library images with the highest similarity from the image index library to form a first similar image set of the query image.
[0064] S42, input each image library image in the image index library into the image retrieval model, calculate the similarity between images based on the global features of the images, retrieve several image library images with the highest similarity including itself from the image index library, and form a second similar image set for each image library image.
[0065] S43. For all similar images in the first similar image set, using the normalized similarity between each image and the query image as a weight, weightedly fuse the global features of all images and superimpose the fused features onto the global features of the query image, thereby updating the global features of the query image.
[0066] The essence of the above step S43 is to achieve query expansion, that is, to update the global feature vector of the query image. The method is to first find the feature vectors of the n images most similar to the query image from the feature matrix of the image library by sorting the similarity scores, and then linearly add the feature vectors of these n images to obtain the updated feature vector. The calculation process is as follows:
[0067]
[0068] Where: V′ q and V q are the updated global features and the global features before the update corresponding to the query image respectively; α is the exponential coefficient, score qi is the similarity score between the image in the image library that is similar to the query image and the query image. In this process, the feature vector of the query image is updated by the feature vector of the image library image, and the feature vector that is more similar to the query image has a higher weight according to the similarity score. Generally speaking, the first retrieval can largely filter out the images of the same category as the query image in the image library. The query expansion method can make the query image better represent this type of image rather than a single image, which can greatly improve the accuracy of the retrieval results.
[0069] S44. For each image library image in the image index library, perform weighted fusion on the global features of all similar images in the respective second similar image sets, where the weighted value is the normalized similarity between each similar image and the current image library image, and the fused features are used as the updated global features of the current image library image.
[0070] The essence of the above step S44 is to achieve database enhancement, that is, to update the feature vector of the image library image. Similar to the above query expansion, this database enhancement also obtains the most similar images through similarity scores, and then updates the feature vector through the feature vector of the most similar image. The calculation formula can be expressed as:
[0071]
[0072] Where: V′ i and V i are the updated global features and the global features before the update corresponding to the image in the i-th image index library respectively.
[0073] Similar to query expansion, database enhancement can aggregate the feature vectors of a single image into feature vectors similar to images of this type, thereby greatly enhancing retrieval accuracy.
[0074] S45. The image retrieval model recalculates the similarity between the updated global features of the query image and the updated global features of each image library image in the image index library, and returns several image library images with the highest similarity as the final retrieval result.
[0075] In the embodiment of the present invention, the similarity between images is calculated by the distance values of their respective global features. The similarity can be calculated by multiplying global vectors, or can be represented by cosine distance or the like.
[0076] It should be noted that the number of images in the image library with the highest similarity returned by the image retrieval model in the present invention can be optimized and adjusted according to actual needs and is not limited.
[0077] It should be noted that the above method steps S1 to S4 can essentially be implemented in the form of a computer program.
[0078] Similarly, based on the same inventive concept, the present invention also provides a computer electronic device corresponding to the image retrieval method based on local feature enhancement and re-ranking provided in the above embodiment, which includes a memory and a processor;
[0079] The memory is used to store computer programs;
[0080] The processor is used to implement the above-mentioned image retrieval method based on local feature enhancement and re-ranking when executing the computer program;
[0081] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention.
[0082] Therefore, based on the same inventive concept, the present invention provides a computer-readable storage medium corresponding to an image retrieval method based on local feature enhancement and reordering, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it can implement the image retrieval method based on local feature enhancement and reordering as described above.
[0083] Therefore, based on the same inventive concept, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, can implement the image retrieval method based on local feature enhancement and re-ranking as described above.
[0084] Specifically, in the computer-readable storage medium of the above three embodiments, the stored computer program is executed by the processor to perform the above steps S1 to S4.
[0085] It is understandable that the above storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. The storage medium may also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, etc., which can store program codes.
[0086] It can be understood that the above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0087] It should also be noted that those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules or steps can be combined or integrated together, and a module or step can also be split.
[0088] The image retrieval method based on local feature enhancement and re-ranking shown in S1 to S4 in the above embodiments is applied to a specific example to demonstrate its technical effect.
[0089] Example
[0090] In this embodiment, the steps of image retrieval based on local feature enhancement and re-ranking are the same as the aforementioned S1 to S4, and the specific process and parameter definition can also be found in the previous text. The specific implementation process in the example is shown below, and the detailed method is as follows:
[0091] Step (1), obtaining training data for neural network training.
[0092] 1.1 In this embodiment, the training data is a landmark image with a fixed resolution after the grouped landmark images are preprocessed. The landmark images and the annotations corresponding to the landmark images can be obtained through the existing public dataset GLDv2-clean.
[0093] GLDv2-clean is an image retrieval dataset. GLDv2 contains more than 5 million images of more than 200,000 different landmarks collected by photographers from all over the world. GLDv2-clean is a subset of GLDv2, containing 1.5 million images and 81,313 classes. The landmarks corresponding to each image are annotated. Using such a large number of images to train the model greatly enhances the generalization of the model.
[0094] In order to verify the generalization of the model, the present invention will also use the ROXFORD5K dataset, the RParis6K dataset and the Revisitop1m dataset as auxiliary datasets for evaluation. The original Oxford and Paris datasets consist of 5063 and 6392 high-resolution images, respectively. Each dataset contains 55 queries, and each landmark contains 5 queries from a total of 11 landmarks. ROXFORD5K and RPARIS6K are re-annotated on the Oxford5K and Paris6K data, fixing the previous labeling problems; increasing the number of queries from 55 to 70; reallocating labels; and proposing three types of evaluation methods: Easy\Medium\Hard. The R1M dataset includes 1,001,001 landmark images. In order to further verify the robustness of the model, the present invention mixes the R1M dataset as interference with the ROXFORD5K dataset and the RParis6K dataset to test the robustness of the model under the influence of so many interfering images.
[0095] 1.2 Perform data preprocessing on image retrieval images and transform the resolution to a fixed size to adapt to the input scale of the feature extraction model.
[0096] In the above data preprocessing operations, the resolution of the color image needs to be transformed into a fixed size to adapt to the input of the network model, and the image needs to be added with noise and rotated. Data enhancement in this way can effectively enhance the robustness of the trained model.
[0097] Step (2), using the above training data to train a feature extraction model;
[0098] This embodiment constructs Figure 1 The feature extraction model shown in the figure is cascaded with the classifier, and the feature extraction model is trained with the image classification task using the training data; the feature extraction model is composed of a backbone network, a local feature enhancement encoder, a feature fusion module and a global pooling module
[0099] like Figure 1 As shown in the figure, the feature extraction model includes a backbone network, a local feature enhancement encoder, a feature fusion module, and a global pooling module. Each module is introduced below.
[0100] 1) The backbone network uses a convolutional neural network, and selects ResNet-101 pre-trained on ImageNet; its input is the landmark image and the output is the initial feature map
[0101] 2) The encoder includes a local feature region cutting module and a local feature enhancement encoding module, which further encodes the output of the backbone network to achieve local feature enhancement; the local feature region cutting module is responsible for expanding each local feature into a local feature map of the area centered on the local feature; the local feature enhancement feature encoding module is responsible for compressing the information in the local feature map of each area into a local vector, so as to achieve the purpose of local feature enhancement.
[0102] 2-1) The local feature region cutting module includes a filling layer and a sliding window cutting layer connected in series. After the vector is input, it first needs to pass through the padding layer. In the padding layer, the feature map is filled with 2 rows or 2 columns of zero data to generate the padded feature map. Then, the layer is cut through the sliding window. The selected sliding window size is 5 and the stride is 1. Convert to cutting feature distribution map The cutting feature distribution diagram It is a series of local feature maps corresponding to 5×5 regions centered on each pixel in the original pixel domain. There are a total of H×W local feature maps s 0 In this process, the role of the filling layer is to ensure that the number of regional feature maps obtained after the sliding window cutting layer is consistent with the number of local feature vectors in the initial feature map; then, through sliding window cutting, each local feature vector in the initial feature map is mapped into a corresponding regional feature map, and the local features are replaced by regional feature maps with broader spatial information, so as to achieve the effect of strengthening each local feature.
[0103] 2-2) If Figure 2 The local feature enhancement coding module includes two local feature coding layers connected in series, each of which consists of a 3*3 convolutional layer, a normalization layer and a linear activation layer. The input of the second local feature coding layer is the output of the first local coding layer. After the first local encoding layer, its dimension becomes Its information is further extracted and condensed. The calculation results of this process are as follows:
[0104] s 1 =max(0,Batchnorm(Conv(s 0 )))
[0105] The calculation method of the second layer local feature enhancement module is the same as the first layer. After passing through the second layer, each feature map in the feature distribution map is cut The dimension becomes Distribution diagram of the entire cutting feature It becomes an enhanced local feature map with more compact information distribution At this point, the local feature enhancement encoding of the entire initial feature map is completed, and the initial feature map is converted into a local enhanced feature map in which each local feature contains information about a region centered on the vector.
[0106] 3) The feature fusion module is used to linearly fuse the output of the local feature enhancement encoder with the initial feature map features to obtain a local feature map that has both enhanced local feature maps. Excellent spatial local information and initial feature map The fusion feature map of global information Its input consists of two parts: the first part is the enhanced local feature map f after the encoder is updated 3 Used to guide the initial feature map f 0 The second part of the input is the initial feature map Used to dominate the fusion feature map f 4 Generation of.
[0107] like Figure 3 As shown in the figure, before feature fusion, the local feature map f needs to be enhanced. 3 Normalization is performed to make the initial weights of the two feature maps consistent, and then the two are concatenated in the channel dimension to generate a feature map The calculation process is as follows:
[0108] f 4 =Concat(f 3 ,f 0)
[0109] The generated feature map is then convolved, ReLU activated, and convolved again. In the first convolution process, the feature map f 4 The 0th dimension is compressed to half of it, generating a dimension that is the same as the enhanced local feature map f 3 And the initial feature map f 0 The second convolution operation only performs linear operations on each feature vector in the feature map, and finally generates a fused feature map f 5 , the calculation results are as follows:
[0110] f 5 =max(0,(f 4 ×W 1 )+B 1 )×W 2 +B 2
[0111] W i and B i are the weights and biases of the ith convolutional layer, respectively. In this module, all pixel positions are processed individually to reduce the overall computational load;
[0112] 4) The global pooling module is implemented through generalized average pooling, which combines the characteristics of maximum pooling and average pooling, and fusion feature map f 5 The information in is extracted to generate a global feature The calculation results are as follows:
[0113]
[0114] Among them, f (g) is the output global feature vector, is the eigenvalue of the kth layer, X k is the feature matrix of the kth layer of the input feature map, p k is the exponential constant.
[0115] 2-3) Define the loss function;
[0116] The present invention adopts the ArcFace loss function introduced from the face recognition task to train the feature extraction model. The feature extraction model needs to be cascaded with a classifier and then trained on the training data according to the image classification task. ArcFace improves the normalization of the classifier weight vector W and the interval m of the additional angle, thereby enhancing the separability between classes and the compactness within the class. By introducing a smaller intra-class variance, a good effect can be shown on global descriptor learning, and its calculation formula is as follows:
[0117]
[0118] where ω i is the i-th row of the weight vector W, f g is the above eigenvector f (g) The result after normalization. y is the one-hot label vector. For example, if the actual type of the query image is k, the value of the k-th bit of the corresponding one-hot label vector y is 1. γ is the scaling factor. AF represents the adjusted cosine similarity, and its calculation formula is:
[0119] AF(s,c)=(1-c)×s+c×cos(acos(s)+m)
[0120] Where s is the cosine similarity, m is the ArcFace margin, and c is a binary value indicating whether it is the actual category.
[0121] Step (4): Obtain the final search results based on the re-ranking method of query expansion and database enhancement
[0122] The present invention adopts a reordering method for simultaneously updating the query image feature vector and the image library image feature vector. Figure 4 As shown in the figure, after the query image and the image index library pass through the feature extraction model, a global feature corresponding to the query image is obtained and recorded as V q At the same time, each image in the image index library also obtains the corresponding global feature, which is recorded as V i , the global features of all the images in the image index library can be represented in matrix form as an image library feature matrix. Thus, similar images can be retrieved based on the image retrieval model. In this embodiment, by multiplying the query image feature vector with the image library feature matrix, the similarity score between the query image and each image in the image index library can be obtained. qi At the same time, by multiplying the image library feature matrix by itself, we can get the similarity scores between images in the image index library. qi For example, the calculation formula can be expressed as follows:
[0123] score qi =V q ×V i
[0124] Where V q is the global feature vector of the query image, V i is the global feature vector corresponding to the i-th image in the image index library.
[0125] After obtaining the similarity score, the feature vector of the query image is first updated to achieve query expansion. qiSort and find the global feature vector V of the n images that are most similar to the query image from the image library feature matrix (n is a parameter that can be adjusted according to actual conditions in the image retrieval model) i Then the feature vectors of these n images are linearly added to obtain the updated global feature vector V′ of the query image. q , the calculation process is as follows:
[0126]
[0127] Where V q is the initial global feature vector of the query image, α is the exponential coefficient, and score qi is the similarity score between the ith image and the query image among n similar images, V i is the global feature vector of the ith image among the n similar images of the query image. In this process, the feature vector of the query image is updated by the feature vector of the image library. And according to the similarity score as the weight, the feature vector that is more similar to the query image has a higher weight. Generally speaking, the first retrieval can largely filter out the images of the same category as the query image in the image index library. The re-ranking method can make the query image better represent this type of image rather than a single image, which can greatly improve the accuracy of the retrieval results.
[0128] In addition to the query image, the global feature vector V of the image in the image index library i will also be updated. Similar to the above method, for each image P in the image index library, the most similar n images are retrieved from the image index library through the similarity score. It should be noted that these n images must include the retrieved image itself. Therefore, in the process of updating its own global feature vector through the global feature vector of its most similar image, it is no longer necessary to use V′ q The global feature vector of the image is added separately in the same way as the calculation of the image index library. Therefore, for any image P in the image index library, the calculation formula for updating the global feature vector is as follows:
[0129]
[0130] Where: V i is the global feature vector of the i-th image before updating among the n similar images of the image P in the image library, V′ i V i The corresponding global feature vector after update. The feature vector of a single image is aggregated into a feature vector similar to that of images of this type, thereby greatly enhancing the retrieval accuracy. After completing the update of the global feature vectors of all images in the image index library, the database enhancement is completed.
[0131] Finally, after query expansion and database enhancement are completed, the image retrieval model can be used again to calculate the updated global feature vector V′ of the query image. q The global feature vector V′ updated with each image P in the image index library i The similarity between them is calculated, and then several image library images with the largest similarity are selected and returned as the final retrieval results.
[0132] Based on this, in the image retrieval method based on local feature enhancement and reordering provided in this embodiment, on the one hand, the feature extraction model introduces a local feature enhancement encoder so that each vector in the feature map contains information about its surrounding vectors, thereby enhancing the spatial locality of the feature map. At the same time, the weight of important areas will be enhanced during processing to reduce interference from irrelevant areas. On the other hand, the Angmu feature fusion module fuses the feature map that extracts the local geometric structure information in the area of interest of the feature map with the initial global feature map, allowing the feature map with more spatial local information to guide the query feature map learning, so that the final query feature map focuses more on the information of important areas. Finally, by introducing reordering, the accuracy of retrieval is improved without occupying model training resources, and the generalization ability of the model is enhanced.
[0133] The present invention is trained and evaluated on the GLDv2-clean dataset, ROxford5K dataset, RParis6K dataset and Revisitop1m dataset. The detailed methods of its training and evaluation are specifically shown below. This embodiment is implemented using the Pytorch deep learning framework. The dataset uses the GLDv2-clean dataset for image enhancement, and a 362×362 pixel crop is taken from the randomly resized image, and then random color jitter is performed. This embodiment uses a batch size of 32 to train the training model for 30 epochs on a single NVIDIA TitanX 24G GPU. The initial learning rate used in training is 1e -2 , the learning rate decay coefficient is 1e -4 , an SGD optimizer with momentum of 0.9. A linear decay scheduler is used, which gradually decays the learning rate to 0 when the required number of steps is reached. The size d of the global features is set to 1024. For the ArcFace edge loss, the margin m is set to 0.2 and the scale γ is set to 32.0 based on experience.
[0134] The trained model in this embodiment will be evaluated on the ROxford5K dataset and the RParis6K dataset. The test set images will be input into the model to generate test results after the same data preprocessing operation as the training set images, and the evaluation indicator is mAP.
[0135] mAP is a common indicator used to evaluate the accuracy of image retrieval algorithms. mAP, as the name implies, is the average value of AP, so you need to calculate AP first and then average it. In information retrieval, AP refers to the average value of the accuracy at different recall rates. However, some current retrieval systems no longer consider the recall rate when calculating AP in order to return results quickly. The calculation results are as follows:
[0136]
[0137] Where X is the set of all images related to the query image in the dataset, x is the ranking of the images in X after image retrieval, and i is the ranking of the image represented by x in X.
[0138] Table 1 Evaluation results on ROxford5K dataset and RParis6K dataset
[0139]
[0140]
[0141] Table 1 shows the evaluation results of various methods on the ROXFORD5K dataset and the RParis6K dataset. It can be found that the overall accuracy of the present invention is higher than that of previous methods, including DELG (ECCV'2020), DOLG (IEEE'2021), FIRe-ASMK (ICLR'2022) and other models. From the perspective of quantitative analysis, the accuracy of the present invention and FIRe-ASMK on the Medium of the ROXFORD5K dataset is close, but the present invention has a significant improvement over FIRe-ASMK in other evaluation performances. The Hard accuracy of the present invention on the ROXFORD5K dataset increased by 4.49, the Medium accuracy of the present invention on the RParis6K dataset increased by 4.00, and the Hard accuracy of the present invention on the RParis6K dataset increased by 8.33. At the same time, except for the Medium index of the ROXFORD5K dataset on DOLG, other indicators have been significantly improved, and they have been improved by 6.87 and 0.63 on the Hard of the ROXFORD5K dataset and the RParis6K dataset, respectively. The present invention is significantly better than the previous methods in terms of the Hard index of the two data sets, which shows that the present invention can better distinguish the difference between two similar pictures and solve the problem that similar pictures are difficult to identify.
[0142] Table 2 Evaluation results of adding interference dataset R1M
[0143]
[0144]
[0145] Table 2 shows the evaluation results of the present invention after adding 1,001,001 interference images on the basis of the ROxford5K dataset and the RParis6K dataset. Compared with models such as DELG (ECCV'2020) and FIRe-ASMK (ICLR'2022), the present invention still has a higher accuracy, indicating that the present invention has stronger robustness.
[0146] in addition, Figure 5 Some specific search and ranking results of the present invention are shown, and it can be seen that the present invention can provide more accurate search results and can better solve the problem that the search target in the query image is easily confused with similar images in the image library.
[0147] The above-described embodiment is only a preferred solution of the present invention, but it is not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.
Claims
1. An image retrieval method based on local feature enhancement and re-ranking, characterized in that: include: S1, obtain images with uniform resolution and grouped by category as training data for the neural network; S2, cascading the feature extraction model and the classifier, and using the training data to train the feature extraction model with an image classification task; The feature extraction model is composed of a backbone network, a local feature enhancement encoder, a feature fusion module and a global pooling module. The input image of the model first passes through the backbone network to extract the initial feature map, and then passes through the local feature enhancement encoder to expand the features of each pixel in the initial feature map into a local feature map corresponding to the area centered on the pixel. All local feature maps are compressed into a feature vector and then recombined into an enhanced local feature map according to the original spatial position relationship of the central pixel. The initial feature map and the enhanced local feature map are then linearly fused by the feature fusion module, and the obtained fused feature map is passed through the global pooling module to obtain the global feature. S3, using the trained feature extraction model to perform feature extraction on the query image and each image in the image index library, respectively, and extracting corresponding global features from each image; S4. The query image and each image library image in the image index library are respectively used as the retrieved images. The image retrieval model calculates the similarity between the images based on the global features of the images, and retrieves several image library images with the highest similarity from the image index library to form a similar image set of the retrieved image; then a reranking method based on query expansion and database enhancement is used to fuse and update the global features of all images in each similar image set into the global features of the corresponding retrieved image itself; finally, the image retrieval model re-retrieves similar images of the query image in the image index library based on the fused and updated global features to obtain the final retrieval result.
2. The image retrieval method based on local feature enhancement and re-ranking according to claim 1, characterized in that: The images in the training data, the query images and the images in the image index library are all landmark images.
3. The image retrieval method based on local feature enhancement and re-ranking according to claim 1, characterized in that: The local feature enhancement encoder is formed by cascading a local feature region cutting module and a local feature enhancement encoding module; The local feature region cutting module includes a filling layer and a sliding window cutting layer. The filling layer first fills the periphery of the pixel region of the initial feature map with a width of 2, and then the sliding window cutting layer uses a 5*5 window to slide on the filled region with a step length of 1, and extracts the local feature map corresponding to the 5*5 size region centered on each original pixel; The local feature enhancement coding module includes two local feature coding layers connected in series, each of which is composed of a 3*3 convolutional layer, a normalization layer and a linear activation layer cascaded; after each local feature map passes through the two local feature coding layers, it is compressed and converted into a feature vector corresponding to the central pixel of the local feature map; all feature vectors obtained by compression conversion are recombined according to the position of the central pixel in the initial feature map to form an enhanced local feature map.
4. The image retrieval method based on local feature enhancement and re-ranking according to claim 1, characterized in that: In the feature fusion module, the enhanced local feature map is normalized and then spliced with the initial feature map. The spliced features are then passed through a convolution layer, a linear activation layer, and then another convolution layer in sequence to achieve linear fusion of spatial local information and global information, and a fused feature map is output.
5. The image retrieval method based on local feature enhancement and re-ranking according to claim 1, characterized in that: When the training data is used to train the feature extraction model for an image classification task, the loss function adopts the ArcFace loss function.
6. The image retrieval method based on local feature enhancement and re-ranking according to claim 1, characterized in that: In S4, the process of obtaining the final search results by using the re-ranking method based on query expansion and database enhancement is as follows: S41, inputting the query image into the image retrieval model, calculating the similarity between images based on the global features of the images, and retrieving several image library images with the highest similarity from the image index library to form a first similar image set of the query image; S42, inputting each image library image in the image index library into the image retrieval model, calculating the similarity between images based on the global features of the images, and retrieving several image library images including the image library image with the highest similarity from the image index library to form a second similar image set for each image library image; S43, for all similar images in the first similar image set, using the normalized similarity between each image and the query image as a weight, weightedly fused the global features of all images and superimposed the fused features onto the global features of the query image, thereby updating the global features of the query image; S44, for each image library image in the image index library, weighted fusion is performed on the global features of all similar images in the respective second similar image set, where the weighted value is the normalized similarity between each similar image and the current image library image, and the fused features are used as the updated global features of the current image library image; S45. The image retrieval model recalculates the similarity between the updated global features of the query image and the updated global features of each image library image in the image index library, and returns several image library images with the highest similarity as the final retrieval result.
7. The image retrieval method based on local feature enhancement and re-ranking according to claim 1, characterized in that: The backbone network adopts a convolutional neural network.
8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the image retrieval method based on local feature enhancement and re-ranking as described in any one of claims 1 to 7 can be implemented.
9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the image retrieval method based on local feature enhancement and re-ranking as claimed in any one of claims 1 to 7 is implemented.
10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the image retrieval method based on local feature enhancement and re-ranking as described in any one of claims 1 to 7 when executing the computer program.
Citation Information
Patent Citations
Image retrieval method and system based on global and local feature rearrangement
CN112966137A
Precise retrieval method for local or combination of online shopping commodity images
CN114610974A
Color film matching method based on image classification retrieval and related device
CN116051870A
Domestic garbage image classification model training method and device based on comparative learning
CN116704293A
Person re-identification method and apparatus for fusing global features with ladder-shaped local features
WO2024021394A1
Cited By
Image feature similarity calculation method and system based on similar feature attention
CN119832278A
