An image retrieval method based on multi-scale feature enhancement
The image retrieval method using multi-scale feature enhancement solves the problems of noise interference and annotation in image retrieval by utilizing CNN networks and distance adjustment of agents in the feature space, thereby improving feature discriminativeness and retrieval accuracy.
Patent Information
- Application Number
- CN202311071890.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-24
- Publication Date
- 2025-10-24
- Estimated Expiration
- 2043-08-24
AI Technical Summary
Existing technologies suffer from severe noise interference in image retrieval, leading to inaccurate feature extraction and affecting retrieval performance. Furthermore, localization methods require additional manual annotation or have poor localization results, while metric learning methods are not entirely satisfactory.
A CNN network is used to extract multi-layer features, and a multi-scale feature-enhanced image similarity measurement network is generated through global max pooling and layer normalization. A proxy is used to narrow the distance between similar classes and widen the distance between dissimilar classes in the feature space, and the training is optimized by combining a metric loss function.
It improves the accuracy and efficiency of image retrieval, enhances the discriminative power of features, and achieves improved retrieval performance across multiple datasets.
Smart Images
Figure CN117150064B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of image retrieval, and particularly relates to an image retrieval method based on multi-scale feature enhancement. BACKGROUND
[0002] Image retrieval is an important task of computer vision, and its goal is to return pictures of the same category as the retrieval picture in the data set. Image retrieval has a wide range of applications in e-commerce platforms and species protection, and how to accurately return pictures of the same category as the retrieval picture in the data set is a crucial step. The usual approach is to use a metric learning method to guide the network to learn the discriminative features of the picture.
[0003] Traditional methods based on metric learning design various metric loss functions by considering different data relationships to guide the network learning, so that the network learns the discriminative features of the image. However, the traditional metric-based method has a significant disadvantage, that is, because the entire picture is used as input to extract features, the features contain a lot of noise due to the large amount of background noise in the picture, thereby reducing the retrieval performance.
[0004] To solve this problem, some scholars propose to use positioning methods to remove the interference of background noise and improve retrieval performance. Some scholars use an unsupervised way to locate objects in the image to remove the interference of background noise, but the positioning effect is not satisfactory due to the lack of object position label information to guide learning, and the effect is not good. Some scholars manually label the parts common to the object (such as the wings and beak of a bird), and distinguish the object by comparing the differences between these parts, but this method of providing position information to guide network learning through manual labeling is too cumbersome and cannot be applied to large data sets.
[0005] Although the positioning method has achieved good retrieval performance, it requires additional manual labeling, and although the metric learning method does not require additional manual labeling, the effect is not satisfactory. Therefore, how to obtain better retrieval results without increasing additional manual labeling is an important issue. SUMMARY
[0006] The application aims to provide an image retrieval method based on multi-scale feature enhancement to solve the above problems. The method uses CNN to extract multi-layer features to obtain feature representations containing more information, improves the discriminative ability of the features, and improves the retrieval performance.
[0007] The technical solution adopted by the application is as follows:
[0008] The application discloses an image retrieval method based on multi-scale feature enhancement, and has the characteristics that the method comprises the following steps:
[0009] Step 1, establishing a multi-scale feature enhancement image similarity measurement network model: taking a CNN network as a backbone network, and sequentially arranging a global maximum pooling layer and a layer normalization layer after the backbone network to form the multi-scale feature enhancement image similarity measurement network model.
[0010] Step 2, training the multi-scale feature enhancement image similarity measurement network model:
[0011] Step 2.1, inputting training pictures into the multi-scale feature enhancement image similarity measurement network model, and extracting preliminary features of the pictures by using the CNN network; the preliminary features are features extracted by the last N layers of the CNN network, and 4 >= N >= 2; wherein, the features extracted by the last n layer are expressed as F n (C n W n H n ), C n , W n , H n respectively represent the dimension, width and height of the features extracted by the last n layer, and n = 1, 2,..., N.
[0012] Step 2.2, performing pooling on each layer in the preliminary features by using the global maximum pooling layer to obtain low-dimensional features of each layer, and the low-dimensional features of the last n layer are expressed as F n (C n， 1,1).
[0013] Step 2.3, performing normalization operation on the low-dimensional features of each layer by using the layer normalization layer, and finally obtaining N layer final feature vectors.
[0014] Step 2.4, each layer in the N layer final feature vectors corresponds to a feature space, M agents are randomly generated in each feature space, the value of M is consistent with the number of categories of the training set pictures, and each agent corresponds to a category in the training set pictures.
[0015] Step 2.5, in the feature space corresponding to each layer, the distance between the final feature vector of the layer and the agent of the same category is pulled in, and the distance between the final feature vector of the layer and the agents of other categories is pulled away; when the distances between the final feature vectors of all layers and the agents of the same category are stable, the trained multi-scale feature enhancement image similarity measurement network model is obtained.
[0016] Step 3, retrieval process: input the picture to be retrieved into the trained multi-scale feature enhanced image similarity measurement network model, obtain the N-layer final feature vector, then splice the N-layer final feature vector to obtain the final feature representation of the picture, and use the final feature representation to calculate the similarity between pictures and return the pictures with the highest similarity in turn.
[0017] Further, in the first 3-5 training rounds of the training process, the network model parameters are fixed to adjust the position of the agent in the feature space.
[0018] Further, in the training process, the metric loss is used for tuning; specifically, in each feature space, the loss value is calculated according to the metric loss function of formula (1):
[0019]
[0020]
[0021] Where, A n is the set of M agents in the last n-layer feature space, x n represents the final feature vector of the last n-layer; f(x n ) represents the vector of the agent corresponding to the picture of the same category as x n , ||a||2 represents the L2 norm of the vector a, -d(x n , a) represents the Euclidean distance between the final feature vector x n and the vector a. A loss is calculated in each feature space, and all the losses are added to obtain the total loss value Loss.
[0022] The image retrieval method based on multi-scale feature enhancement provided by the application adopts a CNN network to extract multi-layer features of a picture and processes each layer feature through a global maximum pooling layer and a layer normalization layer to obtain a final feature vector of each layer. Through training, each layer final feature vector has distinguishability, and when retrieving, they are spliced to obtain a more distinguishable feature representation, so as to improve the retrieval effect. BRIEF DESCRIPTION OF DRAWINGS
[0023] Figure 1 The neural network structure used in the embodiment is shown in the schematic diagram.
[0024] Figure 2 The schematic diagram of the relationship between data and agents. DETAILED DESCRIPTION
[0025] The technical solutions of the application will be described in detail below with reference to the drawings and specific embodiments:
[0026] An image retrieval method based on multi-scale feature enhancement, comprising the following steps:
[0027] Step 1, establishing a multi-scale feature enhancement image similarity measurement network model: taking Resnet50 network as the backbone network, and sequentially setting a global maximum pooling layer and a layer normalization layer after the backbone network to constitute a multi-scale feature enhancement image similarity measurement network model.
[0028] Step 2, training process:
[0029] The CUB200-2011, Cars196 and FGVC data sets are divided respectively, and the first 100 classes (5864 pictures) of CUB200-2011 are taken for training, and the last 100 classes (5924 pictures) are taken for testing; the first 98 classes (8054 pictures) of Cars196 are taken for training, and the last 98 classes (8131 pictures) are taken for testing; the first 50 classes (5000 pictures) of FCVC are taken for training, and the last 50 classes (5000 pictures) are taken for testing.
[0030] Step 2.1, input the training pictures divided from the above three data sets into the multi-scale feature enhancement image similarity measurement network model in turn, and use the Resnet50 network to extract the preliminary features of the pictures; the preliminary features are the features extracted by the last two layers of the CNN network, wherein the features extracted by the last n layer are represented as F n (C n W n H n ), C n , W n , H n respectively represent the dimension, width and height of the features extracted by the last n layer, n=1,2;
[0031] Step 2.2, the global maximum pooling layer is used to pool each layer in the preliminary features respectively to obtain the low-dimensional features of each layer, and the low-dimensional features of the last n layer are represented as F n (C n， 1,1).
[0032] Step 2.3, the layer normalization layer (Layer normalization) is used to normalize the low-dimensional features of each layer respectively, and finally the final feature vectors of the last two layers are obtained.
[0033] Step 2.4, in the final feature vectors of the last two layers, each layer corresponds to a feature space, and M agents are randomly generated in each feature space, the value of M is consistent with the number of categories of the training set pictures, and each agent corresponds to a category in the training set pictures.
[0034] Step 2.5, in each layer corresponding feature space, pull the distance between the final feature vector of the layer and the same class agent, and pull away the distance between the final feature vector of the layer and other class agents; when the distance between the final feature vector of all layers and the same class agent is stable, then the trained multi-scale feature enhanced image similarity measurement network model is obtained.
[0035] Step 3, retrieval process: input the picture to be retrieved into the trained multi-scale feature enhanced image similarity measurement network model, obtain the final feature vector of the last two layers, then splice the final feature vector of the last two layers to obtain the final feature representation of the picture, and use the final feature representation to calculate the similarity between pictures and return the pictures with the highest similarity in turn.
[0036] In the first 5 training rounds of the training process, the network model parameters are fixed to adjust the position of the agent in the feature space.
[0037] In the training process, the metric loss is used for optimization; specifically, in each feature space, the loss value is calculated according to the metric loss function of formula (1):
[0038]
[0039]
[0040] Where, A n is the set of M agents in the nth layer from the end feature space, x n represents the final feature vector of the nth layer from the end; f(x n ) represents the vector returning the same class agent of the picture corresponding to x n , ||a||2 represents the L2 norm of vector a, -d(x n , a) represents the Euclidean distance between the final feature vector x n and the vector a. A loss is calculated in each feature space, and all the losses are added to obtain the total loss value Loss.
[0041] The effect of the model on the corresponding test set after being trained by the CUB200-2011 training pictures: R@1 (75.5%), R@2 (84.7%), R@4 (90.4%), R@8 (94.4%); the effect of the model on the corresponding test set after being trained by the Cars196 training pictures: R@1 (93.4%), R@2 (96.4%), R@4 (97.9%), R@8 (98.9%); the effect of the model on the corresponding test set after being trained by the FGVC training pictures: R@1 (84.1%), R@2 (90.8%), R@4 (94.8%), R@8 (97.2%). The image retrieval method based on multi-scale feature enhancement has good effects on multiple data sets, and the performance on some data sets exceeds all existing methods, which shows that the application can effectively improve the retrieval effect.
[0042] Although the above describes the specific embodiments of the application in order to facilitate the understanding of the application by those skilled in the art, it should be clear that the application is not limited to the scope of the specific embodiments. Any changes that are obvious or equivalent are obvious, and all inventions using the concept of the application are protected.
Claims
1. A method of image retrieval based on multi-scale feature enhancement, characterized in that, The method comprises the following steps: Step 1, establishing a multi-scale feature enhanced image similarity measurement network model: taking a CNN network as a backbone network, and sequentially setting a global maximum pooling layer and a layer normalization layer after the backbone network to constitute the multi-scale feature enhanced image similarity measurement network model; Step 2, training the multi-scale feature enhanced image similarity measurement network model: Step 2.1, input the training picture into the multi-scale feature enhanced image similarity measure network model, and use the CNN network to extract the preliminary feature of the picture; the preliminary feature is the feature extracted by the last N layers of the CNN network, 4≥N≥2; wherein the feature extracted by the last n layer is expressed as F n (C n W n H n ), C n , W n , H n respectively represent the dimension, width and height of the feature extracted by the last n layer, n=1,2,...,N; Step 2.2, the global maximum pooling layer is used to pool each layer of the preliminary features respectively to obtain low latitude features of each layer, and the low latitude features of the last n layer are expressed as F n (C n， 1,1); Step 2.3, performing a normalization operation on low-latitude features of each layer by using the layer normalization layer, and finally obtaining N-layer final feature vectors; Step 2.4, in the N-layer final feature vectors, each layer corresponds to a feature space, M agents are randomly generated in each feature space, the value of M is consistent with the number of categories of the training set pictures, and each agent corresponds to a category in the training set pictures; Step 2.5, in the feature space corresponding to each layer, the distance between the final feature vector of the layer and the agent of the same category is pulled in, and the distance between the final feature vector of the layer and the agent of other categories is pulled away; when the distance between the final feature vectors of all layers and the agent of the same category is stable, the trained multi-scale feature enhanced image similarity measurement network model is obtained; Step 3, the retrieval process: inputting a picture to be retrieved into the trained multi-scale feature enhanced image similarity measurement network model, obtaining N-layer final feature vectors, then splicing the N-layer final feature vectors to obtain a final feature representation of the picture, and using the final feature representation to calculate the similarity between pictures and sequentially returning pictures with the highest similarity.
2. The image retrieval method based on multi-scale feature enhancement of claim 1, wherein, In the training process, the metric loss is used for tuning; specifically, in each feature space, the loss value Loss is calculated according to the metric loss function of formula (1) n : where A n is the set of M agents in the n-th last feature space, x n denotes the final feature vector of the n-th last layer; f(x n ) denotes the vector returning the agents of the same category as the picture corresponding to x n , ||a||2denotes the L2 norm of vector a, -d(x n , a) denotes the Euclidean distance between the final feature vector x n and the vector a; a loss is calculated in each feature space, and all the losses are added to obtain the total loss value Loss.
3. The image retrieval method based on multi-scale feature enhancement according to claim 1 or 2, characterized in that, In the first 3-5 training rounds of the training process, the network model parameters are fixed to adjust the positions of the agents in the feature space.