Image retrieval and recognition method based on autonomous evolutionary loss

Abstract

The application discloses an image retrieval and identification method based on autonomous evolution loss, and proposes a brand-new Softmax metric loss function, which shares parameters with an original Softmax loss function, is similar in form, and has three main differences: different gamma, L2 normalized features, and W of stopping gradient update j . Since the features used in the gradient stopping Softmax loss function are L2 normalized, the distance metric in the training stage is consistent with that in the test stage, and sharing parameters with the original Softmax loss function enables the network to obtain good representation (class center), thereby solving the problem of difficult convergence in training; the gradient of the center is stopped from updating in the gradient stopping Softmax loss function, but the sample features are not stopped from gradient update, and such a setting can force the sample features to be close to the class center on a high sphere. The application better solves the influence of only learning the Softmax loss function on the learning effect of the model when the deep metric learning is applied to the image retrieval task.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image retrieval, and particularly relates to an image retrieval and recognition method based on autonomous evolutionary loss. BACKGROUND

[0002] The basic form of image retrieval task is to give a query image containing a specific instance (such as a specific target, scene, building, etc.), and then find images containing the same instance from the database images. Deep metric learning is one of the important methods widely used in image retrieval tasks.

[0003] In the autonomous evolutionary metric learning task, deep metric learning (DML) aims to learn a similarity metric that can map samples to a high-dimensional space. In the high-dimensional space, samples of the same instance are closer, while samples of different instances are farther apart. Typical deep metric learning applications include image retrieval, person re-identification, etc. Popular methods of deep metric learning include pair-based methods and Softmax-based methods. Pair-based methods focus on finding effective methods to improve the sample weighting strategy of existing pair losses (such as contrastive loss and triplet loss). Pair-based methods directly affect the distance between point pairs in the embedding space, which is closely related to the goal of DML. As for Softmax-based methods, some existing methods consider that using Softmax loss to train the model can also achieve good performance. Compared with pair-based methods, Softmax-based methods can be regarded as using proxies to approximate each class and using all proxies to provide global context for each training iteration.

[0004] Existing research has found that optimizing Softmax-based methods corresponds to the approximate bound optimizer of basic pair loss, indicating that minimizing the Softmax loss is equivalent to maximizing the mutual information between features and labels. In fact, when training a deep metric learning model based on Softmax, the inner product without L2 normalization (i.e., the last fully connected layer) is the most widely used similarity metric, but the features are usually L2 normalized at the test stage, which means that the distance metric used during training is different from the distance used at the test stage. To make up for this gap, a simple method is to use L2 normalization directly during training. However, the introduction of L2 normalization standardization makes it difficult for the network to converge, resulting in training failure.

[0005] Existing researches consider that this is mainly because the range of the L2 normalized inner product output is only [-1, 1], which prevents the probability distribution from approaching 1 even if the samples are well separated. In order to deal with this convergence problem, researchers try to append a scaling layer after the inner product. The scaling layer has a learnable parameter to scale the inner output to a larger value instead of 1, thereby promoting the Softmax loss to continue to decrease, thus helping the network to converge. However, this method cannot guarantee that the network can learn the optimal scaling parameter. SUMMARY

[0006] In view of the defects of the prior art, the present application aims to provide an image retrieval and identification method based on autonomous evolution loss, which solves the problem that only learning the Softmax loss function affects the learning effect of the model when applying deep metric learning to image retrieval tasks.

[0007] The present application is realized by the following technical solutions:

[0008] The image retrieval and identification method based on autonomous evolution loss comprises the following steps:

[0009] (1) using ResNet50 model as the backbone network, and pre-training on the ImageNet large classification dataset;

[0010] (2) replacing the global average pooling with the generalized mean pooling, adding a batch normalization layer without scaling item and bias item to the backbone network, removing the last BN-ReLU module in the backbone network, and using L2 normalized Euclidean distance to calculate the recall rate in the test;

[0011] (3) adjusting all input images to 256*256 resolution and cropping to 224*224 resolution, and not performing data enhancement operation on the input data, but only sampling the input data to 256*256 image size;

[0012] (4) training the model for 100 rounds, setting the parameter learning rate by using the cosine annealing algorithm, and setting γ=30 as the default value;

[0013] (5) using label smoothing for the Softmax loss function, and when the return value of the Softmax loss function is less than 3, the gradient stops the Softmax loss function from starting to join the training;

[0014] The gradient stopping Softmax loss function is shown in the following formula (2):

[0015]

[0016] where N is the number of samples in each input batch, c is the number of classes in the training set, f is the feature vector of the input image, and y is the label of the input image.i y is the feature of the i-th sample i y is the label of the i-th sample, W j is the j-th column of the last fully connected layer, corresponding to the j-th class, denotes L2 normalization, while denotes that the gradient is not allowed to pass through W j is a predefined scalar; Softplus(x) = log(1 + e x ),

[0017] (6) Fix the model parameters and no longer update the network parameters through the stochastic gradient descent algorithm, and only use the network as an image feature extractor;

[0018] (7) Output the features F of the ResNet-50 feature extraction network deployed with the gradient stop Softmax loss function and the removed BN-ReLU module;

[0019] (8) Obtain the feature F q of the query sample after the model feature inference, and extract the features of all images in the image library as a feature sequence {F1,…,F m};

[0020] (9) Calculate the Euclidean distance between the query sample feature F q and the features of all images in the image library: d = ||F q -F i ||2, i = 1, 2, 3,…, m;

[0021] (10) Obtain the distance sequence D = [d1, d2,…, d m ];

[0022] (11) Reorder the distance D by distance, and take the L closest images to the query sample. If there is an image with the same ID as the query sample in the image, it is considered that the image retrieval is successful.

[0023] Further, the step (5) combines the Softmax loss function and the gradient stop Softmax function, and the total loss is as follows:

[0024] L = L softmax + L SGSL (1)

[0025] The gradient stop Softmax loss function and the original Softmax loss function share parameters.

[0026] Further, the image library is composed as follows:

[0027] CUB-200-2011 has 200 classes, 11788 images, the first 100 classes 5864 images are used for training, and the remaining 5924 images are used for testing;

[0028] CAR-196 has 198 classes and 16,185 images, the first 98 classes are used for training 8054 images, and the other 98 classes are used for testing 8131 images;

[0029] Stanford Online Products has 22634 classes and then 120053 images, the first 11318 classes 59551 images are used for training, and the other 11316 classes 60502 images are used for testing;

[0030] In-shop clothing has 50 fine-grained classes and 1000 attributes, containing more than 800,000 images.

[0031] The image retrieval and recognition method based on autonomous evolutionary loss is proposed, a new class Softmax metric loss function is proposed, which shares parameters with the original Softmax loss function, they are similar in form, but there are three main differences: different gamma, L2 normalized features and W j Because the features used in the gradient stopping Softmax loss function are L2 normalized, the distance metric in the training stage is consistent with that in the test stage, and sharing parameters with the original Softmax loss function also enables the network to obtain good representation (class center), thereby solving the problem of difficult convergence in training.

[0032] At the same time, the gradient of the class center is stopped in the gradient stopping Softmax loss function, but the sample feature will not stop the gradient update, such setting can force the sample feature to approach the class center on the high sphere. Better solve the influence of only learning Softmax loss function on the learning effect of the model when applying deep metric learning to image retrieval task. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The overall flow chart of the image retrieval method;

[0034] Figure 2 The schematic diagram of the original Softmax loss function and the gradient stopping Softmax loss function proposed in the application; DETAILED DESCRIPTION

[0035] The application will be further described in detail below in combination with specific embodiments, which are an explanation of the application rather than a limitation.

[0036] As Figure 1As shown, the present application provides an image retrieval and identification method based on autonomous evolution loss, and the specific implementation process is as follows:

[0037] (1) ResNet50 is used as the backbone network, and pre-training is performed on the ImageNet large classification data set;

[0038] (2) Replace the global average pooling with the generalized mean pooling, add a batch normalization layer without scaling and bias to the backbone network, and use L2 normalized Euclidean distance to calculate the recall rate during testing;

[0039] (3) All input images are adjusted to 256*256 resolution and cropped to 224*224 resolution, and the batch input sample number is 64 images (4 images per ID input, a total of 16 IDs). No data enhancement operation is performed on the input data, only the input data is sampled to 256*256 image size;

[0040] (4) The model is trained for 100 rounds, and the cosine annealing algorithm is used to set the parameter learning rate, and the default value of gamma=30 is set;

[0041] (5) Label smoothing is used for the Softmax loss function, and when the return value of the Softmax loss function is less than 3, the gradient stops the Softmax loss function and starts to join the training;

[0042] (6) Fix the model parameters, no longer update the network parameters through the stochastic gradient descent algorithm, and only use the network as an image feature extractor.

[0043] (7) In the actual inference process, the output feature F of the ResNet-50 feature extraction network deployed with the gradient stop Softmax loss function and applied with the BN-ReLU technology is removed.

[0044] (8) For the query example, the feature F obtained after model feature inference q , the present application extracts the features of all images in the image library and stores them as a feature sequence {F1,…,F m}.

[0045] (9) Calculate the Euclidean distance d of F q and the features of all images in the image library: d=||F q -F i ||2,i=1,2,3,…,m

[0046] (10) Get the distance sequence D=[d1,d2,…,d m ],

[0047] (11) The present application reorders the distance D, and takes the L closest images to the query example, and if there is an image with the same ID as the query example in these images, the image retrieval is considered successful.

[0048] As shown in Figure 2 , the present application scheme innovation mainly includes two parts of gradient stop Softmax loss function and removal of BN-ReLU technology.

[0049] 1. Gradient stop Softmax loss function

[0050] Gradient stop Softmax loss function (Stop-Gradient Softmax Loss, SGSL), when training a classification network for metric learning, many existing methods will remove the bias term in the last fully connected layer, and follow this setting. In order to better understand the method of the present application, the original Softmax and its variants are briefly reviewed here. The original bias-free Softmax loss function is shown in formula (1) as follows:

[0051]

[0052] Where N is the number of samples in each input batch, c is the number of classes in the training set, f i is the feature of the i-th sample, y i is the label of the i-th sample. W j is the j-th column of the last fully connected layer, corresponding to the j-th class. In addition, there are Softplus(x)=log(1+e x ) and

[0053] The gradient stop Softmax loss function formula proposed by the present application is similar to the standard Softmax loss function, but has three differences:

[0054] 1) The γ in the gradient stop Softmax loss function formula is not fixed as 1, but a larger value, γ can be regarded as a scaling parameter that controls the loss, but unlike the traditional scaling parameter, the γ of the present application is only added when W j and f i come from different classes;

[0055] 2) W j and f i are L2 normalized;

[0056] 3) The present application does not allow gradient update through W j .

[0057] Therefore, the gradient stopping Softmax loss function proposed by the present application is defined as formula (2) as follows:

[0058]

[0059] wherein represents L2 normalization, and represents that gradient update is not allowed through W j , and γ is a predefined scalar. The meanings of other symbols are the same as formula (1). In the experiment, the original Softmax loss function and the gradient stopping Softmax function proposed by the present application are used in combination, and the total loss is as follows:

[0060] L = L softmax + L SGSL (3)

[0061] As Figure 2 shown in the schematic diagram of the original Softmax loss function and the gradient stopping Softmax loss function proposed by the present application, the gradient stopping Softmax loss function and the original Softmax loss function share parameters. The principle of action of the gradient stopping Softmax loss function proposed by the present application is analyzed. Softplus(x) = log(1+e x ) is a convex monotonically increasing function, which can be regarded as a smooth version of the positive part function max(0,x), and the so-called logarithmic exponential, i.e. , is a function form often encountered in dynamic discrete selection models, which can be regarded as a smooth version of the function of selecting the maximum in a set of data. The greater γ is, the smaller the error is, so the present application sets γ = 30 in formula (2) as the default value.

[0062] Based on the above analysis, formula (2) can be approximately expressed as:

[0063]

[0064] wherein [·] + represents max([·],0), represents that gradient update is not allowed through W j , and the cosine similarity is a normalized version of the inner product of two vectors, which is used to measure the similarity between features that is independent of the size, and is equivalent to the L2 normalized Euclidean distance.

[0065] As can be seen from formula (4), the optimization goal of the gradient stopping Softmax loss function is to make the cosine similarity between f i and W j greater than the cosine similarity between f i and the maximum cosine similarity between them. In other words, the gradient stop Softmax loss function requires the features learned by the network to be closer to the representation of its own class (here, the distance is measured using L2 normalized Euclidean distance) and farther from the representation of other classes, which is closely related to the goal of deep metric learning. In addition, the gradient stop Softmax loss function proposed by the present application does not allow gradient updates through W, so the convergence difficulty problem caused by the application of the gradient stop Softmax loss function in the actual training learning of the model is less.

[0066] 2、Remove BN-ReLU technology

[0067] In deep metric learning, the feature extraction network ResNet50 without the last fully connected layer is often used as the backbone network. Many methods add batch normalization (BN) without scaling and bias terms on top of the backbone network, because it can smooth and normalize the feature distribution, and enhance the compactness within the class. However, such processing will make the last three layers of the backbone network become the form of BN-ReLU-BN, which will increase the learning burden of the model. Adding a continuous BN-ReLU module will not bring any new information to the output features, but may lose some useful information for metric learning. Therefore, the present application applies the BN-ReLU removal technology to remove the last BN-ReLU module in the backbone network.

[0068] 1、Dataset selection

[0069] In terms of dataset setting, the present application uses a general image retrieval dataset to evaluate the recognition ability of the method of the present application in the fine-grained image retrieval task, and analyzes the performance improvement brought by the gradient stop Softmax loss function proposed by the present application. Specifically, the dataset used by the present application and its composition are as follows:

[0070] 1) CUB-200-2011 has 200 classes and 11788 images. The first 100 classes (5864 images) are used for training, and the remaining classes (5924 images) are used for testing.

[0071] 2) CAR-196 has 198 classes and 16,185 images. The first 98 classes are used for training (8054 images), and the other 98 classes (8131 images) are used for testing.

[0072] 3) Stanford Online Products has 22634 classes and 120053 images. The first 11,318 classes (59551 images) are used for training, and the other 11316 classes (60502 images) are used for testing.

[0073] 4) In-shop clothing is a large-scale clothing dataset with comprehensive annotations. It has 50 fine-grained categories and 1000 attributes, and contains more than 800,000 images, which are annotated with a large number of attributes, clothing landmarks and corresponding images taken in different scenes (including stores, street snapshots and consumers).

[0074] 2. Implementation details

[0075] The experiments of the present application were completed using the PyTorch deep learning framework on an NVIDIA GTX 2080Ti graphics processor. The present application uses different numbers of graphics processors for training according to the size of the dataset. Specifically, 4 graphics processors are used for training on the Stanford Online Products dataset, and 2 graphics processors are used for training on other datasets. All experiments use ResNet50 as the backbone network, and are pre-trained on the ImageNet large-scale classification dataset, using generalized mean pooling instead of global average pooling. Similar to most current deep metric learning methods, a batch normalization layer without scaling and bias is added to the backbone network, and L2 normalized Euclidean distance is used to calculate the recall rate at test time.

[0076] In terms of parameter settings, all input images are adjusted to a resolution of 256*256 and cropped to a resolution of 224*224, with a batch input sample size of 64 images (4 images per ID input, for a total of 16 IDs). The model is trained for 100 rounds, with the learning rate set using the cosine annealing algorithm. Set γ=30 as the default value. In order to establish a robust model with good generalization ability, label smoothing is used for Softmax, and in order to ensure the stability of training, the gradient stops the Softmax loss function only when the return value of Softmax is less than 3.

[0077] 3. Model application

[0078] In this stage, the present application does not perform data augmentation operations on the input data, but only samples the input data to an image size of 256*256. At the same time, the present application fixes the model parameters and no longer updates the network parameters through the stochastic gradient descent algorithm, but only uses the network as an image feature extractor. In the actual inference process, the present application uses the output features F of the ResNet-50 feature extraction network which has been deployed with the gradient stopped Softmax loss function and applied the BN-ReLU removal technology. For query examples, the features F obtained after model feature inference q After extracting the features of all images in the image library, the present application stores them as a feature sequence {F1,…,F m} and then calculate F q and the Euclidean distance of all image features in the image library:

[0079] d = ||F q -F i ||2, i = 1, 2, 3,..., m

[0080] Further, the present application obtains the distance sequence D = [d1, d2,..., d m ], and then the present application reorders the distance D, and the present application takes the L closest images to the query example, and if there is an image with the same ID as the query example in these images, it is considered that the image retrieval is successful.

[0081] From the perspective of recall rate, the present application not only solves the problem that the model is difficult to converge, but also achieves 75.9% on CUB-200-2011, 94.7% on CARS196, and 83.1% on SOP, which is at least more than 1.7%, 2.9% and 1.7% compared with the traditional method.

[0082] The above is only a preferred embodiment of the present application and is not intended to limit the present application, and any modification, equivalent replacement and improvement made within the spirit and principle of the present application are within the protection scope of the present application.

Claims

1. An image retrieval and recognition method based on autonomous evolutionary loss, characterized in that Comprise the following steps: (1) using ResNet50 model as the backbone network, and pre-training on the ImageNet large classification dataset; (2) replace the global average pooling with generalized mean pooling, add a batch normalization layer without scaling item and bias item to the backbone network, remove the last BN-ReLU module in the backbone network, and use L2 normalized Euclidean distance to calculate the recall rate during testing; (3) adjust all input images to 256*256 resolution and crop to 224*224 resolution, do not perform data augmentation on the input data, and only sample the input data to 256*256 image size; (4) train the model for 100 rounds, set the parameter learning rate using the cosine annealing algorithm, and set γ=30 as the default value; (5) use label smoothing for the Softmax loss function, and when the return value of the Softmax loss function is less than 3, the gradient stops the Softmax loss function and starts to join the training; The gradient stop Softmax loss function is as follows formula (2): where N is the number of samples in each input batch, c is the number of classes in the training set, f i is the feature of the i-th sample, y i is the label of the i-th sample, W j is the j-th column of the last fully connected layer, corresponding to the j-th class, denotes L2 normalization, while denotes that no gradient update is allowed through W j , and γ is a predefined scalar; Softplus(x) = log(1 + e x ), (6) fix the model parameters and no longer update the network parameters through the stochastic gradient descent algorithm, only use the network as an image feature extractor; (7) output the features F of the ResNet-50 feature extraction network deployed with the gradient stop Softmax loss function and the removed BN-ReLU module; (8) The features F obtained by model feature inference for the query example q The features of all images in the image library are extracted and stored as a feature sequence {F1,..., F m} (9) Compute query example feature F q and the Euclidean distance of all image features in the image library: d = ||F q - F i ||2,i = 1,2,3,…,m; (10) obtaining a distance sequence D = [d1, d2,..., d m ] (11) reorder the distance D, take the L closest images to the query example, and if there is an image with the same ID as the query example in the image, consider this image retrieval successful. 2.The autonomous evolution loss based image retrieval and recognition method of claim 1, wherein: The combination of the Softmax loss function and the gradient stop Softmax function in step (5) is as follows formula: L = L softmax + L SGSL (3) The gradient stop Softmax loss function and the original Softmax loss function share parameters. 3.The autonomous evolution loss based image retrieval and recognition method of claim 1, wherein: The adopted image library is as follows: CUB-200-2011 has 200 classes, 11788 pictures, and the first 100 classes have 5864 images for training, and the remaining classes have 5924 images for testing; CAR-196 has 198 classes and 16,185 images, the first 98 classes are used for training 8054 images, and the other 98 classes are used for testing 8131 images; Stanford Online Products has 22634 classes and 120053 pictures, the first 11318 classes have 59551 images for training, and the other 11316 classes have 60502 images for testing; In-shop clothing has 50 fine-grained classes and 1000 attributes, containing more than 800,000 images.

Images