Unsupervised image embedding learning method based on nearest neighbor and difficult sample mining

Abstract

The present invention is an unsupervised image embedding learning method based on nearest neighbor and difficult sample mining, comprising step 1: building a deep learning framework operating environment; step 2: building a network model for performing unsupervised embedding learning tasks; step 3: using a data set to train the network in an end-to-end manner; step 4: inputting the image trained in step 3 into the network model of the unsupervised embedding learning task, and retrieving the image with the closest similarity according to the cosine similarity between the image features, and if the retrieved images belong to the same category, the retrieval is successful. The method uses a deep neural network to extract features from the image, and then mines the nearest neighbors of the sample according to the cosine similarity between the features, thereby dividing the image into several neighborhoods, and the sample images in the neighborhood share pseudo-category information, so that the model can learn the high discriminative features of the image.

Description

Technical Field

[0001] The invention belongs to the field of unsupervised image embedding learning in pattern recognition and computer vision, and specifically relates to an unsupervised image embedding learning method based on nearest neighbor and difficult sample mining. Background Art

[0002] Due to its powerful ability to learn visual features, deep neural networks have been used as the basic structure for many computer vision tasks, such as object detection, face recognition, image retrieval, pedestrian re-identification, etc. The performance of deep neural networks depends largely on their own capabilities and the amount of training data. In order to increase the capacity of network models, researchers have developed different types of network architectures. At the same time, large-scale datasets such as ImageNet have been collected and used to train very deep neural networks. With complex architectures and large-scale datasets, the performance of deep convolutional neural networks continues to break through the state-of-the-art level of computer vision tasks.

[0003] Deep embedding learning aims to use deep neural networks to learn a discriminative low-dimensional embedding feature from an image. This embedding feature has two properties: 1) positive concentration: the embedding features of samples belonging to the same category should be close to each other; 2) negative separation: the embedding features of samples belonging to different categories should be as far away from each other as possible. Deep embedding learning has many extensive and important applications, such as image retrieval, face recognition, target tracking, pedestrian re-identification, etc. However, collecting and annotating datasets for different tasks is a time-consuming and expensive project. As one of the most widely used datasets for computer vision, ImageNet contains about 1.3 million labeled images covering 1,000 categories, and each image is manually labeled with a class label. When collecting and annotating such a large-scale dataset, it takes a lot of time for many workers. In particular, the annotation of large-scale fine-grained datasets often requires experts in the field to complete. Therefore, learning features from images directly and automatically in an unsupervised manner is a very important and challenging task, and has become a research hotspot in the field of machine learning and computer vision.

[0004] Unsupervised embedding learning requires that the similarity between the learned embedded features is consistent with the visual similarity or category relationship of the input samples. In contrast, the widely studied unsupervised feature learning aims to learn good "intermediate" feature representations using different supervisory signals, by learning a linear classifier or object detector to generalize the learned features to downstream tasks, where the target task still requires labeled samples. However, the learned features may not maintain visual similarity, so their performance is significantly reduced for tasks based on visual similarity, such as nearest neighbor search. Summary of the invention

[0005] In order to solve the above problems, the present invention proposes an unsupervised image embedding learning method with memory nearest neighbors and difficult sample mining. The method uses a deep neural network to extract features from the image, and then mines the nearest neighbors of the samples according to the cosine similarity between the features, thereby dividing the image into several neighborhoods. The sample images in the neighborhood share pseudo-category information, so that the model can learn the highly discriminative features of the image.

[0006] In order to achieve the above object, the present invention is achieved through the following technical solutions:

[0007] The present invention is an unsupervised image embedding learning method based on nearest neighbor and difficult sample mining, comprising the following steps:

[0008] Step 1: Build a deep learning framework operating environment based on Pytorch, including third-party operating libraries such as Numpy and Scipy;

[0009] Step 2: Build a network model that performs unsupervised embedding learning tasks in the deep learning framework operating environment built in step 1. The network model mainly includes five modules: feature extraction module, normalization module, nearest neighbor mining module, difficult sample mining module, and embedding learning module.

[0010] The feature extraction module uses the GoogLeNet network pre-trained on ImageNet as a feature extractor, and adds a 128-dimensional fully connected layer after the fifth pooling layer of GoogLeNet as a feature embedding layer. In the process of extracting features, the training images in the data set given by the unsupervised embedding learning task are randomly cropped to 227×227 for data enhancement.

[0011] The normalization module is connected to the end of the feature extractor to extract image features in the data set given by the unsupervised embedding learning task. The normalization module normalizes each element of the deep feature vector of the image features extracted by the feature extraction module by the modulus of the vector, and normalizes the deep features into a vector with a modulus of 1, so as to facilitate the optimization of network parameters.

[0012] These normalized deep feature vectors are input into the nearest neighbor mining module to form several neighborhoods. The samples in the neighborhood share category information and a category label is assigned to each sample.

[0013] According to the mined sample features, in the embedding learning module, for each sample feature, it is brought closer to the features of similar samples and away from the features of other types of samples. In other words, the embedding learning module combines the difficult positive and negative samples of the difficult sample mining module into one for optimization, updates the network parameters, and enables the model to learn highly discriminative image features;

[0014] Step 3: Use the dataset to train the network in step 2 in an end-to-end manner: randomly crop all images to 227×227 for data augmentation, use the SGD optimizer with a momentum of 0.9 and set the weight decay to 5*10^-4 until the network parameters reach the optimal state and stop training.

[0015] Step 4: After all weight parameters are trained to the optimal level, a completely new image is input to test the model. That is, in the test phase, a test image that has never appeared before is center cropped and used as input. The image with the smallest similarity to the test image is retrieved based on the cosine similarity of the features. If the retrieved image belongs to the same category as the input test image, the retrieval is successful, otherwise it fails.

[0016] A further improvement of the present invention is that the data sets used for training include CUB200 and Cars196.

[0017] The CUB200 dataset is a fine-grained image dataset containing photos of 200 bird species released by the California Institute of Technology in 2011. The dataset contains 11,788 bird images, and each image provides image class labeling information. Among them, 5,864 images of the first 100 categories are used for training, and 5,924 images of the remaining 100 categories are used for testing. Therefore, the training image categories are completely different from the test image categories.

[0018] Cars196 is a fine-grained image dataset released by Stanford University in 2013. It contains 196 types of cars and a total of 16,185 images. Each image provides image class labeling information. 8,054 images from the first 98 categories are used for training, and 8,131 images from the remaining 98 categories are used for testing. Similarly, the training image categories are completely different from the test image categories.

[0019] The beneficial effects of the present invention are as follows: the method utilizes a deep neural network to extract features from an image, and then mines the nearest neighborhood of the sample based on the cosine similarity between the features, thereby dividing the image into several neighborhoods. The sample images within the neighborhood share pseudo-category information. In order to enable the model to converge quickly, a difficult sample mining strategy is used to select more valuable and meaningful samples. In order to achieve the purpose of positive concentration and negative separation, for each sample, it is made to be close to the features of samples of the same type in the feature space and away from the features of samples of other types. The network model is adjusted by optimizing the objective function so that the model can learn the highly discriminative features of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flow chart of the unsupervised image embedding learning method of the present invention. DETAILED DESCRIPTION

[0021] The following will disclose the embodiments of the present invention with drawings. For the purpose of clear description, many practical details will be described together in the following description. However, it should be understood that these practical details should not be used to limit the present invention. That is to say, in some embodiments of the present invention, these practical details are not necessary.

[0022] like Figure 1 As shown, when performing an unsupervised image embedding learning method based on nearest neighbor mining and difficult sample mining, first build a deep learning framework operating environment based on Pytorch, including third-party operating libraries such as Numpy and Scipy, and build a network model for performing unsupervised embedding learning tasks under the built deep learning framework operating environment. The network model mainly includes five modules: feature extraction module, normalization module, nearest neighbor mining module, difficult sample mining module, and embedding learning module.

[0023] The feature extraction module uses the GoogLeNet network pre-trained on ImageNet as a feature extractor, and adds a 128-dimensional fully connected layer after the fifth pooling layer of GoogLeNet as a feature embedding layer. In the process of extracting features, the training images in the data set given by the unsupervised embedding learning task are randomly cropped to 227×227 for data enhancement.

[0024] The normalization module normalizes the feature vector extracted by the network so that the modulus of the vector is 1. Specifically, for a set of unlabeled images X = {x1, x2, ..., x n}, neural network f θ (.) Output the embedded features f of each image θ (x i )∈R l , where θ is the network parameter and l is the feature dimension. Therefore, the normalized features are recorded as:

[0025]

[0026] After obtaining the normalized features, they are input into the nearest neighbor mining module. Specifically, the first step is to embed the features in the embedding space R l Select one feature f as the target feature. In the second step, find the nearest neighbor sample feature set N of the target sample feature among the remaining sample features. k , which is expressed as:

[0027] N k (f) = {f i |s(f i ,f)is ranked top-k in F}(2)

[0028] Where F is the feature set of all training samples extracted by the model, s represents the cosine similarity function of the feature vector, and the nearest neighbor represents the sample with the same semantic information as the target sample. However, in the absence of labels, the nearest neighbors of the target sample will largely contain noise samples, so the nearest neighbor N is directly used. k During supervised training, noise will make the model unstable. Therefore, using graph theory, the present invention starts from a random feature point f r Initially, a tree structure is constructed based on the edges. Then, a depth-first search strategy is used to traverse the features along the edges. The depth-first search strategy is used to give priority to accessing features with large similarities. In order to avoid repeated traversal of features, the present invention limits each feature to be accessed only once. In order to ensure the accessibility of any two features, the present invention implements a loop constraint, that is, in the tree structure, a loop must return to the starting feature. During the search process, the goal is to find mutually non-intersecting loop structures. Therefore, the present invention deletes all features in the found loops from the feature set to simplify and speed up the search. This process is repeated until there are no loops. It is worth noting that some isolated features may be left at the end. These loop structures can be regarded as neighborhoods, and the samples in the neighborhood share category information and are assigned pseudo labels.

[0029] In order to accelerate the convergence of the model, the present invention adds a difficult sample mining module to further improve the performance. In the difficult sample mining module, according to the pseudo-label, the cosine similarity of the sample features is used to mine difficult samples. i , its difficult negative samples are obtained in the following way, namely:

[0030] s ij ≥min{s ik |y k =y i}-∈,y j ≠y i

[0031] Among them, s ij is the cosine similarity between two samples, ∈ is a given margin. Put the mined difficult negative samples into N i Similarly, the following formula is used to mine difficult positive samples, namely:

[0032] s it ≤max{s ik |y k ≠y i}-∈,y t =y i

[0033] Put the mined difficult positive samples into P i middle.

[0034] In the embedding learning module, in order to achieve the purpose of positive concentration and negative separation of sample features in space, the present invention introduces the following loss function, which combines the mined difficult positive and negative samples into a whole, measures the cosine similarity between the target sample and the difficult positive and negative samples, and uses hyperparameters to balance the weights of positive and negative samples, namely:

[0035]

[0036] Where m is the training batch, parameters p and q are used to balance the weights of positive and negative samples, and μ is a fixed value. The network parameters are updated by stochastic gradient descent, that is:

[0037]

[0038] Where α is the learning rate or step size.

[0039] Step 3: Use the dataset to train the network in step 2 in an end-to-end manner: randomly crop all images to 227×227 for data augmentation, use the SGD optimizer with a momentum of 0.9 and set the weight decay to 5*10^-4 until the network parameters reach the optimal state and stop training.

[0040] Step 4: After the model training is completed, a new image should be input to test the model. That is, in the test phase, a test image that has never appeared before is center cropped and used as input. The image with the smallest similarity to the test image is retrieved based on the cosine similarity of the features. If the retrieved image belongs to the same category as the input test image, the retrieval is successful, otherwise it fails.

[0041] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. An unsupervised image embedding learning method based on nearest neighbor and difficult sample mining, characterized by: The unsupervised image embedding learning method comprises the following steps: Step 1: Build a deep learning framework operating environment based on Pytorch; Step 2: Build a network model that performs unsupervised embedding learning tasks in the deep learning framework operating environment built in step 1. The network model mainly includes five modules: feature extraction module, normalization module, nearest neighbor mining module, difficult sample mining module, and embedding learning module; The feature extraction module extracts image features in a data set given by an unsupervised embedding learning task, the normalization module normalizes each element of a deep feature vector of image features extracted by the feature extraction module by dividing it by the modulus of the vector, the nearest neighbor mining module performs neighborhood mining of sample features on the normalized features, and uses the difficult sample mining module to select samples, the embedding learning module combines the difficult positive and negative samples of the difficult sample mining module into one for optimization, and updates the network parameters, so that the model learns highly discriminative image features; Step 3: Use the dataset to train the network in step 2 in an end-to-end manner; Step 4: Input the trained image in step 3 into the network model of the unsupervised embedding learning task in step 2, and retrieve the image with the closest similarity based on the cosine similarity between the image features. If the retrieved images belong to the same category, the retrieval is successful. The nearest neighbor mining module in step 2 forms a neighborhood graph by constructing the nearest neighbors, and uses a depth-first search strategy in the neighborhood graph to mine a strongly connected graph to form a neighborhood, and the samples in the neighborhood share category information; The difficult sample mining module is used in step 2 to mine difficult samples specifically as follows: for the target sample, samples that are dissimilar to it are mined from samples of the same class as difficult positive samples, and samples that are similar to it are mined from samples of different classes as difficult negative samples.

2. The unsupervised image embedding learning method based on nearest neighbor and difficult sample mining according to claim 1, characterized in that: In the embedding learning module in step 2, a loss function is introduced to combine the difficult positive and negative samples mined by the nearest neighbor mining module into a whole, and the cosine similarity between the target sample and the difficult positive and negative samples is measured at the same time. The weights of positive and negative samples are balanced by hyperparameters, and the cosine similarity between them and the feature vector of difficult positive samples is increased, and the cosine similarity between them and the features of difficult negative samples is reduced. By optimizing this goal, the network parameters are updated by the stochastic gradient descent method, so that the model learns highly discriminative image features.

3. The unsupervised image embedding learning method based on nearest neighbor and difficult sample mining according to claim 1, characterized in that: The feature extraction module in step 2 uses the GoogLeNet network pre-trained on ImageNet as a feature extractor, and adds a 128-dimensional fully connected layer after the fifth pooling layer of GoogLeNet as a feature embedding layer. In the process of extracting features, the training images in the data set given by the unsupervised embedding learning task are randomly cropped to 227×227 for data enhancement.

4. The unsupervised image embedding learning method based on nearest neighbor and difficult sample mining according to claim 1, characterized in that: The training of step 3 is as follows: all images are randomly cropped to 227×227 for data augmentation, an SGD optimizer with a momentum of 0.9 is used and the weight decay is set to , until the network parameters reach the optimal state and stop training.

Images