An unsupervised person re-identification method based on camera distribution difference alignment constraint

Abstract

The present invention discloses an unsupervised person re-identification method based on camera distribution difference alignment constraint. By constructing constraints based on the distribution consistency of the source domain and the target domain and alignment constraints based on cross-camera view differences, features with transferability, domain invariance and invariance to cross-camera views are learned, thereby realizing a model that can adapt to unsupervised person re-identification across camera views. The mean and standard deviation of the similarity score between each camera and the whole are calculated to constrain the impact of camera view differences and reduce the differences in clustering results of the same pedestrian under different cameras; through two loss functions, a more robust and stable unsupervised person re-identification network is trained. By utilizing the idea of ​​the invariance of pseudo-labels across camera views, the problem of differences across camera views in unsupervised person re-identification is effectively solved when the sample distribution difference between the source domain and the target domain is large and there is cross-view in the target domain.

Description

Technical Field

[0001] The present invention relates to the technical field of unsupervised pedestrian re-identification, and in particular to an unsupervised pedestrian re-identification method based on camera distribution difference alignment constraints. Background Art

[0002] Person re-identification (re-ID) aims to locate a target person in a surveillance video given a probe image. With the rapid development of deep learning models, most person re-ID methods focus on supervised learning and have achieved high accuracy on public datasets. However, they require a large amount of paired labeled data across each pair of camera views, which limits the scalability for large-scale applications where annotating all images in the dataset is both expensive and infeasible. To address the scalability issue, unsupervised person re-ID has received widespread attention. Unsupervised person re-ID is learned by transferring the source domain to the target domain. However, due to the data bias between the source and target datasets, the performance of the target domain may drop significantly. The most popular solution to this problem is unsupervised domain adaptation (UDA).

[0003] Currently, UDA has been widely used in many aspects such as image classification, target detection, and face recognition. There are three main types of UDA methods. The first type is a clustering-based method that maintains strong performance. By assigning labels to unlabeled training samples and using the generated pseudo-labels of the target domain to optimize the network. Another series uses adversarial generative models (GAN networks) as style converters to convert pedestrian images (with identity annotations) in the source domain to the target domain to learn domain-invariant features. The third type of method attempts to optimize a neural network with soft labels for target domain samples by calculating the similarity with a reference image or feature. Many UDA methods do not take into account the differences across camera views, resulting in different pseudo-labels for the same pedestrian under different cameras, which affects the final performance to a certain extent. Because the clustered unsupervised pedestrian re-identification method obtains a high ranking, the present invention is also based on an unsupervised clustering method, which reduces the impact of camera views by constraining the differences across camera views and improves the final performance of the model. Summary of the invention

[0004] The purpose of the present invention is to provide an unsupervised pedestrian re-identification method based on camera distribution difference alignment constraints. The method utilizes the ideas of domain adaptation and cross-camera view difference alignment. By constructing constraints based on distribution consistency of source domain and target domain and constraints based on cross-camera view invariance, high-level features with transferability, domain invariance and cross-view label invariance are learned. In this way, when the sample distribution difference between the source domain and the target domain is large and the pseudo-labels of the target domain samples are inconsistent across views, an unsupervised pedestrian re-identification model suitable for the target domain is obtained.

[0005] To achieve the above objectives, the present invention adopts the following technical solutions: first, the source data is input into the Convolutional Neural Network (CNN) to obtain a pre-trained network; secondly, the source data set and the target data set are input into the pre-trained network to extract features, and the pseudo-labels are obtained by the DBSCAN clustering method and k-reciprocal coding as the distance metric; next, the network is optimized by two loss functions so that more reliable pseudo-labels can be obtained in each iteration. The first function is the triplet loss function. Since the predicted pseudo-labels are not real labels, identity loss cannot be used. Therefore, the triplet loss function is used to optimize the network training process. The second function is the cross-camera view difference loss function. By adding an additional fully connected layer (FC) to the network, the P output of the fully connected layer is s Identity quantity category, P s is the number of pedestrian identities in the source dataset, so that the similarity score between the target dataset and the source dataset can be obtained. The difference in camera views can be reduced by calculating the similarity score gap between pedestrians of the same identity under different cameras.

[0006] An unsupervised person re-identification method based on camera distribution difference alignment constraint includes the following steps:

[0007] Step 1: manually mark a small portion of the pedestrian data images collected from the camera as the source data set, and use most of the unmarked images as the target data set. Input the marked source data set images to pre-train the convolutional neural network.

[0008] Step 2: Since the target domain dataset has no labeled labels, a pseudo-label training network is obtained by clustering. The previously pre-trained convolutional neural network is used for feature extraction. The pre-trained convolutional neural network can well extract the fine-grained features of pedestrians and focus on some key areas of pedestrians. The source dataset and the target dataset are input respectively, and features are extracted from them. As the distance measurement unit of clustering, the density-based clustering algorithm DBSCAN is used for clustering. Each category corresponds to a pedestrian identity, so that the image of the target dataset obtains a pseudo label.

[0009] Step 3, build the main task module. When the target dataset and the source dataset are fed into the training process, the last fully connected layer of the original convolutional network is deleted, which is used as the basis for the previous clustering. An additional fully connected layer is added, set to 2048-dim, and this fully connected layer uses the pseudo-label obtained in the clustering stage as the label. In particular, this FC layer is also updated during the training phase. This FC layer is used to extract information from key areas and integrate the relationship between features. Pseudo-labels obtained through clustering include correct results and incorrect results. In order to reduce incorrect pseudo-labels during the training process, triple loss is used as a measurement standard. Perform iterative training and reduce the loss through gradient backpropagation to optimize the network.

[0010] Step 4: Construct an auxiliary task module. Add an FC layer based on the original convolutional neural network. This FC layer is set to the number of identities marked in the source dataset. Calculate the similarity scores of the target dataset images and the marked pedestrians in the source dataset. Calculate the distance between each camera similarity score and the overall similarity score as the cross-camera view difference loss to make the scores of people of the same type as close as possible to reduce the impact of camera view differences.

[0011] Step 5: By superimposing the triplet loss of the main task module and the cross-camera view difference loss of the auxiliary task module, the convolutional neural network is continuously optimized iteratively to reduce the two error situations in the clustering process: different pedestrians obtain the same pseudo-label and the same person has inconsistent pseudo-labels under different cameras, thereby improving its model performance.

[0012] Compared with the prior art, the present invention has the following obvious advantages:

[0013] When there is a large difference in sample distribution between the source domain and the target domain, and there are cross-view differences in the target domain images, the present invention constructs constraints based on the distribution consistency of the source domain and the target domain in the main task module, and constructs alignment constraints based on cross-camera view differences in the auxiliary task module, thereby obtaining a model for cross-view unsupervised pedestrian re-identification with good performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1It is a model structure diagram of the present invention;

[0015] Figure 2 is a flow chart of the method of the present invention;

[0016] Figure 3 Example diagram of clustering process;

[0017] Figure 4 This is an example diagram of the full connection process. DETAILED DESCRIPTION

[0018] The present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0019] The model structure of the method of the present invention is as follows Figure 1 As shown, the flowchart of the method is as follows Figure 2 As shown, the specific steps include:

[0020] Step 1: Get the pre-trained model through the source dataset.

[0021] The pre-trained model cannot be used directly to test the target dataset, because if you use this model directly for testing, you will find that the results are not as expected, because there are certain dataset differences between the target dataset and the source dataset. But it is still much better than training the classification model directly on the Imagenet dataset, because the performance of the latter is almost zero. Through the model trained on the source dataset, some useful representations of pedestrian re-identification tasks can be learned, such as better extraction of some image features and attention to some key areas, so that the pseudo-labels initially assigned have a certain degree of accuracy.

[0022] Step 2: Use the pre-trained convolutional neural network to extract features from the source and target datasets as the distance measurement unit for clustering, and use DBSCAN as the clustering method to obtain pseudo labels. In order to better mine the similarity of the target domain dataset, a robust and reliable convolutional neural network is used to extract features, and then a pseudo label is obtained through clustering to guide classification learning. Common clustering methods include K-means clustering, density-based clustering (DBSCAN), maximum expectation (EM) clustering based on Gaussian mixture model, and agglomerative hierarchical clustering. K-means is a commonly used clustering method, but the clustering method requires the clustering clusters to be specified in advance. In re-ID, the number of pedestrians is often unknown, so density clustering DBSCAN is used to obtain pseudo labels.

[0023] In DBSCAN, the distance metric is used as the basis, and the detailed clustering process is as follows: Figure 3In practice, k-reciprocal encoding is used as the distance metric, which is a variation of the Jaccard distance between sets of nearest neighbors. Specifically, using encoder x, all samples from the target dataset T are encoded, and the distance matrix M∈R is calculated using these features. mt×mt :

[0024]

[0025] Among them, M ij The similarity between samples i and j is calculated by the features after the pooling layer. * (i, k) is the refined k-reciprocal set of sample i, obtained by adding some specific reliable constraints.

[0026] According to the similarity matrix M, we can get X i With X j The distance expression

[0027]

[0028] And for each X in the target dataset T i , search for the nearest neighbor in the source data set S, for X i The confidence function of the distance metric is expressed as

[0029]

[0030] Where N S (X i ) represents the nearest point in the source data set S, d W The smaller the confidence, the higher the confidence. w and d j Transform to the same scale and convert d w Normalized and divided by max(d w (x i )). Combining the above formula, we can get the final clustering distance as

[0031] d(X i ,X j )=(1-λ)d j (X i ,X j )+λ(d W (X i )+d W (X j ))

[0032] λ is a hyperparameter ranging from 0 to 1

[0033] Step 3: Build the main task module. In the actual training process, delete the last fully connected layer of the convolutional neural network to extract feature clustering. Add an additional fully connected layer and set it to 2048-dim, such as Figure 1 The main task module of the training process. The pseudo labels obtained in the clustering stage are used as labels. In particular, this FC layer is also updated in the training stage. This FC layer is used to extract information from key areas and integrate the relationship between features. In addition, the pseudo labels of clustering are used as input labels of the target dataset images for classification learning. For the results of clustering errors, triplet loss is used to measure and optimize.

[0034] In the clustering stage, we obtain pseudo labels of the target domain, some of which are incorrect results. Therefore, we use the triplet loss function to measure this inaccuracy and optimize the network. This triplet loss calculates the triplet loss function for each batch by outputting the multidimensional vector:

[0035]

[0036] Among them, N b is the batch size for training, p and n are the most dissimilar positive and most similar negative samples to image a, and x p , x n , x a are the features of the corresponding positive, negative and query images respectively.

[0037] Step 4: Build the auxiliary task module. In addition, add an FC layer to calculate the similarity score between the target dataset and the source dataset. By calculating the distance between each camera similarity score and the overall similarity score as the loss, the impact of view differences can be reduced to achieve cross-camera view difference alignment. The detailed implementation process is as follows: Figure 4 .

[0038] In unsupervised person re-identification, the identity information of the image is often not available, but in practice the camera information that captured the image can be obtained, such as the camera label information, the camera number under which the image was taken, and the camera location information. Most re-ID recognition is based on cross-view pairs consisting of two people's images captured by different camera views. Therefore, the pseudo-label of the same person should be consistent between different cameras. From the perspective of clustering feature distribution, the features of unlabeled images in the target domain should depend on the appearance features of the person, reducing the influence of the camera. However, the actual clustering results will be affected by the lighting of different cameras and some other factors. An additional FC layer is added to the original network structure, and the output of the FC layer is P s Identity quantity category, P sis the number of identities in the source dataset. In this way, we can get the similarity score between the target dataset image and the source dataset.

[0039] Based on the above analysis, a loss with multi-label difference constraints across views is proposed:

[0040]

[0041] Among them, P(y) is the similarity score of the images in the target dataset, P v (y) is the similarity score of the image in the vth camera. d(.,.) is the distance between the two scores. Some distribution distance metrics can be used to calculate this loss. For example, KL divergence, JS divergence and Wasserstein distance. Since P(y) is a two-dimensional variable distributed between 0 and 1 and the distribution is skewed normal, the Wasserstein distance is used, so the simplified Wasserstein distance can be used. The above formula can be updated to:

[0042]

[0043] Among them, μ v is the mean similarity score of the images under the Vth camera, and μ is the mean similarity score of the images in the entire dataset. v is the standard deviation of the Vth camera, and σ is the standard deviation of the image similarity scores of the entire dataset.

[0044] Since the network is in a dynamic learning process, the pseudo-label is not static, so there is an update strategy to store a matrix of similarity scores in memory. N p is the number of people in the source dataset, N u is the number of unlabeled target dataset images. During the continuous training process, this similarity score matrix is ​​also updated in the following way:

[0045] h i ←θh i +(1-θ)y i

[0046] Among them, θ∈[0,1] is a hyperparameter, y i is the similarity score of the current training process.

[0047] Step 5: By superimposing the triplet loss of the main task and the cross-camera view difference loss of the auxiliary task, we can continuously iterate and optimize the network training, reduce the number of pseudo-label errors in the clustering process, and obtain a more robust model.

[0048] Based on the above analysis, the loss function of the main network structure can be defined as Loss total =L c +L tri By superimposing two loss functions and iteratively training the network, a robust and stable unsupervised person re-identification model is obtained, which can also overcome the differences caused by cross-camera views.

[0049] So far, the specific implementation process of the present invention has been described.

Claims

1. An unsupervised person re-identification method based on camera distribution difference alignment constraint, characterized by: The method comprises the following steps: Step 1: From the pedestrian data images collected by the camera, manually mark a small part as the source data set, and most of the unmarked images as the target data set; input the marked source data set images to pre-train the convolutional neural network; Step 2: Since the target domain dataset has no labeled labels, a pseudo-label training network is obtained by clustering. The previously pre-trained convolutional neural network is used for feature extraction. The pre-trained convolutional neural network extracts fine-grained features of pedestrians and focuses on the preset pedestrian area. The source dataset and the target dataset are input respectively, and features are extracted. As the distance measurement unit for clustering, the density-based clustering algorithm DBSCAN is used for clustering. Each category corresponds to a pedestrian identity, so that the image of the target dataset obtains a pseudo label. Step 3, build the main task module; in the process of sending the target data set and the source data set for training, delete the last fully connected layer of the original convolutional neural network, which is used as the basis for the previous clustering; add an additional fully connected layer, set to 2048-dim, and use the pseudo-label obtained in the clustering stage as the label; this FC layer is also updated in the training stage; this FC layer is used to extract information from the preset area and integrate the relationship between features; use the triple loss as the measurement standard; perform iterative training, and optimize the network by reducing the loss through gradient backpropagation; the triple loss function is as follows: Among them, N b is the batch size for training, p and n are the most dissimilar positive sample and the most similar negative sample to image a, x p , x n , x a are the features of the corresponding positive, negative, and query images respectively; Step 4: Build an auxiliary task module. Add another FC layer on the basis of the original convolutional neural network. This FC layer is set to the number of identities marked in the source dataset. Calculate the similarity scores of the target dataset images and the marked pedestrians in the source dataset. Calculate the distance between each camera similarity score and the overall similarity score as the cross-camera view difference loss, so that the scores of people of the same type are close to each other, thereby reducing the impact of camera view differences. Specifically: The auxiliary task module optimizes the alignment of cross-camera view differences by adding an additional FC layer, the output of which is P s Identity quantity category, P s is the number of identities in the source dataset; obtain the similarity score between the target dataset and the source dataset; propose the loss of multi-label difference constraint across views: Loss c =∑ v d(P v (y),P(y)) 2 Among them, P(y) is the similarity score of the images in the target dataset, P v (y) is the similarity score of the image in the vth camera; d(.,.) is the distance between the two scores; since P(y) is a two-dimensional variable distributed between 0 and 1 and has a skewed normal distribution, the Wasserstein distance is used to simplify the Wasserstein distance and update the above formula to: Among them, μ v is the mean similarity score of the images under the vth camera, μ is the mean similarity score of the images in the entire dataset; σ v is the standard deviation under the vth camera, and σ is the standard deviation of the image similarity scores of the entire dataset; Step 5: By superimposing the triplet loss of the main task module and the cross-camera view difference loss of the auxiliary task module, the convolutional neural network is continuously optimized iteratively to reduce the two errors in the clustering process: different pedestrians obtain the same pseudo-label and the same person has inconsistent pseudo-labels under different cameras, thereby improving the model performance; In DBSCAN, based on distance metric, k-reciprocal encoding is used as the distance metric, which is a variation of the Jaccard distance between the nearest neighbor sets; using encoder X, all samples from the target dataset T are encoded and the similarity matrix M∈R is calculated. mt×mt : Among them, M ij The similarity between samples i and j is calculated by the features after the pooling layer, R * (i, k) is the refined k-reciprocal set of sample i, obtained by adding preset reliable constraints; According to the obtained similarity matrix M, we can get X i With X j The distance expression is: And for each X in the target dataset T i , search for the nearest neighbor in the source data set S, X i The confidence function of the distance metric is expressed as: Where N S (X i ) represents the nearest point in the source data set S, d W The smaller the confidence, the higher the confidence. w and d j Transform to the same scale and convert d w Normalize and divide by max(d w (X i )); Combining the above formula, the final clustering distance is: d(X i ,X j )=(1-λ)d h (X i ,X j )+λ(d w (X i )+d w (X j )) λ is a hyperparameter ranging from 0 to 1.

2. The unsupervised person re-identification method based on camera distribution difference alignment constraint according to claim 1, characterized in that: The constructed unsupervised person re-identification network has three parts: a pre-trained convolutional neural network based on the source dataset, a main task module that obtains pseudo labels through clustering for classification learning, and an auxiliary task module that uses camera view differences to guide the clustering process; The main task module clusters the features extracted from the source and target datasets through pre-trained convolutional neural networks to obtain pseudo labels for classification learning. The specific mathematical description is as follows: Unsupervised person re-identification, there is a source dataset {X S ,Y S }, including N S pedestrian images and each image There are corresponding labels in P S is the number of identities in the source dataset, and there is also a target dataset {X t }, including N t pedestrian images; but the target dataset {X t The identity of each image in} is unknown; in the training set, the input space consists of camera cross-view image pairs, and the output space consists of labels that mark the same or different identities; (X,X)∈R Ns×Ns , the output space is Y = {l|l∈{0,1}}, where 1 represents the same identity and 0 represents different identities.

Images