Person re-identification method and device based on contrastive learning and adversarial training

Abstract

The present invention discloses a person re-identification method and device based on contrastive learning and adversarial training. The method not only designs a basic module, but also utilizes the concepts of contrastive learning and adversarial training to design a viewpoint confusion module and an identity prototype module. The viewpoint confusion module is trained using contrastive learning and, with the help of the gradient reversal layer of the viewpoint discriminator, enables the feature extractor of the basic module to acquire the ability to confuse viewpoint differences. The identity prototype module takes a weighted average based on the confidence level as the identity prototype, thereby clustering the distribution of sample features of the same identity in the feature space. With the help of the viewpoint confusion module and the identity prototype module, the present invention improves person re-identification performance.

Description

Technical Field

[0001] This invention belongs to the behavioral field of machine vision and also to the field of intelligent video surveillance. Specifically, it relates to a person re-identification method and device based on contrastive learning and adversarial training, aiming to alleviate the problem of viewpoint differences degrading the performance of person re-identification algorithms. Background Art

[0002] Person re-identification is a key topic in the field of machine vision and a cutting-edge research direction that has garnered significant attention in recent years. It holds significant theoretical significance and promising applications in areas such as intelligent surveillance, human-computer interaction, and virtual reality. Its core objective is to retrieve pedestrian images from multiple cameras, encompassing multiple disciplines including image processing, computer vision, pattern recognition, and artificial intelligence.

[0003] Viewpoint difference is a challenge in person re-identification. Due to viewpoint difference, the intra-class difference of pedestrian samples increases, while the inter-class difference may decrease, which has a negative impact on recognition performance. Existing person re-identification algorithms for viewpoint difference can be roughly divided into two categories: (1) Data-based methods. This type of method uses generative adversarial networks, image synthesis, and network-based viewpoint generators to expand the dataset so that the dataset contains more images from different viewpoints for feature extractors to learn. (2) Feature-based methods. The main idea of this type of method is to bring the distribution of features of the same identity category closer in the feature space. Summary of the Invention

[0004] The present invention specifically relates to a pedestrian re-identification method and device based on contrastive learning and adversarial training, aiming to alleviate the problem that perspective differences reduce the performance of pedestrian re-identification algorithms.

[0005] The technical contents of the present invention include:

[0006] A pedestrian re-identification method based on contrastive learning and adversarial training is characterized in that the image to be detected is input into the basic module to obtain the pedestrian recognition result, wherein,

[0007] The basic module includes: a feature extractor, an identity classifier and a Softmax layer, wherein the feature extractor is used to obtain image features of the image to be detected, the identity classifier is used to calculate the classification probability correction value of the image features, and the Softmax layer is used to classify the classification probability correction value to obtain the pedestrian recognition result;

[0008] The training process of the basic module includes:

[0009] A basic loss is obtained according to the triplet loss and identity classification loss of the training basic module, wherein the triplet includes: an anchor sample, a positive sample, and a negative sample;

[0010] Using the basic module to obtain sample multi-labels of the sample, and constructing a set of perspective positive / negative sample pairs based on the sample multi-labels;

[0011] The image features of the samples are reduced in dimensionality through a view discriminator equipped with a gradient reversal layer. Based on the set of view positive / negative sample pairs and the low-dimensional image features, a view confusion loss is obtained to guide the parameter update of the feature extractor and the view discriminator.

[0012] Taking the kth element of the sample multi-label as the classification confidence of the sample image belonging to the kth identity category, and calculating the weighted average of the image features belonging to the same identity category according to the classification confidence to obtain the identity prototype of the identity category;

[0013] Constructing an identity prototype triple based on the identity prototype and obtaining the identity prototype triple loss during training, wherein the identity prototype triple includes: an anchor sample feature, a positive sample identity prototype, and a negative sample identity prototype;

[0014] Training is performed based on the base loss, view confusion loss, and identity prototype triplet loss.

[0015] Furthermore, the identity classifier includes: a batch normalization layer and a fully connected layer.

[0016] Furthermore, constructing a set of perspective positive / negative sample pairs based on the multi-label samples includes:

[0017] Calculate the consistency of multiple labels of any two samples where y i with y j represents the multi-label of sample i and sample j, k represents the kth identity category in the dataset, Represents the kth element of the multi-label of sample i;

[0018] Calculate the sampling threshold of the perspective positive sample pair set Sampling threshold of the set of viewpoint negative samples

[0019] According to the consistency A, the sampling threshold and sampling threshold Construct a set of perspective positive / negative sample pairs.

[0020] Furthermore, the view confusion loss Among them, the perspective loss function corresponding to sample i is bsz represents the batch processing size, represents the set of perspective positive sample pairs corresponding to sample i, represents the set of perspective negative sample pairs corresponding to sample i, z i represents the low-dimensional feature corresponding to sample i, and τ represents the contrastive learning temperature parameter.

[0021] Furthermore, the guiding feature extractor and the viewpoint discriminator to perform parameter updating includes:

[0022] For the parameters of the view discriminator Where μ represents the learning rate;

[0023] For the parameters of the feature extractor Wherein, λ represents the hyperparameter of the gradient reversal layer.

[0024] Furthermore, the identity prototype Among them, N represents the number of samples of category k around us, Represents the kth element of the multi-label of sample i, feat i Represents the image features of sample i.

[0025] Furthermore, identity prototype triple loss Among them, A Represents the anchor sample feature, IDP P Represents the positive sample identity prototype, IDP N Represents the negative sample identity prototype, and m represents the identity prototype triplet loss boundary.

[0026] A storage medium stores a computer program thereon, wherein the computer program implements any of the above methods when executed by a processor.

[0027] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform any one of the above methods.

[0028] Compared with the prior art, the present invention has at least the following advantages:

[0029] 1. This paper proposes CLATNet to address the cross-view problem that is prevalent in pedestrian re-identification tasks, and experimentally verifies the effectiveness of the method on multiple common datasets.

[0030] 2. Based on the ideas of contrastive learning and adversarial training, this paper designs a perspective confusion module and an identity prototype module, which can improve the performance of the basic modules without additional data labeling.

[0031] 3. The CLATNet proposed in this paper uses the view confusion module and the identity prototype module in the training phase to enable the feature extractor to extract features that are more robust to view changes, while only retaining the basic module for recognition in the testing phase. The overall network scale is lightweight and maintains a relatively fast testing speed.

[0032] 4. The CLATNet proposed in this paper has strong portability. The basic module can be replaced by any existing network model, and its performance is improved by the view confusion module and identity prototype module. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is the overall network framework diagram of the present invention

[0034] Figure 2 This is a schematic diagram of the perspective confusion module of the present invention

[0035] Figure 3 This is a schematic diagram of the identity prototype module of the present invention

[0036] Figure 4 This is the visualization result of the improved feature distribution of the present invention DETAILED DESCRIPTION

[0037] The present invention will be further described below through specific embodiments and accompanying drawings.

[0038] This paper uses the residual neural network ResNet50 as the basic feature extractor and utilizes contrastive learning and adversarial training to design a view confusion module and an identity prototype module to improve performance by addressing view differences. The view confusion module proposes a view discriminator with a structure similar to that of a multilayer perceptron and is trained using contrastive learning, eliminating the need for additional view labels. The view discriminator also includes a gradient reversal layer, enabling adversarial training between the view discriminator and the feature extractor. This allows the view discriminator to acquire the ability to discriminate view differences during training, while the feature extractor simultaneously acquires the ability to confuse view differences. Furthermore, the present invention proposes an identity prototype module in which sample features of the same identity are weighted averaged based on confidence as the identity prototype. The identity prototype triplet loss is then calculated based on the identity prototypes, clustering the distribution of sample features of the same identity in the feature space. Driven by the view confusion and identity prototype modules, the proposed network, CLATNet (Constrative Learning and Adversarial Training Network), demonstrates significant performance improvements over the basic feature extractor.

[0039] 1. Pedestrian Re-ID Network CLATNet Promoted by Contrastive Learning and Adversarial Training

[0040] The overall network CLATNet framework is as follows Figure 1 As shown in Figure 2, the image is input into a ResNet50-based feature extractor to generate 2048-dimensional features, which are then fed into the base module, view confusion module, and identity prototype module. During the training phase, CLATNet optimizes the weighted sum of the loss functions of these three modules. During the testing phase, only the base module is retained, and the pedestrian identity recognition result is obtained based on the output of the identity classifier in the base module.

[0041] In the base module, a combination of the commonly used triplet loss and identity classification loss is used as the training loss function. Each triplet in the triplet loss consists of an anchor sample, a positive sample, and a negative sample. The positive sample has the same identity label as the anchor sample, while the negative sample has a different identity label from the anchor sample. Each input sample serves as an anchor sample and is paired with other samples in the same training batch to calculate the triplet loss. The view confusion module and identity prototype module utilize contrastive learning to improve the performance of the base module without the need for additional labeling. An introduction to the view confusion module and identity prototype module follows.

[0042] Figure 4 The feature distribution before and after adding the view confusion module and identity prototype module is shown. It can be seen that the view confusion module and identity prototype module proposed by CLATNet in this method promote the feature extractor's ability to increase the inter-class distance and reduce the intra-class distance.

[0043] 2. View Confusion Module

[0044] Some existing methods use camera labels as viewpoint labels to learn viewpoint-related information. However, this assumption does not conform to the actual situation very well, because pedestrians in the same camera may show different viewpoint directions, while pedestrians in different cameras may be captured with similar viewpoints, that is, there is no strict correspondence between camera labels and viewpoint labels. To this end, the present invention uses the idea of contrastive learning to design a viewpoint confusion module. Instead of using camera labels or additional viewpoint labels, it uses known sample identity labels and multi-label consistency given by the classifier to obtain viewpoint positive and negative sample pairs for contrastive learning. Specifically, the present invention proposes the following two assumptions: (1) If two samples have the same identity label, and the multi-label consistency given by the classifier is low, it can be safely assumed that the difference in appearance such as viewpoint difference has caused classification difficulties for the classifier, and they are recorded as viewpoint negative sample pairs with different viewpoints. (2) If two samples have different identity labels, and the multi-label consistency given by the classifier is high, it can be safely assumed that the similarity of viewpoint has caused classification difficulties for the classifier, and they are recorded as viewpoint positive sample pairs with the same viewpoint. The view discriminator reduces the 2048-dimensional features output by the feature extractor to obtain 128-dimensional features. Then, the view contrast loss is calculated on the 128-dimensional features based on the constructed view positive / negative sample pairs. As the view discriminator acquires the ability to recognize view differences, the feature extractor counteracts this by acquiring the ability to confuse view differences through the gradient reversal layer in the view discriminator. The view confusion module diagram is shown below. Figure 2 shown.

[0045] When training the model, first obtain 2048-dimensional features based on the feature extractor, and obtain the multi-label of the classifier through the batch normalization layer, the fully connected layer and the softmax layer (Formula 1). Then calculate the consistency A (Formula 2) based on the multi-label. And obtain the positive and negative sample pairs of the view according to the sample identity label and the consistency threshold (Formula 3-6). The loss function of contrastive learning is shown in Formula 7-8. The parameter update of the feature extractor and the view discriminator guided by this loss function is shown in Formula 9-10. The meaning of the symbols involved in the formula is shown in

[0046] Table 1:

[0047]

[0048]

[0049] Table 1

[0050] y i =Softmax(FC(BN(feat i ))). (1)

[0051]

[0052]

[0053]

[0054]

[0055]

[0056]

[0057]

[0058]

[0059]

[0060] 3. Identity Prototype Module

[0061] Due to differences in perspective, features of the same identity may show significant differences. Since the identity remains the same, it can be assumed that there exists some kind of robust representation that is less affected by perspective differences. This method designs a representation called an identity prototype. For samples belonging to the k-th identity category, the k-th element of the sample's multi-label is used as its classification confidence. Based on the classification confidence, this method calculates the weighted average of features belonging to the same identity and uses the average as the identity prototype of this identity. Based on the identity prototype proposed above, the identity prototype triplet loss is further calculated, and the positive and negative samples in the original triplet loss are replaced with positive and negative identity prototypes, thereby clustering the distribution of sample features of the same identity in the feature space.

[0062] The definition of identity prototype is shown in formula 11, and the identity prototype triple loss based on identity prototype modification is shown in formula 12. The meaning of the symbols involved in the formula is shown in Table 1. A Represents the anchor sample feature, IDP P Represents the positive sample identity prototype, IDP N Represents the negative sample identity prototype.

[0063]

[0064]

[0065] Finally, it should be noted that the above implementation cases are only used to illustrate the technical solution of the present invention and are not limiting. Although the present invention is described in detail using examples, ordinary technicians in this field should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A person re-identification method based on contrastive learning and adversarial training, characterized in that: The image to be detected is input into the basic module to obtain the pedestrian recognition result, where: The basic module includes: a feature extractor, an identity classifier and a Softmax layer, wherein the feature extractor is used to obtain image features of the image to be detected, the identity classifier is used to calculate the classification probability correction value of the image features, and the Softmax layer is used to classify the classification probability correction value to obtain the pedestrian recognition result; The training process of the basic module includes: A basic loss is obtained according to the triplet loss and identity classification loss of the training basic module, wherein the triplet includes: an anchor sample, a positive sample, and a negative sample; Using the basic module to obtain sample multi-labels of the sample, and constructing a set of perspective positive / negative sample pairs based on the sample multi-labels; The image features of the samples are reduced in dimensionality through a view discriminator equipped with a gradient reversal layer. Based on the set of view positive / negative sample pairs and the low-dimensional image features, a view confusion loss is obtained to guide the parameter update of the feature extractor and the view discriminator. Taking the kth element of the sample multi-label as the classification confidence of the sample image belonging to the kth identity category, and calculating the weighted average of the image features belonging to the same identity category according to the classification confidence to obtain the identity prototype of the identity category; Constructing an identity prototype triple based on the identity prototype and obtaining the identity prototype triple loss during training, wherein the identity prototype triple includes: an anchor sample feature, a positive sample identity prototype, and a negative sample identity prototype; Training is performed based on the base loss, view confusion loss, and identity prototype triplet loss.

2. The method according to claim 1, wherein The identity classifier includes: a batch normalization layer and a fully connected layer.

3. The method according to claim 1, wherein The method of constructing a perspective positive / negative sample pair set based on the multi-label sample includes: Calculate the consistency of multiple labels of any two samples where y i with y j represents the multi-label of sample i and sample j, k represents the kth identity category in the dataset, Represents the kth element of the multi-label of sample i; Calculate the sampling threshold of the set of perspective positive sample pairs Sampling threshold of the set of viewpoint negative samples According to the consistency A, the sampling threshold and sampling threshold Construct a set of perspective positive / negative sample pairs.

4. The method according to claim 1, wherein View Confusion Loss Among them, the perspective loss function corresponding to sample i is bsz represents the batch processing size, represents the set of perspective positive sample pairs corresponding to sample i, represents the set of perspective negative sample pairs corresponding to sample i, z i represents the low-dimensional feature corresponding to sample i, and τ represents the contrastive learning temperature parameter.

5. The method according to claim 4, wherein The guiding feature extractor and the view discriminator to perform parameter updating include: For the parameters of the view discriminator Where μ represents the learning rate; For the parameters of the feature extractor Wherein, λ represents the hyperparameter of the gradient reversal layer.

6. The method according to claim 1, wherein the identity prototype in, N represents the number of samples of category k. Represents the kth element of the multi-label of sample i, feat i Represents the image features of sample i.

7. The method according to claim 1, wherein Identity prototype triplet loss Among them, A Represents the anchor sample feature, IDP P Represents the positive sample identity prototype, IDP N Represents the negative sample identity prototype, and m represents the identity prototype triplet loss boundary.

8. A person re-identification device based on contrastive learning and adversarial training, comprising: The basic module includes: a feature extractor, an identity classifier, and a Softmax layer. The feature extractor is used to obtain image features of the image to be detected, the identity classifier is used to calculate the classification probability correction value of the image features, and the Softmax layer is used to classify the classification probability correction value to obtain the pedestrian recognition result; The basic module is further used to obtain a basic loss based on the triple loss and identity classification loss of the training basic module; and obtain a sample multi-label of the sample; A perspective confusion module provided with a gradient reversal layer is used to construct a set of perspective positive / negative sample pairs based on the multi-label samples; reduce the dimensionality of the image features of the samples; and obtain a perspective confusion loss based on the set of perspective positive / negative sample pairs and the low-dimensional image features to guide the feature extractor and the perspective discriminator to update parameters; An identity prototype module is configured to use the kth element of the sample multi-label as the classification confidence of the sample image belonging to the kth identity category, and calculate a weighted average of image features belonging to the same identity category according to the classification confidence to obtain an identity prototype of the identity category; construct an identity prototype triple based on the identity prototype, and obtain the identity prototype triple loss during training, wherein the identity prototype triple includes: an anchor sample feature, a positive sample identity prototype, and a negative sample identity prototype; The basic module is trained based on basic loss, perspective confusion loss and identity prototype triplet loss.

9. A storage medium having a computer program stored thereon, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 7.

Images