Unsupervised pedestrian re-identification method based on camera perception map learning
By using a camera-aware graph learning framework, and leveraging feature extractors and graph convolutional networks to learn semantic relevance between the inside and outside of the camera, the problem of low recognition accuracy caused by camera style differences in unsupervised pedestrian re-identification is solved, and more efficient pedestrian recognition is achieved.
Patent Information
- Application Number
- CN202211082601.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-06
- Publication Date
- 2026-01-27
- Estimated Expiration
- 2042-09-06
AI Technical Summary
Existing unsupervised pedestrian re-identification methods perform poorly when dealing with camera style differences and lack effective semantic associations between inside and outside the camera, resulting in low recognition accuracy.
A camera-aware graph learning framework is adopted. The semantic correlations within and between cameras are learned through feature extractors and graph convolutional networks. Pseudo-labels are generated using clustering algorithms, an in-memory dictionary is constructed, and image samples are fused into a unified feature space through graph convolutional networks. Nonparametric contrast loss and camera adversarial loss are constructed for training until the model converges.
It improves the accuracy of pedestrian re-identification, enhances the robustness of features, and effectively improves the recognition effect.
Smart Images
Figure CN115457596B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and pattern recognition technology, specifically relating to an unsupervised pedestrian re-identification method based on camera perception map learning. Background Technology
[0002] Pedestrian re-identification aims to retrieve the same pedestrian from images captured by multiple cameras given a query image. Pedestrian re-identification technology has attracted significant attention in the field of computer vision and plays a crucial role in an increasing number of video surveillance applications. In recent years, thanks to the rapid development of deep neural networks, the performance of supervised person re-identification has been significantly improved. However, supervised models heavily rely on large amounts of pairwise labeled training data, which greatly limits their feasibility and scalability in practical applications, as manually annotating cross-camera identity labels is a time-consuming and tedious task. Therefore, there is an urgent need for an unsupervised person re-identification scheme to overcome the drawbacks of supervised models.
[0003] Currently, most existing unsupervised person re-identification methods are based on domain adaptation, requiring prior knowledge to be learned from additional, identity-labeled source datasets. These domain adaptation methods also require person identity-labeled data, and their performance is closely related to the differences between the source and target domains. Although some research focuses on using fully unsupervised methods to solve the person re-identification problem, the performance of existing methods is still far from satisfactory. Furthermore, due to the high inter-class similarity within the same camera and the significant intra-class differences between different cameras, learning identity information without paired identity labels to supervise training is a rather challenging task.
[0004] To mitigate the impact of camera style differences, many previous methods have used data augmentation to improve the robustness of the model. However, generating images with different camera styles is relatively isolated from the re-identification feature learning process and introduces additional errors. At the same time, existing methods ignore the semantic relationships within and between cameras during training, resulting in unsatisfactory efficiency and accuracy. Summary of the Invention
[0005] This invention addresses existing problems by designing a novel camera-aware map learning framework from the perspective of feature aggregation to understand the semantic correlations within and between cameras, thereby effectively improving the accuracy of pedestrian re-identification. First, a feature extractor extracts pedestrian image features and uses a clustering algorithm to obtain pseudo-labels, while simultaneously constructing an in-memory dictionary. Each camera is treated as an independent image domain, and samples are extracted from each domain to form mini-batch training samples. All image sample features from the mini-batch training samples are used to construct a camera-aware map, and a graph convolutional network is used to fuse the image samples into a unified feature space. Ideal image features are generated through feature aggregation. Next, a memory-based nonparametric contrastive loss is constructed to train the feature extractor and the graph convolutional network, and a camera adversarial loss is constructed to further train the feature extractor, enabling it to generate more robust and effective image features. The aforementioned steps are then repeated multiple times until the model converges. Finally, the feature extractor maps the pedestrian images to the feature space, and the feature vectors of the query pedestrian image and the candidate pedestrian images are compared and ranked using Euclidean distance, ultimately obtaining the pedestrian re-identification result. This invention utilizes graph convolutional networks to learn semantic correlations between and within image domains, effectively improving recognition accuracy.
[0006] To achieve the above objectives, this invention provides an unsupervised person re-identification method based on camera-perceptual map learning, which includes the following steps:
[0007] S1. Use a feature extractor to extract pedestrian image features and apply a clustering algorithm to the image feature vector to obtain pseudo-identity labels. At the same time, construct a memory dictionary to store cluster features.
[0008] S2. Treat each camera as an independent image domain and extract samples from each image domain to form mini-batch training samples;
[0009] S3. Use all image sample features from the mini-batch training samples to construct a camera perception map, use a graph convolutional network to fuse the image samples into a unified feature space, and generate ideal image features through feature aggregation.
[0010] S4. Construct a memory-based nonparametric contrastive loss to train the feature extractor and graph convolutional network, and construct a camera adversarial loss to train the feature extractor. After each training iteration, use the feature vectors in each mini-batch training sample to update the feature representation in the memory dictionary.
[0011] S5. Repeat steps S1-S4 multiple times until the model converges.
[0012] S6. Use a feature extractor to map the pedestrian image into the feature space, and compare and sort the feature vectors of the pedestrian image to be queried with the feature vectors of the candidate pedestrian images using Euclidean distance, and finally obtain the pedestrian re-identification result.
[0013] As an improvement to the present invention, the memory dictionary constructed using the clustering results in step S1 is as follows:
[0014]
[0015] in, Represents a memory dictionary, N m Indicates the number of clusters. These represent the 1st, 2nd, and Nth elements stored in the dictionary, respectively. m 1 eigenvector.
[0016] As an improvement to this invention, the memory dictionary is initialized using the average value of each cluster, specifically as follows:
[0017]
[0018] Where, m k This represents the k-th feature vector stored in the memory dictionary. This represents the set of all feature vectors in the k-th cluster. express The number of eigenvectors in f i express The i-th eigenvector in the dataset.
[0019] As an improvement to the present invention, in step S2, the pedestrian image dataset is divided into different image subdomains according to the camera that took the image, and an equal number of image samples are extracted from each image domain to form a mini-batch training sample. Specifically, this can be expressed as follows:
[0020]
[0021] Where B represents a mini-batch of training samples, b 1,i b 2,i b V,i Let N represent the i-th image sample taken by cameras 1, 2, and V, respectively, where V represents the total number of cameras and N is the image sample taken by cameras 1, 2, and V. b This represents the number of images extracted from an image domain.
[0022] As another improvement of the present invention, step S3 further includes:
[0023] S31. Use a feature extractor to extract feature vectors from pedestrian images in a small batch of training samples;
[0024] S32. Use all image sample features from the mini-batch training samples obtained in step S31 to construct the camera perception map. in Let represent a directed graph, and ε represent the set of edges in the directed graph. It represents the set of nodes in a directed graph, where each node in the set is associated with an image feature vector;
[0025] The weight of ε is given by an N. a ×N a Matrix of size Let N represent this, where N a =V×N b This represents the number of image samples in the mini-batch training samples; the element assignment rules in the matrix are as follows:
[0026]
[0027] in, Let y represent the value of the element in the i-th row and j-th column of the matrix. i ,y j c represents the pseudo-labels of the i-th and j-th pedestrian images in the mini-batch training samples. i ,c j Represents the camera labels of the i-th and j-th pedestrian images in the mini-batch training samples;
[0028] S33. Add a self-connect and normalize the matrix as follows:
[0029]
[0030] Where A represents the standardized matrix, D represents the degree matrix, and I represents the identity matrix;
[0031] S34. Use a graph convolutional network to update the node features on the camera's perception map. Input matrix A and the node set into the graph convolutional network to obtain the updated node feature set; the propagation rule of node features in the graph convolutional network is defined as:
[0032]
[0033] in, Let G represent the node feature sets propagated to the l-th layer and the (l-1)-th layer, respectively. (l) This represents the l-th layer of a convolutional neural network;
[0034] The final set of node features after the graph convolutional network update is as follows:
[0035]
[0036] in, Let n represent the set of node features output by the graph convolutional network. a This represents the number of training samples in a mini-batch training sample. These represent the 1st, 2nd, and Nth outputs of the graph convolutional network, respectively. a Individual node characteristics.
[0037] As a further improvement of the present invention, in step S4, a memory-based nonparametric contrastive loss is constructed to train the feature extractor, wherein the contrastive loss function is:
[0038]
[0039] in, Let f represent the contrastive loss function used to train the feature extractor, E represent the expected value, log represent the logarithm, exp represent the Euler number exponentiation, and f q This represents the feature vector of the query instance. m represents the positive clustering feature vector of the query instance in the feature dictionary. i Let represent the i-th eigenvector in the feature dictionary, and τ1 represent the temperature hyperparameter.
[0040] A memory-based nonparametric contrastive loss function is constructed to train a graph convolutional network, wherein the contrastive loss function is:
[0041]
[0042] in, This represents the contrastive loss function used to train the graphical convolutional network. This represents the feature vector of a query instance in a graph convolutional network. The feature dictionary contains information about z. q The positive clustering feature vector, where τ2 represents the temperature hyperparameter.
[0043] The feature extractor is trained by constructing a camera adversarial loss function, wherein the adversarial loss function is:
[0044]
[0045] in, Let φ represent the camera adversarial loss function, b represent the feature extractor, and b represent the feature loss function. c,i Let represent the i-th image sample taken by camera c, and D represent the discriminator.
[0046] As a further improvement of the present invention, the update rule for the k-th feature vector in the memory dictionary in step S4 is as follows:
[0047]
[0048] in, Let represent the k-th feature vector in the memory dictionary during the T-th and T-1-th training iterations, respectively; α represent the momentum update coefficient; and f represent the pedestrian image feature vector in the mini-batch training samples corresponding to the k-th feature vector in the memory dictionary.
[0049] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0050] (1) This invention proposes an unsupervised pedestrian re-identification framework based on camera perception map learning. This framework can learn the semantic correlation between cameras and between cameras from the perspective of feature aggregation, thereby enhancing the robustness of pedestrian image representation and effectively improving the accuracy of pedestrian re-identification.
[0051] (2) The present invention designs a novel directed graph to represent the relationship between image features that have the same identity between cameras and different identities within cameras.
[0052] (3) In order to generate ideal feature representations, we use graph convolutional networks to fuse image samples into a unified space and realize semantic transfer in the feature space. Attached Figure Description
[0053] Figure 1 This is a flowchart of the unsupervised pedestrian re-identification method based on camera-perceptual map learning of the present invention;
[0054] Figure 2 This is a schematic diagram of the framework of the unsupervised pedestrian re-identification system of the present invention. Detailed Implementation
[0055] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.
[0056] Example 1
[0057] An unsupervised person re-identification method based on camera-perceptual map learning, such as Figure 1 As shown, the specific implementation of this method includes the following steps:
[0058] The specific implementation process of step S1 is as follows:
[0059] S11. Use a feature extractor to extract features from all pedestrian images in the dataset without identity labels;
[0060] S12. The DBSCAN clustering algorithm is applied to image feature vectors to generate pseudo-identity labels. In this process, the image feature vectors are divided into different feature clusters. Outlier samples that are not clustered will not participate in the training process of the model.
[0061] S13. Before training begins, design a memory-based dictionary to store cluster features:
[0062]
[0063] in, Represents a memory dictionary, N m Indicates the number of clusters. These represent the 1st, 2nd, and Nth elements stored in the dictionary, respectively. m 1 eigenvector;
[0064] The dictionary is initialized using the average representation of all instances belonging to each cluster:
[0065]
[0066] Where, m k This represents the k-th feature vector stored in the dictionary. This represents the set of all feature vectors in the k-th cluster. express The number of eigenvectors in f i express The i-th eigenvector in;
[0067] In this embodiment, the feature extractor uses a ResNet-50 pre-trained on ImageNet as the backbone network. The maximum distance between two samples in DBSCAN is 0.6, and the minimum neighbor element in the core point is 4.
[0068] The specific implementation process of step S2 is as follows:
[0069] S21. Divide the pedestrian image dataset into different image subdomains according to the camera that took the image, and extract an equal number of image samples from each image domain to form a mini-batch training sample. This can be represented as follows:
[0070]
[0071] Where B represents a mini-batch of training samples, b 1,i b 2,i b V,i Let N represent the i-th image sample taken by cameras 1, 2, and V, respectively, where V represents the total number of cameras and N is the image sample taken by cameras 1, 2, and V. b This represents the number of images extracted from an image domain.
[0072] S22. Perform preprocessing on pedestrian images in a small batch of training samples, such as resizing, random flipping, edge filling, random cropping, and random erasing.
[0073] In this embodiment, the input image size is fixed at 256 pixels × 128 pixels, the probability of random flipping and random erasing is 0.5, the width of the fill edge is 10 pixels, and the size of the random crop is 256 pixels × 128 pixels. b Set to 16, V value is 8.
[0074] refer to Figure 2 The specific implementation process of step S3 is as follows:
[0075] S31. Utilize the feature vector of pedestrian images using a feature extractor;
[0076] S32. Use all image sample features from the mini-batch training samples to construct a camera perception map. in Let represent a directed graph, and ε represent the set of edges in the directed graph. This represents the set of nodes in a directed graph, where each node is associated with an image feature vector. In practice, the weights of ε are represented by an N. a ×N a Matrix of size Let N represent this, where N a =V×N b This represents the number of image samples in the mini-batch training samples. The element assignment rules in the matrix are as follows:
[0077]
[0078] in, Let y represent the value of the element in the i-th row and j-th column of the matrix. i ,y j c represents the pseudo-labels of the i-th and j-th pedestrian images in the mini-batch training samples. i ,c j Represents the camera labels of the i-th and j-th pedestrian images in the mini-batch training samples;
[0079] S33. Add a self-connect and normalize the matrix as follows:
[0080]
[0081] Where A represents the standardized matrix, D represents the degree matrix, and I represents the identity matrix;
[0082] S34. Use a graph convolutional network to update the node features on the camera's perception map. Input matrix A and the node set into the graph convolutional network to obtain the updated node feature set; the propagation rule of node features in the graph convolutional network is defined as:
[0083]
[0084] in, Let G represent the node feature sets propagated to the l-th layer and the (l-1)-th layer, respectively. (l) This represents the l-th layer of a convolutional neural network. The final set of node features after the convolutional network update is:
[0085]
[0086] in, This represents the set of node features output by the graph convolutional network. These represent the 1st, 2nd, and Nth outputs of the graph convolutional network, respectively. a Each node's characteristics;
[0087] In this embodiment, the graph convolutional network has 1 layer, the discriminator is a binary classifier with an input dimension of 1024 and an output dimension of 2, the two hidden layers have a dimension of 1024, and a batch normalization layer is inserted into the convolutional layer.
[0088] The specific implementation process of step S4 is as follows:
[0089] S41. Construct a contrastive loss function to train the feature extractor. The contrastive loss function used to train the feature extractor is:
[0090]
[0091] in, Let f represent the contrastive loss function used to train the feature extractor, E represent the expected value, log represent the logarithm, exp represent the Euler number exponentiation, and f q This represents the feature vector of the query instance. m represents the positive clustering feature vector of the query instance in the feature dictionary. i τ1 represents the i-th feature vector in the feature dictionary, and τ1 represents the temperature hyperparameter.
[0092] S42. Construct a contrastive loss function to train the graph convolutional network. The contrastive loss function used to train the graph convolutional network is:
[0093]
[0094] in, This represents the contrastive loss function used to train the graphical convolutional network. This represents the feature vector of a query instance in a graph convolutional network. The feature dictionary contains information about z. q The positive clustering feature vector, τ2 represents the temperature hyperparameter; in this embodiment, τ1 is set to 0.05 and τ2 is set to 0.05.
[0095] S43. Design a discriminator to determine whether a feature vector comes from a feature extractor or a graph convolutional network.
[0096] S44. Constructing a camera adversarial loss to train the feature extractor:
[0097]
[0098] in, Let φ represent the camera adversarial loss function, b represent the feature extractor, and b represent the feature loss function. c,i Let represent the i-th image sample taken by camera c, and D represent the discriminator;
[0099] S45. After each training iteration, the feature vectors from each mini-batch of training samples are used to update the feature representations in the memory dictionary. Formally, the update rule for the k-th feature vector in the memory dictionary is as follows:
[0100]
[0101] in, Let represent the k-th feature vector in the memory dictionary in the T-th and T-1-th training iterations, respectively; α represents the momentum update coefficient; and f represents the pedestrian image feature vector corresponding to the k-th feature vector in the memory dictionary in the mini-batch training samples.
[0102] Step S5: Repeat steps S1-S4 multiple times until the model converges;
[0103] Step S6: After training, feature vectors of pedestrian images are extracted using a feature extractor. For the feature vector to be queried, the Euclidean distance between it and candidate feature vectors is calculated, and the candidate pedestrian identity corresponding to the smallest Euclidean distance is finally selected as the recognition result. This invention utilizes graph convolutional networks to learn semantic correlations between and within the image domain, effectively improving recognition accuracy.
[0104] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.
Claims
1. An unsupervised person re-identification method based on camera-perceptual map learning, characterized in that, The method includes the following steps: S1. Use a feature extractor to extract pedestrian image features and apply a clustering algorithm to the image feature vector to obtain pseudo-identity labels. At the same time, construct a memory dictionary to store cluster features. S2. Treat each camera as an independent image domain and extract samples from each image domain to form mini-batch training samples; S3. Use all image sample features from the mini-batch training samples to construct a camera perception map, use a graph convolutional network to fuse the image samples into a unified feature space, and generate ideal image features through feature aggregation. S31. Use a feature extractor to extract feature vectors from pedestrian images in a small batch of training samples; S32. Use all image sample features from the mini-batch training samples obtained in step S31 to construct the camera perception map. in Let represent a directed graph, and ε represent the set of edges in the directed graph. It represents the set of nodes in a directed graph, where each node in the set is associated with an image feature vector; The weight of ε is given by an N. a ×N a Matrix of size Let N represent this, where N a =V×N b This represents the number of training samples in a mini-batch of training samples; the element assignment rules in the matrix are as follows: in, Let y represent the value of the element in the i-th row and j-th column of the matrix. i ,y j c represents the pseudo-labels of the i-th and j-th pedestrian images in the mini-batch training samples. i ,c j N represents the camera labels of the i-th and j-th pedestrian images in the mini-batch training samples. b This represents the number of images extracted from an image domain; S33. Add a self-connect and normalize the matrix as follows: Where A represents the standardized matrix, D represents the degree matrix, and I represents the identity matrix; S34. Use a graph convolutional network to update the node features on the camera's perception map. Input matrix A and the node set into the graph convolutional network to obtain the updated node feature set; the propagation rule of node features in the graph convolutional network is defined as: in, Let G represent the node feature sets propagated to the l-th layer and the (l-1)-th layer, respectively. (l) This represents the l-th layer of a convolutional neural network; The final set of node features after the graph convolutional network update is as follows: in, N represents the set of node features output by the graph convolutional network. a This represents the number of training samples in a mini-batch training sample. These represent the 1st, 2nd, and Nth outputs of the graph convolutional network, respectively. a Each node's characteristics; S4. Construct a memory-based nonparametric contrastive loss to train the feature extractor and graph convolutional network, and construct a camera adversarial loss to train the feature extractor. After each training iteration, use the feature vectors in each mini-batch training sample to update the feature representation in the memory dictionary. S41. Construct a contrastive loss function to train the feature extractor. The contrastive loss function used to train the feature extractor is: in, This represents the contrastive loss function used to train the feature extractor. The expression represents mathematical expectation, log represents logarithmic operation, exp represents the exponential operation of Euler's number, and f q This represents the feature vector of the query instance. m represents the positive clustering feature vector of the query instance in the feature dictionary. i Let N represent the i-th eigenvector in the feature dictionary, τ1 represent the temperature hyperparameter, and N... m Indicates the number of clusters; S42. Construct a contrastive loss function to train the graph convolutional network. The contrastive loss function used to train the graph convolutional network is: in, This represents the contrastive loss function used to train the graphical convolutional network. This represents the feature vector of a query instance in a graph convolutional network. The feature dictionary contains information about z. q The positive clustering feature vector, where τ2 represents the temperature hyperparameter; τ1 is set to 0.05, and τ2 is set to 0.05; S43. Design a discriminator to determine whether a feature vector comes from a feature extractor or a graph convolutional network. S44. Constructing a camera adversarial loss to train the feature extractor: in, Let φ represent the camera adversarial loss function, b represent the feature extractor, and b represent the feature loss function. c,i Let represent the i-th image sample taken by camera c, and D represent the discriminator; S45. After each training iteration, the feature vectors from each mini-batch of training samples are used to update the feature representations in the memory dictionary. Formally, the update rule for the k-th feature vector in the memory dictionary is as follows: in, Let represent the k-th feature vector in the memory dictionary in the T-th and T-1-th training iterations, respectively; α represents the momentum update coefficient; and f represents the pedestrian image feature vector corresponding to the k-th feature vector in the memory dictionary in the mini-batch training samples. S5. Repeat steps S1-S4 multiple times until the model converges. S6. Use a feature extractor to map the pedestrian image into the feature space, and compare and sort the feature vectors of the pedestrian image to be queried with the feature vectors of the candidate pedestrian images using Euclidean distance, and finally obtain the pedestrian re-identification result.
2. The unsupervised person re-identification method based on camera-perceptual map learning as described in claim 1, characterized in that, The memory dictionary constructed using the clustering results in step S1 is as follows: in, Represents a memory dictionary, N m Indicates the number of clusters. These represent the 1st, 2nd, and Nth elements stored in the dictionary, respectively. m 1 eigenvector.
3. The unsupervised person re-identification method based on camera-perceptual map learning as described in claim 1, characterized in that... The initialization rule for the memory dictionary in step S1 is as follows: Where, m k This represents the k-th feature vector stored in the memory dictionary. This represents the set of all feature vectors in the k-th cluster. express The number of eigenvectors in f i express The i-th eigenvector in the matrix.
4. The unsupervised person re-identification method based on camera-perceptual map learning as described in claim 1, characterized in that, In step S2, the pedestrian image dataset is divided into different image subdomains according to the camera used to capture the image. An equal number of image samples are extracted from each image domain to form a mini-batch training sample. Specifically, this can be represented as follows: Where B represents a mini-batch of training samples, b 1,i b 2,i b V,i Let N represent the i-th image sample taken by cameras 1, 2, and V, respectively, where V represents the total number of cameras and N is the image sample taken by cameras 1, 2, and V. b This represents the number of images extracted from an image domain.