Cross-domain pedestrian re-identification method and system based on global-local graph convolutional network

Abstract

The application relates to a global-local graph convolution network-based cross-domain pedestrian re-identification method and system, which comprises the following steps: extracting global features of source domain and target domain images by using a feature extractor; performing horizontal segmentation on the global features to obtain upper body features and lower body features, and using affine transformation mapping to obtain local features with higher granularity; using a DBSCAN clustering method to generate pseudo labels for target domain images without labels, and then using a self-step learning strategy to generate reliable labels; using a graph convolution network to fuse the global features, the upper body features, the lower body features and the local features with higher granularity of the same image; storing the global features and the local features with higher granularity in a global feature storage library and a local feature storage library respectively; training a network model by using a contrast learning loss function, and using the trained network model to perform re-identification on images. The method and system are beneficial to improving the accuracy of cross-domain pedestrian re-identification.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a cross-domain pedestrian re-identification method and system based on a global-local graph convolutional network. BACKGROUND

[0002] The target of pedestrian re-identification is to retrieve the image of a pedestrian from different cameras. Advanced techniques have achieved good results in the field of deep learning. However, there are still some complex challenges, such as changes in the posture of pedestrians, changes in the color of clothes, changes in light, occlusions and some background noise, which often occur in complex scenes, and solving domain differences is the most critical means to solve the above problems.

[0003] There are mainly two methods to solve this problem here, one is unsupervised domain adaptation (UDA) and the other is unsupervised learning (USL). The former uses labeled source domain data and unlabeled target data to train the neural network. The latter completely relies on unlabeled data to train the neural network.

[0004] In the UDA method, SpCL proposes a self-paced learning method and a contrastive learning method to create more dependent clustering, which can better distinguish outlier samples and reliable samples. The MDIF method designs a graph convolutional network based on multi-information fusion, which can fuse the distance between different domains through feature fusion, thereby reducing the difference between domains. The HGA method proposes a coarse-grained and fine-grained graph pair network to extract features from a coarse-grained and fine-grained perspective. However, local fine-grained information is the key to solving domain differences, and fine-grained information has not been well mined. In addition, even though many supervised methods use fine-grained information, they lack the relationship between local features and global features. SUMMARY

[0005] The purpose of the present application is to provide a cross-domain pedestrian re-identification method and system based on a global-local graph convolutional network, which is beneficial to improve the accuracy of cross-domain pedestrian re-identification.

[0006] To achieve the above purpose, the technical scheme adopted by the present application is as follows: a cross-domain pedestrian re-identification method based on a global-local graph convolutional network, comprising the following steps:

[0007] A. extracting global features G of source domain and target domain images by using a feature extractor i ;

[0008] B. horizontally cutting the global features to obtain upper body features U i and lower body features L iand use affine transformation to map to get higher granularity local feature F i ;

[0009] C. Using DBSCAN clustering method to generate pseudo-labels for unlabeled target domain images, and using self-learning strategy to generate reliable labels;

[0010] D. Using graph convolution network to fuse global features, upper body features, lower body features and higher granularity local features of the same image;

[0011] E. Storing global features and higher granularity local features in global feature storage and local feature storage respectively;

[0012] F. Using contrastive learning loss function to train network model, and using trained network model to perform image re-identification.

[0013] Further, the implementation method of step A is:

[0014] Constructing a feature extractor: modifying the ResNet-50 network, retaining its basic network modules, removing the pooling after the basic network, and modifying the convolution with a step of 2 in the fourth layer residual block to a step of 1;

[0015] Then input the source domain and target domain images into the modified ResNet-50 to extract features, and obtain the ith global feature G i ∈R 2048×24×8 , wherein 2048, 24 and 8 respectively represent the number of channels, image width and image height.

[0016] Further, the implementation method of step B is:

[0017] First, cut the global feature horizontally to obtain the upper body feature and the lower body feature of the pedestrian, and the dimensions of the two features are both 2048x12x8; second, in order to obtain the higher granularity local feature F in the ith image, affine transformation is used for the upper body feature and the lower body feature, so that the mapped local feature is more accurate, and for the upper body feature U i , F i p The mapping formula is as follows:

[0018]

[0019] wherein represents the coordinate system matrix of the higher granularity feature F i p ; (x u , yu ) represents the upper body feature U i is the coordinate system matrix of the lower body feature L θ is the coefficient mapping matrix of the affine transformation; θ ij is the coefficient, i = {1, 2} and j = {1, 2}, b l is the bias quantity, l = {0, 1}; for the lower body feature L i , the same method is used to map the feature into a local feature with higher granularity; then the dimension of the feature is averaged and pooled to reduce to 256 dimensions.

[0020] Further, the implementation method of the step C is:

[0021] ResNet-50 is used as the network skeleton to extract the features of the target domain image, and the features are clustered to obtain the corresponding pseudo label;

[0022] First, the DBSCAN clustering method is used to cluster the features, so that the unlabeled features obtain pseudo labels; then, the independence score and the compactness score are calculated to further generate more reliable pseudo labels, and the calculation formula of the independence score is as follows:

[0023]

[0024] Wherein is the sample set in the same cluster as the target domain feature under the standard cluster radius; represents the sample set in the same cluster as the target domain image after expanding the cluster radius; if the independence score is high, it means that after expanding the radius of the cluster, no more samples are included; the calculation formula of the compactness score is as follows:

[0025]

[0026] Wherein, is the sample set in the same cluster as the target domain image after reducing the cluster radius; a higher compactness score means that if the radius of the cluster is reduced, more clusters are generated; according to the independence and compactness scores, two threshold values α and β are set respectively, if S indep > α and S comp > β, the cluster is retained, otherwise all samples in the cluster are judged as unreliable outliers; all features in a cluster are assigned the same pseudo label; through the above strategy, more reliable clusters are obtained, and more reliable pseudo labels are assigned to each feature.

[0027] Further, in step D, the graph convolutional network contains four features X = {G', U', L', F'}, i.e. 256-dimensional global features after dimension reduction Upper body features Lower body features and more fine-grained local features The global features, upper body features, lower body features and local features correspond to the nodes in the three-layer graph convolutional network respectively; the graph convolutional network is represented as Graph = (X, E), where X represents the set of nodes, and E represents the set of edges;

[0028] The set of edges in the graph convolutional network is represented as an adjacency matrix A (l) using 0-1 values and Euclidean distance as the weight of the edge value; the adjacency matrix is represented as follows:

[0029]

[0030] where i and j represent two different samples respectively; d(i, j) represents the Euclidean distance between the two samples;

[0031] The forward propagation formula of the l-th layer of the graph convolutional network is represented as follows:

[0032]

[0033] where represents a learnable parameter matrix in the (l-1)-th layer; represents a normalized adjacency matrix; I represents an identity matrix, and p represents the nonlinear mapping of the LeakyReLU function; and represent the input features and output features of the l-th layer respectively, d l-1 and d l are the dimensions of the features.

[0034] Further, the implementation method of step E is as follows:

[0035] the features of the p-th part in the k-th cluster are saved in the local feature repository in the form of cluster centers, i.e. Each cluster center in the local feature repository is represented as follows:

[0036]

[0037] where C k represents the sample set in the k-th cluster; with the training of the model, the features in the local feature repository are also updated; if the p-th local feature of the k-th cluster corresponds to a cluster center​​ The cluster center is updated as follows:

[0038]

[0039] wherein, represents the set of all samples in the kth cluster in a training batch; m is a dynamic update coefficient;

[0040] For global features, all global features are saved in a global feature repository, i.e. The repository saves the global center feature of each cluster And with the training of the model, the features in the global feature library are also dynamically updated.

[0041] Further, the implementation method of the step F is:

[0042] The cosine similarity between each local sample and the cluster center feature is calculated as the loss function, and the local loss function of all fine-grained features is defined as follows:

[0043]

[0044] wherein, N F is the total number of fine-grained features; is the center of the cluster where the feature is located, and T is a set parameter; the smaller the local contrast learning loss is, the closer the sample is to the cluster center of the cluster where it is located, and the farther away from other cluster centers;

[0045] By introducing a global loss function L global to guide the neural network to learn more robust global features;

[0046] In order to balance the relationship between global and local loss functions, a hyperparameter λ is introduced, and the entire loss function is represented as follows:

[0047] L=L global +λL local .

[0048] The application also provides a global-local graph convolutional network-based cross-domain pedestrian re-identification system, which comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor, and when the processor executes the computer program instructions, the above-mentioned method steps can be realized.

[0049] Compared with the prior art, the present application has the following beneficial effects: the present application provides a global-local graph convolutional cross-domain pedestrian re-identification method and system to solve the problem of cross-domain pedestrian re-identification based on unsupervised learning, so as to learn more robust local features. In order to better segment the global features of pedestrians, the present application uses two ways of horizontal segmentation and affine transformation to obtain fine-grained local features; at the same time, the present application proposes a global-local graph convolutional network to fuse global and local feature information; in order to better utilize the fused feature information, the present application saves the feature information of each cluster in the form of cluster center in the storage library, and uses a contrastive learning loss function to compare features with features, so as to finally achieve the purpose of training the network. Compared with some famous pedestrian re-identification methods, the global-local graph convolutional network provided by the present application obtains more excellent performance on multiple cross-domain pedestrian re-identification datasets. BRIEF DESCRIPTION OF DRAWINGS

[0050] Figure 1 is a method implementation flowchart of an embodiment of the present application.

[0051] Figure 2 is a comparison result table of the method of the present application and other cross-domain pedestrian re-identification methods in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The present application will be further described below in combination with the drawings and embodiments.

[0053] It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs.

[0054] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and furthermore, it should be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0055] As shown in Figure 1 , the present embodiment provides a cross-domain pedestrian re-identification method based on deep mutual learning, comprising the following steps:

[0056] A. extracting global features G of source domain and target domain images by using a feature extractor i .

[0057] The construction feature extractor: modify the ResNet-50 network, retain its basic network module, remove the pooling behind the basic network, and modify the convolution with a step of 2 in the fourth layer residual block to a step of 1.

[0058] Then the source domain and target domain images are input into the modified ResNet-50 to extract features, and the i-th global feature G i ∈R 2048×24×8 , wherein 2048, 24, and 8 represent the channel number, image width, and image height, respectively.

[0059] B. The global feature is horizontally cut to obtain the upper body feature U i and the lower body feature L i , and an affine transformation is used to map to obtain a local feature F i with higher granularity.

[0060] First, the global feature is horizontally cut to obtain the upper body feature and the lower body feature of the pedestrian, and the dimensions of the two features are both 2048x12x8; second, in order to obtain a local feature with higher granularity in the i-th image wherein p represents the p-th local feature in the global feature, an affine transformation operation is used on the upper body feature and the lower body feature, so that the mapped local feature is more accurate, and for the upper body feature U i , the mapping formula of F i p is as follows:

[0061]

[0062] wherein represents the coordinate system matrix of the fine-grained feature F i p ; (x u , y u ) represents the coordinate system matrix of the upper body feature U i ; A θ is the coefficient mapping matrix of the affine transformation; θ ij is the coefficient (i={1, 2} and j={1, 2}), b l is the bias, and l={0, 1}; for the lower body feature L i , the same method is used to map the feature to a local feature with higher granularity; then the dimension of the feature is averaged and pooled to reduce to 256 dimensions.

[0063] C. Use the DBSCAN clustering method to generate pseudo-labels for the target domain images without labels, and use the self-step learning strategy to generate reliable labels.

[0064] ResNet-50 is used as the network framework to extract the features of the target domain image, and the features are clustered to obtain corresponding pseudo labels.

[0065] First, the DBSCAN clustering method is used to cluster the features, so that the unlabeled features obtain pseudo labels. Then, the independence score and the compactness score are calculated to further generate more reliable pseudo labels. The formula for calculating the independence score is as follows:

[0066]

[0067] Wherein is the target domain feature The sample set in the same cluster; represent the target domain image The sample set in the same cluster; if the independence score is high, it means that after expanding the cluster radius, no more samples are included; the formula for calculating the compactness score is as follows:

[0068]

[0069] Wherein, is the target domain image The sample set in the same cluster; a higher compactness score means that if the cluster radius is reduced, more clusters are generated; according to the independence and compactness scores, two thresholds α and β are set respectively, if S indep > α and S comp > β, the cluster is retained, otherwise all samples in the cluster are judged as unreliable outliers; all features in a cluster are assigned the same pseudo label; through the above strategy, more reliable clusters are obtained, and more reliable pseudo labels are assigned to each feature.

[0070] D. Use the graph convolution network to fuse the global features, upper body features, lower body features and higher granularity local features of the same image.

[0071] The graph convolution network contains four features X={G', U', L', F'}, namely the 256-dimensional global feature G' after dimension reduction Upper body features Lower body features And higher granularity local features F' Global features, upper body features, lower body features and local features correspond to nodes in the three-layer graph convolution network respectively; the graph convolution network is represented as Graph=(X, E), wherein X represents the set of nodes, and E represents the set of edges.

[0072] The set of edges in the graph convolution network is represented as an adjacency matrix A (l) In the form of 0-1 value and Euclidean distance are used as the weight of edge value respectively; the adjacency matrix is represented as follows:

[0073]

[0074] Where i and j represent two different samples respectively; d(i,j) represents the Euclidean distance between the two samples.

[0075] The forward propagation formula of the lth layer of the graph convolution network is represented as follows:

[0076]

[0077] Where represents a learnable parameter matrix in the (l-1)th layer; represents an adjacency matrix that has been normalized; I represents an identity matrix, and p represents a nonlinear mapping of the LeakyReLU function; and represent the input feature and the output feature of the lth layer respectively, d l-1 and d l are the dimensions of the features.

[0078] E. Store the global features and the local features with higher granularity in the global feature repository and the local feature repository respectively.

[0079] Save the features of all p-th parts in the k-th cluster in the form of cluster center, i.e. mean value in the local feature repository, i.e. Therefore, each cluster center in the local feature repository is represented as follows:

[0080]

[0081] Where C k represents the sample set in the k-th cluster; with the training of the model, the features in the local feature repository are also updated constantly; if the cluster center corresponding to the p-th local feature in the k-th cluster is the cluster center is updated as follows:

[0082]

[0083] Where, represents the set of all samples in the k-th cluster in a training batch; m is a dynamic update coefficient.

[0084] Similar to the local features, for the global features, all the global features are saved to a global feature repository, i.e. The global center feature of each cluster is saved in the repository And as the model is trained, the features in the global feature library are also dynamically updated.

[0085] F. The network model is trained using the contrast learning loss function, and the trained network model is used for image re-identification.

[0086] The cosine similarity between each local sample and the cluster center feature is calculated as the loss function, and the local loss function of all fine-grained features is defined as follows:

[0087]

[0088] Where, N F is the total number of fine-grained features; is the center of the cluster where the feature is located, and T is a set parameter; the smaller the local contrast learning loss, the closer the sample is to the cluster center of the cluster it belongs to, and the farther away from other cluster centers.

[0089] Similarly, by introducing a global loss function L global to guide the neural network to learn more robust global features.

[0090] In order to balance the relationship between global and local loss functions, a hyperparameter λ is introduced, and the entire loss function is represented as follows:

[0091] L=L global +λL local .

[0092] The application also provides a global-local graph convolutional network-based cross-domain pedestrian re-identification system, which includes a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, the above-mentioned method steps can be implemented.

[0093] In this embodiment, the recognition results of the method proposed by the application and other cross-domain pedestrian re-identification methods are compared. Figure 2 Table 1 shows the comparison results of the method proposed by the application and other cross-domain pedestrian re-identification methods in the PersonX-to-Market-1501 task. As can be seen from Table 1, the method of the application has better recognition effect than the existing cross-domain pedestrian re-identification method.

[0094] Those skilled in the art will appreciate that embodiments of the application can be readily used as software, hardware, or a combination of software and hardware. In one

[0095] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks can represent code, circuits, hardware components, or executable computer program instructions on a machine-readable or computer-readable storage medium which, when executed by a computer or other Figure 1 an apparatus configured to perform the functions specified in the flowchart block or blocks.

[0096] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks can represent code, circuits, hardware components, or executable computer program instructions on a machine-readable or computer-readable storage medium which, when executed by a computer or other Figure 1 an apparatus configured to perform the functions specified in the flowchart block or blocks.

[0097] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flowcharts and / or blocks can represent code, circuits, hardware components, or executable computer program instructions on a machine-readable or computer-readable storage medium which, when executed by a computer or other Figure 1 an apparatus configured to perform the functions specified in the flowchart block or blocks.

[0098] The above descriptions are only preferred embodiments of the present application, and are not intended to limit the present application to other forms. Any person skilled in the art can make modifications or alterations to the above-described embodiments without departing from the technical scope of the present application. Any simple modifications, equivalent changes and alterations made to the above-described embodiments according to the technical essence of the present application still fall within the protection scope of the present application.

Claims

1. A global-local graph convolution network-based cross-domain pedestrian re-identification method, characterized in that, The method comprises the following steps: A. extracting global features of the source domain and the target domain images using a feature extractor ; B. The global features are horizontally cut to obtain upper body features and lower body features , and an affine transformation is used to map to obtain local features with higher granularity ; C. generating pseudo labels for the unlabeled target domain images using the DBSCAN clustering method, and generating reliable labels using the self-training strategy; D. using a graph convolution network to fuse the global features, upper body features, lower body features and local features of higher granularity of the same image; E. storing the global features and the local features of higher granularity in a global feature storage and a local feature storage, respectively; F. training the network model using a contrastive learning loss function, and using the trained network model to perform re-identification on the image; The implementation method of step B is: First, the global features are horizontally segmented to obtain the upper and lower body features of the pedestrian, both with dimensions of 2048×12×8; second, to obtain finer-grained local features in the i-th image... ,in Indicates the first For each local feature, an affine transformation operation is applied to the upper and lower body features to make the mapped local features more accurate. For the upper body features... , The mapping formula is as follows: wherein represents a coordinate system matrix of the fine-grained feature ; represents a coordinate system matrix of the upper body feature ; is a coefficient mapping matrix of the affine transformation; is a coefficient, i = {1, 2} and j = {1, 2}, is a bias, l = {0, 1}; for the lower body feature , the same method is used to map the features into local features with higher fine-grained; then the dimension of the global feature, the upper body feature, the lower body feature and the local feature is average-pooled to 256 dimensions.

2. The global-local graph convolution network based cross-domain pedestrian re-identification method according to claim 1, characterized in that, The implementation method of step A is: Constructing a feature extractor: modifying the ResNet-50 network, retaining its basic network module, removing the pooling after the basic network, and modifying the convolution with a step size of 2 in the fourth layer residual block to a step size of 1; Then the source domain and target domain images are input into the modified ResNet-50 to extract features to obtain the ith global feature where 2048, 24, and 8 represent the number of channels, the image width, and the image height, respectively.

3. The global-local graph convolutional network based cross-domain person re-identification method according to claim 1, characterized in that, The implementation method of step C is: ResNet-50 is used as the network skeleton to extract the features of the target domain images, and the features are clustered to obtain corresponding pseudo labels; First, the DBSCAN clustering method is used to cluster the features, so that the unlabeled features obtain pseudo labels; then, the independence score and the compactness score are calculated to further generate more reliable pseudo labels, and the calculation formula of the independence score is as follows: wherein is the target domain feature the set of samples within the same cluster; represent the target domain feature the set of samples within the same cluster; if the independence score is high it represents that no more samples are included after expanding the cluster radius; the compactness score is calculated as follows: wherein, is the feature of the target domain after the radius of the cluster is reduced a sample set within the same cluster; a higher compactness score represents that more clusters are generated if the radius of the cluster is reduced; according to the independence and compactness scores, two threshold values are set respectively and if and , the cluster is retained, otherwise all samples in the cluster are identified as unreliable outliers; all features in a cluster are assigned the same pseudo label; more reliable clusters are obtained through the above strategy, and then more reliable pseudo labels are assigned to each feature.

4. The global-local graph convolutional network based cross-domain person re-identification method according to claim 1, characterized in that, In the step D, the graph convolution network comprises four features , i.e. 256-dimensional global features after dimension reduction , upper body features , lower body features , and local features with higher granularity ; the global features, the upper body features, the lower body features and the local features correspond to nodes in three layers of the graph convolution network respectively; the graph convolution network is represented as , wherein denotes a set of nodes, denotes a set of edges; The set of edges in the graph convolution network is represented as an adjacency matrix In the form of a 0 value and Euclidean distance as the weight of the edge value respectively; the adjacency matrix is represented as follows: wherein and denote two different samples; denotes the Euclidean distance between two samples; The forward propagation formula of the graph convolution network layer is represented as follows: The forward propagation formula of the graph convolution network layer is represented as follows: wherein denotes the input features of the layer; denotes a normalized adjacency matrix; denotes the identity matrix, denotes a non-linear mapping of the function; and denote the input features and the output features of the layer, respectively, and is the dimension of the features.

5. The global-local graph convolutional network based cross-domain person re-identification method according to claim 1, characterized in that, The implementation method of step E is: The first All the th clusters Local features In the form of cluster center, i.e. average value Stored in a local feature repository, i.e. Then, each cluster center in the local feature repository is represented as follows: wherein represents a set of samples within the th cluster; as the model is trained, the features in the local feature repository are also updated; if the th local feature of the th cluster corresponds to a cluster center , then the cluster center is updated as follows: wherein, represents a set of all samples within the kth cluster in a training batch; is a dynamic update coefficient; For global features, all global features are saved into a global feature repository, i.e. the global center feature of each cluster is saved in the repository and the features in the global feature repository are dynamically updated as the model is trained.

6. The global-local graph convolution network based cross-domain person re-identification method according to claim 5, characterized in that, The implementation method of step F is: The cosine similarity between each local sample and the cluster center feature is calculated as the loss function, and the local loss function of all local features of higher granularity is defined as follows: wherein, is the total number of fine-grained features; is the feature center of the cluster where the feature is a set parameter; the smaller the local contrastive learning loss is, the closer the sample is to the cluster center of the cluster where the sample is located, and the farther away from other cluster centers; By introducing a global loss function to guide the neural network to learn more robust global features; To balance the relationship between global and local loss function, a hyper-parameter is introduced The whole loss function is represented as follows: 。 7. A global-local graph convolutional network based cross-domain pedestrian re-identification system, characterized in that, The computer program instructions stored in the memory and capable of being executed by the processor can implement the method of any one of claims 1-6 when the processor executes the computer program instructions.

Images