A gait recognition method and system based on unsupervised domain adaptation

Abstract

The present invention relates to a gait recognition method and system based on unsupervised domain adaptation. First, the original gait silhouette sequence is obtained. The unsupervised adaptive gait recognition network is used to assign pseudo labels to the unlabeled data, and the labeled source domain data and the unlabeled target domain data are sampled as input data respectively. The backbone network is used to extract the input sequence features to obtain the fine-grained spatiotemporal features of the source domain and the target domain. For each local feature, the global environment aggregation is used to establish the association between each feature in the time dimension, and the global feature is extracted. The global motion pattern of each part is constrained by the triplet loss, and the triplet loss is calculated by weighted summation. Each normalized feature is used to initialize and update the hybrid memory unit, and supervision is provided for the unlabeled data during the training process. The present invention can learn knowledge from the labeled source domain data and transfer it to the target domain, effectively realizing high-precision gait recognition on unlabeled data.

Description

Technical Field

[0001] The present invention relates to a gait recognition method and system based on unsupervised domain adaptation, and belongs to the technical field of computer vision and biometrics. Background Art

[0002] Gait recognition is a biometric technology that identifies individuals based on their unique walking patterns. It can identify pedestrians of the same identity at distances of more than 50 meters from multiple cameras at different angles, regardless of their attire. Recent advances in deep learning and the availability of large amounts of labeled data have greatly facilitated the application of gait recognition. However, accurately labeling large amounts of data remains a daunting challenge. This is particularly true when using cameras at long distances, where key information such as faces is often blurred. This makes it extremely difficult to label gait sequences of the same individual across different clothing, viewing angles, and carrying conditions.

[0003] Significant progress has been made in person re-identification (PRI) methods based on unsupervised adaptation. State-of-the-art methods often employ clustering algorithms, such as k-means, density-based clustering, and minimum entropy clustering, to generate pseudo-labels for unlabeled data from global features. However, due to inherent differences in data between gait recognition and person re-identification, applying unsupervised domain adaptation methods to gait recognition remains a significant challenge. Specifically, person re-identification data typically contains informative features, such as texture and color, which allow for correlation in the unsupervised RI task even without labels. In contrast, gait recognition data consists of black-and-white silhouette sequences, lacking any texture information, posing a significant challenge for unsupervised tasks. Furthermore, gait data exhibits significantly greater intra-class variability than inter-class variability, further complicating the challenges of unsupervised gait recognition methods.

[0004] Gait recognition methods based on unlabeled data are currently divided into two main categories: self-supervised gait recognition methods based on unlabeled data and unsupervised domain adaptive gait recognition methods based on unlabeled data. Self-supervised methods based on unlabeled data perform random data augmentation on the input sequence to obtain two sets of input data, which are then fed simultaneously into the network to learn the differences between them. While self-supervised methods based on unlabeled data have relatively simple tasks for gait recognition, they rely heavily on data augmentation methods. Specifically, from a kinematic perspective, the gait process is highly complex, involving the coordinated control between various muscles and brain centers in the human body. Muscle distribution, skeletal structure, and motor nerves vary from person to person due to differences in upbringing and individual genetics. Therefore, while the gait characteristics of the same person are relatively stable across different scenarios, they vary between individuals, and data augmentation methods cannot simulate all scenarios in real-world scenarios. Currently, self-supervised methods based on unlabeled data perform poorly, making them difficult to apply in real-world scenarios.

[0005] Gait recognition methods based on unsupervised domain adaptation primarily learn gait knowledge from labeled source domain data and then use this knowledge to assign pseudo-labels to unlabeled data. The network then updates and learns from the pseudo-labeled data and uses this new knowledge to further assign pseudo-labels to unlabeled data. This iterative update allows the network to continuously learn from unlabeled data.

[0006] In order to apply gait recognition methods to large-scale unlabeled data, or to meet the application of gait recognition under realistic conditions, a gait recognition neural network technology with high recognition accuracy is urgently needed. Summary of the Invention

[0007] The present invention addresses the shortcomings and deficiencies of existing technologies. To address the problems of existing methods, such as neglecting the fine-grained classification problem of gait recognition and the lack of unsupervised methods for gait recognition, a method and system for gait recognition based on unsupervised domain adaptation are creatively proposed. This invention can learn knowledge from labeled source domain data and transfer it to the target domain, effectively achieving high-precision gait recognition on unlabeled data.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] In one aspect, the present invention proposes a gait recognition method based on unsupervised domain adaptation, comprising the following steps:

[0010] Step 1: Obtain the original gait video data from the camera and obtain an unlabeled gait silhouette sequence.

[0011] Step 2: Preprocess the gait video data obtained by the camera to obtain the original gait silhouette sequence.

[0012] Specifically, data preprocessing includes extracting the contour edges of the human figure in the image, aligning the center of the human body with the center of the image, and scaling the gait image to a uniform size.

[0013] Step 3: Use the unsupervised adaptive gait recognition network to assign pseudo labels to the unlabeled data, and then sample labeled source domain data and unlabeled target domain data as input data.

[0014] Step 4: Use the backbone network to extract the features of the input sequence and obtain fine-grained spatiotemporal features of the source and target domains.

[0015] Specifically, the input sequence consists of continuously sampled frames. The number of samples can be between 20 and 40 frames, and motion changes are allowed between adjacent frames.

[0016] Furthermore, the backbone network extracts gait features from the input sequences of the source and target domains, including fine-grained motion pattern extraction and global environment aggregation.

[0017] Furthermore, fine-grained motion pattern extraction can be used to extract features from the data in the source and target domains to obtain the spatiotemporal features of each sequence.

[0018] Furthermore, horizontal pooling can be used to divide the feature map horizontally into a fixed number of features, and then optimize the local features separately.

[0019] Step 5: For each local feature, use global environment aggregation to establish the association between each feature in the time dimension and further extract the global feature.

[0020] Among them, global environment aggregation refers to further modeling the spatiotemporal features of each sequence, selecting robust global motion patterns, and removing a large amount of interference.

[0021] Furthermore, maximum pooling and global average pooling can be used to process the features to obtain the most expressive gait features.

[0022] Step 6: Use the triplet loss to constrain the global motion pattern of each part, and calculate the triplet loss using a weighted summation method.

[0023] Specifically, the global motion pattern is batch normalized to map the features onto the Gaussian sphere, and the cross-entropy loss is used to constrain the normalized features of each part, and then the cross-entropy loss is calculated using the weighted summation method.

[0024] Step 7: Each normalized feature is used to initialize and update the hybrid memory unit and provide supervision on unlabeled data during training.

[0025] Specifically, the backbone network extracts features of all data in the source and target domains, assigns pseudo labels to unlabeled target domain data using a clustering algorithm, and then extracts class center features from data of all labels to initialize the hybrid memory unit.

[0026] During the network training process, the features of the hybrid memory unit are used to calculate the contrast loss of the sampled data, and the features stored in the hybrid memory unit are dynamically updated according to the sampled features.

[0027] Finally, the back-propagation algorithm is used to iteratively update the parameters of the gait neural network.

[0028] On the other hand, in order to achieve the purpose of the present invention, according to the above method, the present invention further proposes a gait recognition system based on unsupervised domain adaptation, including an image preprocessing module, a fine-grained motion pattern extraction module, a horizontal segmentation module, a global environment aggregation module, and a hybrid memory module.

[0029] The image preprocessing module is used to scale the size of each frame image in the gait silhouette sequence to a uniform size;

[0030] The fine-grained motion pattern extraction module is used to extract features of source domain data and target domain data, store useful gait knowledge learned from the source domain data, and transfer it to the target domain to obtain correct pseudo labels.

[0031] The horizontal segmentation module is used to horizontally segment the height dimension of the feature map, dividing the complete gait feature into local features, thereby constraining each local feature;

[0032] The global environment aggregation module is used to extract global spatiotemporal features for each part and select robust motion patterns in the temporal dimension using maximum pooling and global average pooling.

[0033] The hybrid memory module is used to store the class center features of the source domain data and the target domain data, and supervise the source domain data and the target domain data.

[0034] The relationship between the above modules is as follows:

[0035] The output end of the image preprocessing module is connected to the input end of the fine-grained motion pattern processing module; the output end of the fine-grained motion pattern processing module is connected to the input end of the horizontal segmentation module; the output end of the horizontal segmentation module is connected to the input end of the global environment aggregation module; the output end of the global environment aggregation module is connected to the input end of the hybrid memory module;

[0036] Beneficial effects

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] The present invention effectively solves the technical problem of learning gait features from large-scale unlabeled data, as well as the technical problem of low accuracy of gait recognition networks due to the lack of an unsupervised framework, and provides a simple and effective technical solution for identifying pedestrians in real-world scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is an overall flow chart of the method of the present invention;

[0040] Figure 2 It is an algorithm flow chart of the method of the present invention;

[0041] Figure 3 A pseudo-label generation graph for the method of the present invention;

[0042] Figure 4 Schematic diagram of the structure of the system of the present invention. DETAILED DESCRIPTION

[0043] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0044] Example

[0045] like Figure 1 As shown, a gait recognition method based on unsupervised domain adaptation includes the following steps:

[0046] Step S10: Obtain original gait video data from the camera and obtain an unlabeled gait silhouette sequence.

[0047] Step S20: Data preprocessing mainly includes extracting the outline edges of the human figure in the image, aligning the center of the human body with the center of the image, and scaling the gait image to a uniform size. The labeled data is used as the source domain data, and the unlabeled data is used as the target domain data.

[0048] In the embodiment of the present invention, the size of the gait images is uniformly scaled to 64×64.

[0049] Step S30: The labeled source domain data and the unlabeled target domain data are simultaneously input into the neural network, and gait features are obtained after feature extraction.

[0050] like Figure 2 As shown, step S30 includes 3 sub-steps:

[0051] Step S30 - 1 : Utilize a fine-grained motion pattern extraction module to extract gait features from labeled source domain data and unlabeled target domain data.

[0052] Specifically, fine-grained motion pattern extraction includes feature extraction and horizontal segmentation.

[0053] Furthermore, fine-grained motion pattern extraction consists of cyclic time-shifted convolution. For the gait feature V, For the feature dimension C, it is divided into s parts, v i Indicates the characteristics of each part, Represents the real number domain, C represents the number of channels, T represents the number of sampling frames, H represents the height of the feature, and W represents the width of the feature. The feature dimension C is divided into s parts, and then the features of each part are cyclically shifted.

[0054]

[0055] in, Represents the characteristics of the first part of the right circular shift, Represents the Tth feature in the time domain dimension of the first part, Represents the characteristics of the second part of the left circular shift, Represents the Tth feature in the time domain dimension of the second part.

[0056] For v1 and v2, right shift and left shift operations are performed along the time dimension respectively. When shifting right, the features of the last frame are added to the first frame, and when shifting left, the features of the first frame are added to the last frame.

[0057] Utilizing cyclic time-shifted convolution, 2D convolution can be used to extract temporal features.

[0058] Furthermore, the maximum pooling layer adopts a specially designed 1×2×2 step size. This pooling method only pools the width dimension of the feature map, fully retaining the most important time and height dimensions in the feature map.

[0059] Furthermore, the lateral segmentation divides the gait feature graph horizontally along the height dimension, dividing the global gait features into local features, and constraining and processing them separately.

[0060] H=MaxPool3d(V)+AvgPool3d(V) (2)

[0061] Among them, H represents the local gait features obtained by horizontal segmentation, V represents the gait features extracted by the fine-grained motion pattern network, MaxPool3d represents maximum pooling, and AvgPool3d represents global average pooling.

[0062] Step S30-2: For each local feature, use global environment aggregation to establish a connection between global features, and further obtain a discriminative global spatiotemporal feature while removing a large number of interference points.

[0063] Among them, motion mode selection is to further establish the global spatiotemporal relationship from the primary features of the points, and then use the primary features to combine into a more distinctive gait feature.

[0064] Furthermore, maximum pooling can be used to process the time dimension of the features and select the most discriminative gait features.

[0065] Step S30-3: Use triplet loss, cross entropy loss, and contrast loss functions to constrain local features respectively, and use the weighted summation method to calculate the final loss. Finally, use the backpropagation algorithm to iteratively update the parameters of the gait neural network:

[0066] L tr =Max(d(a,p)-d(a,n)+m) (3)

[0067] Among them, L tr represents the triplet loss, d(a,p) represents the Euclidean distance between the selected sample and the positive sample pair with the same label, a represents the anchor point, and p represents the positive sample; d(a,n) represents the Euclidean distance between the selected sample and the negative sample pair with different labels, n represents the negative sample; m represents the Euclidean distance difference between the selected sample and the positive sample pair and the negative sample pair.

[0068]

[0069] Among them, L ce represents the cross entropy loss, y represents the label of the sample, Represents the predicted label.

[0070] L_co=-log{exp(f,c + ) / (∑(f,c))}

[0071] L c =L tr +αL ce +βL co (5)

[0072] Among them, L co represents the contrast loss function, f represents the input feature, c + represents the class center feature with the same label as the input feature, and c represents all the stored class center features. The final loss function L cIt is a weighted sum of triplet loss, cross entropy loss, and contrast loss. α and β are hyperparameters and can usually be set to 0.3.

[0073] Step S40: using a clustering algorithm to assign pseudo labels to the unlabeled data, and using the pseudo labels to update the neural network.

[0074] like Figure 3 As shown, step S40 includes three sub-steps:

[0075] Step S40 - 1 : using a clustering algorithm to assign pseudo labels to unlabeled data.

[0076] Specifically, the clustering method can adopt a density-based clustering algorithm to assign a pseudo label to each unlabeled feature.

[0077] Furthermore, during the training process, only samples with assigned pseudo labels are sampled, and discrete points do not participate in the training.

[0078] Step S40 - 2 : Extracting class center features from labeled data and data to which pseudo labels have been assigned.

[0079] Specifically, the class center is directly extracted for labeled data, while the class center is extracted for pseudo-labeled data according to the assigned label.

[0080] Step S40-3: Initialize the hybrid memory module using the class center features of the labeled data and the unlabeled data, and provide supervision during the training process.

[0081] Step S50: The gait features extracted by the neural network are matched with the features in the storage library. By calculating the similarity between the sequences, a matching sequence with a known identity is obtained, thereby obtaining the pedestrian's identity information. Finally, the pedestrian's features are updated to the storage library as known pedestrian features.

[0082] Figure 4 This is a schematic diagram of the system structure of the present invention:

[0083] like Figure 4 As shown, a gait recognition system based on unsupervised domain adaptation includes an image acquisition module M10, an image preprocessing module M20, a gait recognition module M30 and a gait retrieval module M40, wherein:

[0084] The image acquisition module M10 is used to acquire the original gait video data from the camera, obtain a gait silhouette sequence, and store the gait video data in the server.

[0085] Image preprocessing module M20 is used to extract the outline edges of the human figure in the image, align the center of the human body with the center of the image, and scale the gait image to a uniform size. The labeled data is used as the source domain data, and the unlabeled data is used as the target domain data.

[0086] The gait recognition module M30 first extracts gait features from labeled source and unlabeled target domain data using the fine-grained motion pattern extraction module. It then uses the global context aggregation module to establish connections between each local feature and further obtain discriminative global spatiotemporal features while removing interference points. The gait recognition module M30 includes a fine-grained feature extraction submodule M31 and a motion pattern selection submodule M32.

[0087] The gait retrieval module M40 is used to match the features of the unknown identity sequence with the features of the storage library, obtain the identity information of the pedestrian, and then update the features and store them in the library.

[0088] The connection relationship of the above modules is:

[0089] The output of the image acquisition module M10 is connected to the input of the image pre-processing module M20.

[0090] The output of the image preprocessing module M20 is connected to the input of the gait recognition module M30. Inside the gait recognition module M30, the output of the fine-grained motion pattern extraction submodule M31 is connected to the input of the global environment aggregation submodule M32.

[0091] The output end of the gait recognition module M30 is connected to the input end of the gait retrieval module M40.

[0092] The above-mentioned technical means can solve the problem of learning gait features from large-scale unlabeled data and the problem of low accuracy of gait recognition network due to the lack of an unsupervised framework, providing a gait recognition neural network method and system with a relatively simple and effective network structure for deployment in real-world scenarios.

Claims

1. A gait recognition method based on unsupervised domain adaptation, characterized in that: The following steps are involved: Step 1: Obtain the original gait video data from the camera and obtain an unlabeled gait silhouette sequence; Step 2: Preprocess the gait video data obtained by the camera to obtain the original gait silhouette sequence; Data preprocessing includes extracting the outline edges of the human figure in the image, aligning the center of the human body with the center of the image, and scaling the gait image to a uniform size. Step 3: Use the unsupervised adaptive gait recognition network to assign pseudo labels to the unlabeled data, and then sample the labeled source domain data and the unlabeled target domain data as input data; Step 4: Use the backbone network to extract the features of the input sequence and obtain fine-grained spatiotemporal features of the source and target domains; Step 5: For each local feature, use global environment aggregation to establish the association between each feature in the time dimension and further extract the global feature; Among them, global environment aggregation refers to further modeling the spatiotemporal characteristics of each sequence, selecting robust global motion patterns, and removing interference; Step 6: Use triplet loss to constrain the global motion mode of each part, and calculate the triplet loss using a weighted summation method; Step 7: Each normalized feature is used to initialize and update the hybrid memory unit and provide supervision for unlabeled data during training; During network training, the features of the hybrid memory unit are used to calculate the contrast loss of the sampled data, and the features stored in the hybrid memory unit are dynamically updated according to the sampled features; Finally, the back-propagation algorithm is used to iteratively update the parameters of the gait neural network.

2. The gait recognition method based on unsupervised domain adaptation according to claim 1, characterized in that: In step 2, the input sequence consists of continuously sampled frames. The number of samples is between 20 and 40 frames, and motion changes are allowed between adjacent frames.

3. The gait recognition method based on unsupervised domain adaptation according to claim 1, characterized in that: In step 2, the backbone network extracts gait features from the input sequences of the source and target domains, including fine-grained motion pattern extraction and global environment aggregation.

4. The gait recognition method based on unsupervised domain adaptation according to claim 1, characterized in that: In step 2, fine-grained motion pattern extraction is used to extract features from the data in the source and target domains to obtain the spatiotemporal features of each sequence.

5. The gait recognition method based on unsupervised domain adaptation according to claim 1, characterized in that: In step 2, horizontal pooling is used to divide the feature map horizontally into a fixed number of features, and then the local features are optimized separately.

6. The gait recognition method based on unsupervised domain adaptation according to claim 4, characterized in that: Fine-grained motion pattern extraction includes feature extraction and horizontal segmentation; Fine-grained motion pattern extraction consists of cyclic time-shifted convolution. For gait feature V, V = For the feature dimension C, it is divided into s parts, v i Indicates the characteristics of each part, Represents the real number domain, C represents the number of channels, T represents the number of sampling frames, H represents the height of the feature, and W represents the width of the feature; the feature dimension C is divided into s parts, and then the features of each part are cyclically shifted; in, Represents the characteristics of the first part of the right circular shift, Represents the Tth feature in the time domain dimension of the first part, Represents the characteristics of the second part of the left circular shift, Represents the Tth feature in the time domain dimension of the second part; For v1 and v2, right shift and left shift operations are performed along the time dimension respectively. When right shifting, the features of the last frame are added to the first frame, and when left shifting, the features of the first frame are added to the last frame. Utilizing cyclic time-shifted convolution, 2D convolution can be used to extract temporal features.

7. The gait recognition method based on unsupervised domain adaptation according to claim 1, characterized in that: In step 5, the features are processed using maximum pooling and global average pooling to obtain the most expressive gait features; Among them, the maximum pooling layer adopts a specially designed 1×2×2 step size. This pooling method only pools the width dimension of the feature map; Horizontal segmentation divides the gait feature graph horizontally along the height dimension, dividing the global gait features into local features, and constraining and processing them separately; H=MaxPool3d(V)+AvgPool3d(V) Among them, H represents the local gait features obtained by horizontal segmentation, V represents the gait features extracted by the fine-grained motion pattern network, MaxPool3d represents maximum pooling, and AvgPool3d represents global average pooling.

8. The gait recognition method based on unsupervised domain adaptation according to claim 1, characterized in that: In step 6, the global motion pattern is batch normalized to map the features onto the Gaussian sphere, and the cross-entropy loss is used to constrain the normalized features of each part, and then the cross-entropy loss is calculated using the weighted summation method; The triplet loss, cross entropy loss, and contrast loss functions are used to constrain local features respectively, and the weighted summation method is used to calculate the final loss. Finally, the back-propagation algorithm is used to iteratively update the parameters of the gait neural network: L tr =Max(d(a,p)-d(a,n)+m) Among them, L tr represents the triplet loss, d(a,p) represents the Euclidean distance between the selected sample and the positive sample pair with the same label, a represents the anchor point, and p represents the positive sample; d(a,n) represents the Euclidean distance between the selected sample and the negative sample pair with different labels, n represents the negative sample; m represents the Euclidean distance difference between the selected sample and the positive sample pair and the negative sample pair; Among them, L ce represents the cross entropy loss, y represents the label of the sample, represents the predicted label; L_co=-log{exp(f,c + ) / (∑(f,c))} L c =L tr +αL ce +βL co Among them, L co represents the contrast loss function, f represents the input feature, c + represents the class center feature with the same label as the input feature, c represents all stored class center features; the final loss function L c It is a weighted sum of triplet loss, cross entropy loss and contrast loss, where α and β represent hyperparameters.

9. The gait recognition method based on unsupervised domain adaptation according to claim 1, characterized in that: In step 7, the backbone network extracts features of all data in the source and target domains, assigns pseudo labels to unlabeled target domain data using a clustering algorithm, and then extracts class center features from all labeled data to initialize the hybrid memory unit.

10. A gait recognition system based on unsupervised domain adaptation, using the gait recognition method based on unsupervised domain adaptation according to claim 1, characterized in that: It includes image preprocessing module, fine-grained motion pattern extraction module, horizontal segmentation module, global environment aggregation module, and hybrid memory module; The image preprocessing module is used to scale the size of each frame image in the gait silhouette sequence to a uniform size; The fine-grained motion pattern extraction module is used to extract features of source and target domain data, store useful gait knowledge learned from the source domain data, and transfer it to the target domain to obtain correct pseudo-labels; The horizontal segmentation module is used to horizontally segment the height dimension of the feature map, dividing the complete gait feature into local features, thereby constraining each local feature; The global environment aggregation module is used to extract global spatiotemporal features for each part and select robust motion patterns in the temporal dimension using maximum pooling and global average pooling; The hybrid memory module is used to store the class center features of the source domain data and the target domain data, and supervise the source domain data and the target domain data; The relationship between the above modules is as follows: The output end of the image preprocessing module is connected to the input end of the fine-grained motion pattern processing module; the output end of the fine-grained motion pattern processing module is connected to the input end of the horizontal segmentation module; the output end of the horizontal segmentation module is connected to the input end of the global environment aggregation module; the output end of the global environment aggregation module is connected to the input end of the hybrid memory module.

Images