A Gait Recognition Method Based on Multimodal Data Input

Abstract

This invention discloses a gait recognition method based on multimodal data input. The steps of this invention are: 1. Converting the human body parsing sequence dataset and the human skeleton sequence dataset into .pkl files; 2. Constructing a feature fusion module, which deeply fuses the features from the human body parsing module and the human skeleton module; 3. Constructing a spatiotemporal feature extraction module. The first branch of the spatiotemporal feature extraction module first uses a one-dimensional convolution with a kernel size of 3 to extract features along the time dimension, then enters the ReLU activation function, and after passing through a convolution kernel, enters the ReLU activation function again, obtaining a time-dimensional feature vector through time-dimensional pooling and convolution kernels; the second branch then uses a one-dimensional convolution to extract features along the spatial dimension, and the subsequent parts are consistent with the time-dimensional branch. This invention integrates human keypoints and fine-grained parsing sequences, while taking into account both information richness and positioning accuracy, thereby improving the performance of gait recognition.

Description

Technical Field

[0001] This invention relates to the fields of human posture estimation, gait recognition, and related fields. The specific goal is to design a novel and effective neural network to perform more accurate gait recognition by using multimodal gait data as input. Background Technology

[0002] Gait recognition refers to the authentication or identification of a person's identity through their walking posture or footprints, and is considered one of the most promising methods for long-distance identity recognition. Its advantages mainly include being contactless, requiring no pedestrian cooperation during the recognition process, and being difficult to conceal or disguise. Therefore, gait recognition has broad application prospects and economic value in fields such as security monitoring, human-computer interaction, medical diagnosis, and access control systems. However, gait recognition faces many challenges in practical applications, primarily because pedestrians are affected by external environmental and personal factors during walking (such as different road conditions, different times, different perspectives, different clothing, and different items carried), resulting in strong intra-class variations in the extracted gait features.

[0003] Currently, mainstream gait recognition methods can be divided into two categories: model-based methods, which mainly use human skeleton sequences as gait representations; and appearance-based methods. Recent research has found that using fine-grained human body analytical sequences as gait representations can extract richer information entropy. While fine-grained human body analytical sequences are information-rich, their pixel-level segmentation results in lower segmentation accuracy; whereas human skeleton sequences, although sparse, provide more accurate localization of key human points. Both mainstream methods have made some progress, but each still has its limitations. Therefore, we aim to explore a gait recognition method that integrates human key points and fine-grained analytical sequences to simultaneously balance information richness and localization accuracy, thereby improving gait recognition performance. Summary of the Invention

[0004] This invention designs a gait recognition method that combines human body analysis data and human skeleton data. In the field of deep learning, this invention is the first to propose a model that combines human body analysis data and human skeleton data.

[0005] Step 1: Convert the human body parsing sequence dataset and the human skeleton sequence dataset into pkl files;

[0006] Step 2: Construct a feature fusion module, which deeply fuses the features of the human body parsing module and the human skeleton module;

[0007] Step 3: Construct a spatiotemporal feature extraction module, which has two branches. In the first branch, the spatiotemporal feature extraction module first uses a one-dimensional convolution with a kernel size of 3 to extract features along the time dimension, then enters the ReLU activation function, and then enters the ReLU activation function again after passing through a 3*3 convolution kernel. After time-dimensional pooling, it enters the ReLU activation function again to obtain the time-dimensional feature vector through a 1*1 convolution kernel. In the second branch, the spatiotemporal feature extraction module then uses a one-dimensional convolution with a kernel size of 1 to extract features along the spatial dimension, then enters the ReLU activation function, and then enters the ReLU activation function again after passing through a 3*3 convolution kernel. After spatial-dimensional pooling, it enters the ReLU activation function again to obtain the spatial-dimensional feature vector through a 1*1 convolution kernel.

[0008] Furthermore, step 1 is implemented as follows:

[0009] The input human skeleton sequence is set according to the skeleton sequence of the COCO dataset;

[0010] The input human body parsing sequence is divided into 11 categories according to head, torso, left arm, right arm, left hand, right hand, left leg, right leg, left foot, right foot and bodysuit, corresponding to joint numbers 1-11 respectively.

[0011] Furthermore, human skeletal sequence Used for inputting the human skeleton module, where the frame number is N, 3 represents the number of channels, and P represents the joint number;

[0012] Furthermore, human body sequence analysis Used for inputting the human body parsing module, where C represents the number of channels, H and W represent the height and width of the input human body parsing sequence, respectively, and n represents the frame number, with a value range of 1-N;

[0013] The human skeleton sequence and the human parsing sequence are processed by the human skeleton module and the human parsing module, respectively, to obtain the human skeleton sequence feature S. ske and human body sequence features S par :

[0014] S ske =SF(I ske (1)

[0015] S par =CP(I par (2)

[0016] In this context, SF and CP represent the human skeleton module and the human body analysis module, respectively.

[0017] Furthermore, the human skeleton module comprises three graph convolutional networks, which are finally processed by global max pooling and global average pooling to obtain S. ske The human body analysis module represents a cross-site structure, and S is obtained through the output of the human body analysis module.par ;

[0018]

[0019] Where β is a learnable parameter used to control the weight of the k-th body part, ⊙ represents element-wise multiplication, and Q... k It is the parsing mask for the k-th body part.

[0020] Furthermore, step 2 is implemented as follows:

[0021] The input to the feature fusion module FFM is the human skeleton sequence features S. ske and human body sequence features S par Then calculate S. par and S ske The correlation between them is used as the weight matrix; based on the weight matrix, the weighted human skeleton sequence features S are... ske The calculation is performed, and finally the result is compared with the original input human body parsing sequence features S. par Human skeleton sequence features S ske The features are concatenated together and then passed through a fully connected layer to enter the next module, resulting in the fused feature F. f Human skeleton sequence features S ske The output after the spatiotemporal feature extraction module and the fully connected layer is F. ske Human body sequence features S par The output after the graph convolutional module and the fully connected module is F. par ;

[0022] W ek =Concat(H1,H2,...,H) 11 )*T k (4)

[0023]

[0024] F f =FC(Concat(S) par W ek (6)

[0025] Among them, W ek H represents the weight matrix. k T represents the correlation coefficient matrix of the corresponding k-th body part; k This is the proportion matrix of the k-th body part; Both are predefined correlation matrices for the k-th body part, T k This is a matrix showing the proportion of different body parts.

[0026] Furthermore, in step 3, the inputs to the two parallel spatiotemporal feature extraction modules are the human skeleton sequence features S. ske and fusion feature F f . Fuse features F f Human skeleton sequence features S ske The output F after the spatiotemporal feature extraction module and the fully connected layer ske Human body sequence features S par The output F after passing through the graph convolution module and the fully connected module par The three components are combined to form the final output.

[0027] The present invention has the following effects:

[0028] This invention is a gait recognition method that integrates human body key points and fine-grained analytical sequences, while taking into account both the richness of information and the accuracy of positioning, thereby improving the performance of gait recognition.

[0029] The network proposed in this invention outperforms classic methods previously proposed in the field of gait recognition in practical performance. This invention presents the first neural network that combines two different inputs: a skeleton sequence and a human body parsing sequence. It achieves an end-to-end method for multimodal inputs in gait recognition and provides a certain improvement in gait recognition performance. Attached Figure Description

[0030] Figure 1 Feature fusion module;

[0031] Figure 2 Spatiotemporal feature extraction module;

[0032] Figure 3 Overall structure; Detailed Implementation

[0033] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0034] like Figure 1-3 As shown, a gait recognition method based on multimodal data input includes the following steps:

[0035] Step 1: Convert the human body analysis dataset and the human skeleton dataset into pkl files;

[0036] Step 2: Construct the Feature Fusion Module (FFM). This module deeply fuses the features from the human body parsing module and the human skeleton module. The specific fusion process involves associating the semantics between different human body parsing regions and the skeleton. See [link to FFM structure] for details. Figure 1 .

[0037] Step 3: Construct a Spatiotemporal Feature Extraction Module (TSEM). In real-world scenarios, occlusion and other issues may occur, and gait data collected from the same viewpoint may change due to variations in the shooting angle. Therefore, a TSEM is needed to aggregate spatiotemporal features. This module has two branches. First, it uses a one-dimensional convolution with a kernel size of 3 to extract features along the time dimension, then enters a ReLU activation function, followed by a 3x3 convolution kernel before entering another ReLU activation function. Finally, it uses time-dimensional pooling followed by a 1x1 convolution kernel to obtain the time-dimensional feature vector. The module then uses a one-dimensional convolution with a kernel size of 1 to extract features along the spatial dimension, with subsequent steps consistent with the time-dimensional branch.

[0038] The input human skeleton sequence is set according to the skeleton sequence of the COCO dataset. The input human parsing sequence is divided into 11 categories according to the head, torso, left arm, right arm, left hand, right hand, left leg, right leg, left foot, right foot and bodysuit, corresponding to joint numbers 1-11 respectively.

[0039] like Figure 3 As shown, human skeleton sequence Input the human skeleton module of the network of this invention, where the frame number is N, 3 represents the number of channels, and P represents the joint number. Simultaneously, the human parsing sequence... Input the human body parsing module of the network of this invention, where C represents the number of channels, H and W represent the height and width of the input parsing sequence, respectively, and n represents the frame number, with a value range of 1-N.

[0040] The human skeleton sequence and the human parsing sequence are processed by the human skeleton module and the human parsing module, respectively, to obtain the human skeleton sequence feature S. ske and human body sequence features S par :

[0041] S ske =SF(I ske (1)

[0042] S par =CP(I par (2)

[0043] Among them, SF and CP represent the human skeleton module and the human body analysis module, respectively;

[0044] The human body analysis module corresponds to Figure 3 The content within the large dashed box at the bottom center includes the average pooling part; the human skeleton module contains three graph convolutional networks, which are finally processed by global max pooling and global average pooling to obtain S. ske ;

[0045] The human skeleton module corresponds to Figure 3 The three graph convolutional networks and transformer encoders are concatenated and then combined with global max pooling and global average pooling modules; the human body parsing module represents the cross-part structure, as shown in Formula 3, and the output of the human body parsing module is S. par .

[0046]

[0047] Where β is a learnable parameter used to control the weight of the k-th body part, ⊙ represents element-wise multiplication, and Q... k It is the parsing mask for the k-th body part.

[0048] Furthermore, step 2 is implemented as follows:

[0049] To fuse human body parsing sequences and human skeleton sequences, this patent proposes a Feature Fusion Module (FFM). The input to the FFM is the human skeleton sequence features S. ske and human body sequence features S par Then calculate S. par and S ske The correlation between them is used as a weight matrix;

[0050] Based on the weight matrix, the weighted human skeleton sequence features are calculated, and finally the calculated results are compared with the original input human parsed sequence features S. par Human skeleton sequence features S ske The features are concatenated together and then passed through a fully connected layer to enter the next module, resulting in the fused feature F. f Human skeleton sequence features S ske The output after the spatiotemporal feature extraction module and the fully connected layer is F. ske Human body sequence features S par The output after the graph convolutional module and the fully connected module is F. par .

[0051] W ek =Concat(H1,H2,...,H) 11 )*T k (4)

[0052]

[0053] F f =FC(Concat(S) par W ek (6)

[0054] Among them, W ek H represents the weight matrix. kT represents the correlation coefficient matrix corresponding to the k-th part of the body. k This is the proportion matrix of the k-th body part; Both are predefined correlation matrices for the k-th body part, T k This is a matrix showing the proportion of different body parts.

[0055] Furthermore, step 3 is implemented as follows:

[0056] To fully utilize the features of the time and space dimensions, this invention designs a spatiotemporal feature extraction module.

[0057] The inputs to the two parallel spatiotemporal feature extraction modules are the human skeleton sequence features S. ske and fusion feature F f Specifically, the spatiotemporal feature extraction module first uses a one-dimensional convolution with a kernel size of 3 to extract features along the time dimension, then enters the ReLU activation function, and then enters the ReLU activation function again after passing through a 3*3 convolution kernel. After time-dimensional pooling, it passes through a 1*1 convolution kernel to obtain the time-dimensional feature vector. The spatiotemporal feature extraction module then uses a one-dimensional convolution with a kernel size of 1 to extract features along the spatial dimension, and the subsequent parts are consistent with the time-dimensional branch.

[0058] F fusion feature F f Human skeleton sequence features S ske The output F after the spatiotemporal feature extraction module and the fully connected layer ske Human body sequence features S par The output F after passing through the graph convolution module and the fully connected module par The three components are concatenated as the final output, which facilitates subsequent calculations of the loss function and accuracy.

[0059] In this study, the triplet loss function and cross-entropy loss function were used to evaluate the model performance during training. Cosine / Euclidean distance was used during inference. Finally, the performance was tested on the Gait3D test dataset using four commonly used metrics in the gait recognition field: Rank-1, Rank-5, mAP, and mINP. The results are shown in Table 1.

[0060] Example:

[0061] As shown in Table 1, the bottom row represents the method proposed in this invention. Compared with previously proposed methods (the higher the number, the better the effect), the results demonstrate the effectiveness of the network proposed in this invention.

[0062] Table 1

[0063]

[0064] The first row of Table 1 contains four metrics: Rank-1, Rank-5, mAP, and mINP, which are commonly used evaluation metrics in the field of gait recognition. Rank-1 is a commonly used metric in retrieval tasks, representing the proportion of correct answers ranked first in the search results. In face recognition or image retrieval, if the first result retrieved is the correct match, then the Rank-1 of that search result is 1; otherwise, it is 0.

[0065] Rank-5: Similar to Rank-1, but here we consider whether the correct answer is among the top five search results. If the correct answer appears in the top five search results, then the Rank-5 of that search result is 1; otherwise, it is 0.

[0066] mAP (mean Average Precision): This is a metric for evaluating classifier performance, especially in object detection tasks. It calculates the average precision across all classes. For each class, the Precision-Recall curve is first calculated, and then the area under this curve (i.e., average precision, AP) is calculated. mAP is the average AP across all classes, providing a comprehensive evaluation of the model's overall performance.

[0067] mINP (mean Intersection over Union for Person): This metric is specifically used to evaluate the performance of human detection tasks. It calculates the average ratio of the intersection to the union of all detected bounding boxes and ground truth bounding boxes. While INP (Intersection over Union) is a metric that measures the degree of overlap between two regions, mINP is the average of this metric across all detection results, and is commonly used to evaluate the accuracy of human detection.

[0068] As can be seen from the table, the network proposed in this invention outperforms previous classic methods in the field of gait recognition in practical performance. This invention proposes the first neural network that combines two different inputs: a skeleton sequence and a human body parsing sequence. It achieves an end-to-end method for multimodal inputs in gait recognition and improves gait recognition performance to a certain extent.

Claims

1. A gait recognition method based on multimodal data input, characterized in that, Includes the following steps: Step 1: Convert the human body parsing sequence dataset and the human skeleton sequence dataset into pkl files; Step 2: Construct a feature fusion module, which deeply fuses the features of the human body parsing module and the human skeleton module; Step 3: Construct a spatiotemporal feature extraction module, which has two branches. In the first branch, the module first extracts features along the time dimension using a one-dimensional convolution with a kernel size of 3, then enters the ReLU activation function, followed by a 3x3 convolution kernel, and finally obtains the time-dimensional feature vector through temporal pooling and a 1x1 convolution kernel. In the second branch, the module then extracts features along the spatial dimension using a one-dimensional convolution with a kernel size of 1, followed by the ReLU activation function, followed by a 3x3 convolution kernel, and finally obtains the spatial-dimensional feature vector through spatial pooling and a 1x1 convolution kernel. Step 2 is implemented as follows: The input to the feature fusion module FFM is the human skeleton sequence features S. ske and human body sequence features S par Then calculate S. par and S ske The correlation between them is used as the weight matrix; based on the weight matrix, the weighted human skeleton sequence features S are... ske The calculation is performed, and finally the result is compared with the original input human body parsing sequence features S. par Human skeleton sequence features S ske The features are concatenated together and then passed through a fully connected layer to enter the next module, resulting in the fused feature F. f Human skeleton sequence features S ske The output after the spatiotemporal feature extraction module and the fully connected layer is F. ske Human body sequence features S par The output after the graph convolutional module and the fully connected module is F. par ; W ek =Concat(H1,H2,...,H 11 )*T k (4) F f =FC(Concat(S par ,W ek )) (6) Among them, W ek H represents the weight matrix. k T represents the correlation coefficient matrix of the corresponding k-th body part; k This is the proportion matrix of the k-th body part; Both are predefined correlation matrices for the k-th body part, T k A matrix showing the proportions of different body parts; In step 3, the inputs to the two parallel spatiotemporal feature extraction modules are the human skeleton sequence features S. ske and fusion feature F f ; F fusion feature F f Human skeleton sequence features S ske The output F after the spatiotemporal feature extraction module and the fully connected layer ske Human body sequence features S par The output F after passing through the graph convolution module and the fully connected module par The three components are combined to form the final output.

2. The gait recognition method based on multimodal data input according to claim 1, characterized in that, Step 1 is implemented as follows: The input human skeleton sequence is set according to the skeleton sequence of the COCO dataset; The input human body parsing sequence is divided into 11 categories according to head, torso, left arm, right arm, left hand, right hand, left leg, right leg, left foot, right foot and bodysuit, corresponding to joint numbers 1-11 respectively.

3. The gait recognition method based on multimodal data input according to claim 2, characterized in that, Human skeleton sequence Used for inputting the human skeleton module, where the frame number is N, 3 represents the number of channels, and P represents the joint number; Human body sequence analysis Used for inputting the human body parsing module, where C represents the number of channels, H and W represent the height and width of the input human body parsing sequence, respectively, and n represents the frame number, with a value range of 1-N; The human skeleton sequence and the human parsing sequence are processed by the human skeleton module and the human parsing module, respectively, to obtain the human skeleton sequence feature S. ske and human body sequence features S par : S ske =SF(I ske ) (1) S par =CP(I par ) (2) In this context, SF and CP represent the human skeleton module and the human body analysis module, respectively.

4. The gait recognition method based on multimodal data input according to claim 3, characterized in that, The human skeleton module comprises three graph convolutional networks, which are then subjected to global max pooling and global average pooling to obtain S. ske The human body analysis module represents a cross-site structure, and S is obtained through the output of the human body analysis module. par ; Where β is a learnable parameter used to control the weight of the k-th body part, ⊙ represents element-wise multiplication, and Q... k It is the parsing mask for the k-th body part.

Images