Gait recognition method based on multi-modal data input

Abstract

The invention discloses a gait recognition method based on multi-modal data input. The method comprises the following steps: 1, converting a human body analysis sequence data set and a human body skeleton sequence data set into a pkl file; 2, constructing a feature fusion module which deeply fuses the features of the human body analysis module and the features of the human body skeleton module; 3, constructing a spatio-temporal feature extraction module and a first branch of the spatio-temporal feature extraction module, extracting features along a time dimension by using one-dimensional convolution with a kernel size of 3, entering a ReLu activation function, entering the ReLu activation function after passing through a convolution kernel, and obtaining a time dimension feature vector through the convolution kernel by time dimension pooling; the second branch extracts features along the spatial dimension by using one-dimensional convolution, and the subsequent part is kept consistent with the time dimension branch; according to the method, the human body key points and the fine-grained analysis sequence are fused, and the richness of information and the accuracy of positioning are considered, so that the performance of gait recognition is improved.

Description

Technical Field

[0001] The present invention relates to human posture estimation, gait recognition and other related fields. The specific goal is to use multimodal gait data as input and design a novel and effective neural network to perform more accurate gait recognition. Background Art

[0002] Gait recognition refers to the authentication or identification of identity through the posture or footprints of a person walking, and is considered to be one of the most promising methods for long-distance identity recognition. The advantages mainly include no need for contact, no need for pedestrian cooperation in the recognition process, and difficulty in hiding and disguising. Therefore, gait recognition has broad application prospects and economic value in the fields of security monitoring, human-computer interaction, medical diagnosis, and access control systems. Gait recognition faces many difficulties in practical applications, mainly manifested in that pedestrians are affected by external environment and personal factors during walking (such as different road conditions, different time, different perspectives, different clothing, different carrying objects, etc.), resulting in strong intra-class changes in the extracted gait features.

[0003] At present, the mainstream gait recognition methods can be divided into two categories: one is the model-based method, which mainly uses the human skeleton sequence as the gait representation; the other is the appearance-based method. The latest research has found that using the human fine-grained parsed sequence as the gait representation can extract richer information entropy. Although the human fine-grained parsed sequence is rich in information, the segmentation accuracy is low due to the pixel-level segmentation; while the human skeleton sequence is sparse, the positioning of the key points of the human body is relatively accurate. Both of these mainstream methods have made some progress, but they still have some shortcomings. Therefore, we intend to explore a gait recognition method that integrates human key points and fine-grained parsed sequences to take into account both the richness of information and the accuracy of positioning, thereby improving the performance of gait recognition. Summary of the invention

[0004] The present invention designs a gait recognition method combining human body analytical data and human body skeleton data. In the field of deep learning, the present invention is the first to propose a model combining human body analytical data and human body skeleton data.

[0005] Step 1: Convert the human body parsing sequence dataset and human body skeleton sequence dataset into pkl files;

[0006] Step 2: Construct a feature fusion module, which deeply fuses the features of the human body analysis module and the features of the human skeleton module;

[0007] Step 3: Construct a spatiotemporal feature extraction module. The spatiotemporal feature extraction module has two branches. In the first branch, the spatiotemporal feature extraction module first uses a one-dimensional convolution with a kernel size of 3 to extract features along the time dimension, then enters the ReLu activation function, and then enters the ReLu activation function after passing through a 3*3 convolution kernel. Through time dimension pooling, a 1*1 convolution kernel is used to obtain the time dimension feature vector. In the second branch, the spatiotemporal feature extraction module uses a one-dimensional convolution with a kernel size of 1 to extract features along the spatial dimension, then enters the ReLu activation function, and then enters the ReLu activation function after passing through a 3*3 convolution kernel. Through space dimension pooling, a 1*1 convolution kernel is used to obtain the space dimension feature vector.

[0008] Furthermore, step 1 is specifically implemented as follows:

[0009] The input human skeleton sequence is set according to the skeleton sequence of the COCO dataset;

[0010] The input human body parsing sequence is divided into 11 categories according to the head, torso, left arm, right arm, left hand, right hand, left leg, right leg, left foot, right foot and jumpsuit, corresponding to joint numbers 1-11 respectively.

[0011] Furthermore, the human skeleton sequence Used to input the human skeleton module, where the frame number is N, 3 represents the number of channels, and P represents the joint number;

[0012] Further, human body analysis sequence Used to input the human body analysis module, where C represents the number of channels, H and W represent the height and width of the input human body analysis sequence respectively, and n represents the frame number, and the value range is 1-N;

[0013] The human skeleton sequence and the human parsing sequence are respectively passed through the human skeleton module and the human parsing module to obtain the human skeleton sequence feature S ske and human parsing sequence features S par :

[0014] S ske =SF(I ske ) (1)

[0015] S par =CP(I par ) (2)

[0016] Among them, SF and CP represent the human skeleton module and the human body parsing module respectively.

[0017] Furthermore, the human skeleton module includes three graph convolutional networks, and finally obtains S through global maximum pooling and global average pooling. ske The human body analysis module represents the cross-part structure, and S is obtained through the output of the human body analysis module.par ;

[0018]

[0019] Where β is a learnable parameter used to control the weight of the kth body part, ⊙ represents element-wise multiplication, and Q k is the parsed mask of the k-th body part.

[0020] Furthermore, step 2 is specifically implemented as follows:

[0021] The input of the feature fusion module FFM is the human skeleton sequence feature S ske and human parsing sequence features S par ; Then calculate S par and S ske The correlation between them is used as a weight matrix; according to the weight matrix, the weighted human skeleton sequence feature S ske Calculate and finally compare the calculated result with the original input human body analysis sequence feature S par , human skeleton sequence features S ske The features are spliced ​​together and then enter the next module through the fully connected layer to obtain the fused feature F f ; Human skeleton sequence features S ske The output after the spatiotemporal feature extraction module and the fully connected layer is F ske , human body parsing sequence feature S par The output after the graph convolution module and the fully connected module is F par ;

[0022] W ek =Concat(H1,H2,...,H 11 )*T k (4)

[0023]

[0024] F f =FC(Concat(S par ,W ek )) (6)

[0025] Among them, W ek represents the weight matrix, H k represents the correlation coefficient matrix corresponding to the kth part of the body; T k is the proportion matrix of the kth body part; are the predefined correlation matrices of the kth body part, T k The matrix of the proportions of various parts of the body.

[0026] Furthermore, the inputs of the two parallel spatiotemporal feature extraction modules in step 3 are the human skeleton sequence features S ske And the fusion feature F f . The fusion feature F f , human skeleton sequence features S ske The output F after the spatiotemporal feature extraction module and the fully connected layer ske , human body analysis sequence features S par The output F after the graph convolution module and the fully connected module par The three are concatenated as the final output.

[0027] The present invention has the following effects:

[0028] The present invention is a gait recognition method that integrates human key points and fine-grained analytical sequences, while taking into account both the richness of information and the accuracy of positioning, thereby improving the performance of gait recognition.

[0029] The network proposed in this invention is better than the classic methods proposed in the field of gait recognition in actual effect. This invention proposes the first neural network that combines two different inputs, skeleton sequence and human body analysis sequence, to achieve an end-to-end method for multimodal input in gait recognition, and has a certain improvement in gait recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 Feature fusion module;

[0031] Figure 2 Spatiotemporal feature extraction module;

[0032] Figure 3 Overall structure; DETAILED DESCRIPTION

[0033] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0034] like Figure 1-3 As shown, a gait recognition method based on multimodal data input includes the following steps:

[0035] Step 1: Convert the human body analysis dataset and human skeleton dataset into pkl files;

[0036] Step 2: Construct a feature fusion module (FFM). This module deeply fuses the features of the human body analysis module and the features of the human skeleton module. The specific process of fusion is to associate the semantics between different human body analysis regions and skeletons. The specific structure of the feature fusion module (FFM) is shown in Figure 1 .

[0037] Step 3: Construct a spatiotemporal feature extraction module (TSEM). In real scenes, there will be occlusions and other situations, and the gait data collected from the same perspective will change due to changes in the shooting angle. It is necessary to construct a spatiotemporal feature extraction module to aggregate spatiotemporal features. The spatiotemporal feature extraction module has two branches. The spatiotemporal feature extraction module first uses a one-dimensional convolution with a kernel size of 3 to extract features along the time dimension, then enters the ReLu activation function, and then enters the ReLu activation function after passing through a 3*3 convolution kernel. Through time dimension pooling and a 1*1 convolution kernel, the time dimension feature vector is obtained; the spatiotemporal feature extraction module then uses a one-dimensional convolution with a kernel size of 1 to extract features along the spatial dimension, and the subsequent part is consistent with the time dimension branch.

[0038] The input human skeleton sequence is set according to the skeleton sequence of the COCO dataset. The input human parsed sequence is divided into 11 categories according to head, torso, left arm, right arm, left hand, right hand, left leg, right leg, left foot, right foot and jumpsuit, corresponding to joint numbers 1-11 respectively.

[0039] like Figure 3 As shown, the human skeleton sequence Input the human skeleton module of the network of the present invention, where the frame number is N, 3 represents the number of channels, and P represents the joint number. Input the human body parsing module of the network of the present invention, where C represents the number of channels, H and W represent the height and width of the input parsing sequence respectively, and n represents the frame number, and the value range is 1-N.

[0040] The human skeleton sequence and the human parsing sequence are respectively passed through the human skeleton module and the human parsing module to obtain the human skeleton sequence feature S ske and human parsing sequence features S par :

[0041] S ske =SF(I ske ) (1)

[0042] S par =CP(I par ) (2)

[0043] Among them, SF and CP represent the human skeleton module and human body parsing module respectively;

[0044] The human body analysis module corresponds to Figure 3 The content of the large dotted box in the middle and lower part plus the average pooling part; the human skeleton module contains three graph convolutional networks, and finally S is obtained after global maximum pooling and global average pooling. ske ;

[0045] The human skeleton module corresponds to Figure 3 The three graph convolutional networks and transformer encoders are connected in series and then added with global maximum pooling and global average pooling modules; the human body parsing module represents the cross-part structure, as shown in Formula 3. The output of the human body parsing module is S par .

[0046]

[0047] Where β is a learnable parameter used to control the weight of the kth body part, ⊙ represents element-wise multiplication, and Q k is the parsed mask of the k-th body part.

[0048] Furthermore, step 2 is specifically implemented as follows:

[0049] In order to fuse the human body analysis sequence and the human body skeleton sequence, this patent invented a feature fusion module (FFM). The input of FFM is the human body skeleton sequence feature S ske and human parsing sequence features S par ; Then calculate S par and S ske The correlation between them is used as the weight matrix;

[0050] According to the weight matrix, the weighted human skeleton sequence features are calculated, and finally the calculated results and the original input human body analysis sequence features S par , human skeleton sequence features S ske The features are spliced ​​together and then enter the next module through the fully connected layer to obtain the fused feature F f . Human skeleton sequence features S ske The output after the spatiotemporal feature extraction module and the fully connected layer is F ske , human body parsing sequence feature S par The output after the graph convolution module and the fully connected module is F par .

[0051] W ek =Concat(H1,H2,...,H 11 )*T k (4)

[0052]

[0053] F f =FC(Concat(S par ,W ek )) (6)

[0054] Among them, W ek represents the weight matrix, H kRepresents the correlation coefficient matrix of the corresponding specific body part k. k is the proportion matrix of the kth body part; are the predefined correlation matrices of the kth body part, T k The matrix of the proportions of various parts of the body.

[0055] Furthermore, step 3 is implemented as follows:

[0056] In order to make full use of the time dimension and space dimension features, the present invention designs a spatiotemporal feature extraction module.

[0057] The inputs of the two parallel spatiotemporal feature extraction modules are the human skeleton sequence features S ske And the fusion feature F f ; Specifically, the spatiotemporal feature extraction module first uses a one-dimensional convolution with a kernel size of 3 to extract features along the time dimension, then enters the ReLu activation function, and then enters the ReLu activation function after passing through a 3*3 convolution kernel, and obtains the time dimension feature vector through time dimension pooling and a 1*1 convolution kernel; the spatiotemporal feature extraction module then uses a one-dimensional convolution with a kernel size of 1 to extract features along the spatial dimension, and the subsequent parts are consistent with the time dimension branch.

[0058] The fusion feature F f , human skeleton sequence features S ske The output F after the spatiotemporal feature extraction module and the fully connected layer ske , human body analysis sequence features S par The output F after the graph convolution module and the fully connected module par The three are concatenated as the final output to facilitate the subsequent calculation of loss function and accuracy.

[0059] This network training uses the triplet loss function and the cross entropy loss function to evaluate the model performance during training, and the cosine / Euclidean distance is used for inference. Finally, the performance is tested on the Gait3D test dataset using four commonly used indicators in the field of gait recognition: Rank-1, Rank-5, mAP, and mINP. The results are shown in Table 1.

[0060] Example:

[0061] As shown in Table 1, the bottom row is the method proposed by the present invention. Compared with the previously proposed methods (the higher the number, the better the effect), the results show the effectiveness of the network proposed by the present invention.

[0062] Table 1

[0063]

[0064] The four indicators in the first row of Table 1, Rank-1, Rank-5, mAP, and mINP, are commonly used evaluation indicators in the field of gait recognition. Rank-1: This is a commonly used indicator in retrieval tasks, indicating the proportion of correct answers ranked first in the retrieval results. In face recognition or image retrieval, if the first retrieved item is the correct match, then the Rank-1 of the retrieval result is 1; if not, it is 0.

[0065] Rank-5: Similar to Rank-1, but here it considers whether the correct answer is in the first five search results. If the correct answer appears in the first five search results, then the Rank-5 of the search result is 1; if not, it is 0.

[0066] mAP (mean Average Precision): This is an indicator for evaluating the performance of a classifier, especially in object detection tasks. It calculates the average of the average precision of all categories. For each category, the Precision-Recall curve is first calculated, and then the area under this curve (i.e., the average precision, AP) is calculated. mAP is the average of the AP of all categories, which can give a comprehensive evaluation of the overall performance of the model.

[0067] mINP (mean Intersection over Union for Person): This metric is specifically used to evaluate the performance of human detection tasks. It calculates the average of the ratio of the intersection and union of all detection boxes and the true boxes. INP (Intersection over Union) is an indicator that measures the degree of overlap between two regions, and mINP is the average of this indicator over all detection results, which is usually used to evaluate the accuracy of human detection.

[0068] As can be seen from the table, the network proposed in this invention is better than the classic methods proposed in the field of gait recognition in actual effect. This invention proposes the first neural network that combines two different inputs, skeleton sequence and human body analysis sequence, to achieve an end-to-end method for multimodal input in gait recognition, and has a certain improvement in gait recognition effect.

Claims

1. A gait recognition method based on multimodal data input, characterized in that: The following steps are involved: Step 1: Convert the human body parsing sequence dataset and human body skeleton sequence dataset into pkl files; Step 2: Construct a feature fusion module, which deeply fuses the features of the human body analysis module and the features of the human skeleton module; Step 3: Construct a spatiotemporal feature extraction module. The spatiotemporal feature extraction module has two branches. In the first branch, the spatiotemporal feature extraction module first uses a one-dimensional convolution with a kernel size of 3 to extract features along the time dimension, then enters the ReLu activation function, and then enters the ReLu activation function after passing through a 3*3 convolution kernel. Through time dimension pooling, a 1*1 convolution kernel is used to obtain the time dimension feature vector. In the second branch, the spatiotemporal feature extraction module uses a one-dimensional convolution with a kernel size of 1 to extract features along the spatial dimension, then enters the ReLu activation function, and then enters the ReLu activation function after passing through a 3*3 convolution kernel. Through space dimension pooling, a 1*1 convolution kernel is used to obtain the space dimension feature vector.

2. A gait recognition method based on multimodal data input according to claim 1, characterized in that: Step 1 is implemented as follows: The input human skeleton sequence is set according to the skeleton sequence of the COCO dataset; The input human body parsing sequence is divided into 11 categories according to the head, torso, left arm, right arm, left hand, right hand, left leg, right leg, left foot, right foot and jumpsuit, corresponding to joint numbers 1-11 respectively.

3. The gait recognition method based on multimodal data input according to claim 2, characterized in that: Human skeleton sequence Used to input the human skeleton module, where the frame number is N, 3 represents the number of channels, and P represents the joint number; Human body analysis sequence Used to input the human body analysis module, where C represents the number of channels, H and W represent the height and width of the input human body analysis sequence respectively, and n represents the frame number, and the value range is 1-N; The human skeleton sequence and the human parsing sequence are respectively passed through the human skeleton module and the human parsing module to obtain the human skeleton sequence feature S ske and human parsing sequence features S par : S ske =SF(I ske ) (1) S par =CP(I par ) (2) Among them, SF and CP represent the human skeleton module and the human body parsing module respectively.

4. A gait recognition method based on multimodal data input according to claim 2 or 3, characterized in that: The human skeleton module contains three graph convolutional networks, and finally obtains S through global maximum pooling and global average pooling. ske The human body analysis module represents the cross-part structure, and S is obtained through the output of the human body analysis module. par ; Where β is a learnable parameter used to control the weight of the kth body part, ⊙ represents element-wise multiplication, and Q k is the parsed mask of the k-th body part.

5. A gait recognition method based on multimodal data input according to claim 1, 2 or 3, characterized in that: Step 2 is implemented as follows: The input of the feature fusion module FFM is the human skeleton sequence feature S ske and human parsing sequence features S par ; Then calculate S par and S ske The correlation between them is used as a weight matrix; according to the weight matrix, the weighted human skeleton sequence feature S ske Calculate and finally compare the calculated result with the original input human body analysis sequence feature S par , human skeleton sequence features S ske The features are spliced ​​together and then enter the next module through the fully connected layer to obtain the fused feature F f ; Human skeleton sequence features S ske The output after the spatiotemporal feature extraction module and the fully connected layer is F ske , human body parsing sequence feature S par The output after the graph convolution module and the fully connected module is F par ; W ek =Concat(H1,H2,...,H 11 )*T k (4) F f =FC(Concat(S par ,W ek )) (6) Among them, W ek represents the weight matrix, H k represents the correlation coefficient matrix corresponding to the kth part of the body; T k is the proportion matrix of the kth body part; are the predefined correlation matrices of the kth body part, T k The matrix of the proportions of various parts of the body.

6. The gait recognition method based on multimodal data input according to claim 5, characterized in that: The inputs of the two parallel spatiotemporal feature extraction modules in step 3 are the human skeleton sequence features S ske And the fusion feature F f .

7. The gait recognition method based on multimodal data input according to claim 6, characterized in that: The fusion feature F f , human skeleton sequence features S ske The output F after the spatiotemporal feature extraction module and the fully connected layer ske , human body analysis sequence features S par The output F after the graph convolution module and the fully connected module par The three are concatenated as the final output.

Images