A multimodal gait recognition method based on posture induction

Abstract

This invention belongs to the field of deep learning, specifically relating to a posture-induced multimodal gait recognition method aimed at improving the accuracy of gait recognition in complex environments. The method includes constructing a multimodal dataset containing gait video sequences, corresponding depth map sequences, and motion-captured posture sequences by combining a full-body inertial motion capture device, a binocular depth camera, and an RGB camera. Cross-device gait feature alignment preprocessing is performed on the multimodal dataset to obtain spatiotemporally aligned motion-captured postures and video postures. A 3D posture induction module is constructed, using the spatiotemporally aligned motion-captured postures as a reference, and trained under supervised training via graph feature alignment loss to extract induced posture information from the video postures. A multimodal recognition model fusing induced posture information and gait contour features is constructed to achieve gait recognition.

Description

Technical Field

[0001] This invention relates to the field of deep learning, and more particularly to a method for multimodal gait recognition based on posture-induced gait. Background Technology

[0002] Gait recognition, as an important branch of biometric technology, has shown great application potential in fields such as security monitoring and intelligent transportation due to its characteristics of long-distance non-contact recognition and no need for active cooperation from the target. However, complex environmental interference and high computing resource requirements remain the main obstacles to its large-scale implementation.

[0003] Existing gait recognition methods are mainly divided into two categories: appearance-based methods, which take contours as input and provide rich appearance information, are sensitive to changes in clothing and viewpoint, and struggle to capture internal structural features of the human body; and model-based methods, which take skeletons as input and possess stable motion structure information, making them less susceptible to external interference, but lack key appearance details. Single-modal methods are not robust enough in complex scenarios such as viewpoint changes and clothing occlusion. Multimodal fusion has become an important direction for improving performance, but existing multimodal methods often fail to effectively utilize real 3D pose information, resulting in insufficient feature complementarity mining, and there is still room for improvement in recognition accuracy in complex environments. Summary of the Invention

[0004] The purpose of this invention is to provide a posture-induced multimodal gait recognition method, which aims to improve the accuracy of gait recognition in complex environments.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] This invention provides a multimodal gait recognition method based on posture induction, comprising: S1: constructing a multimodal dataset containing gait video sequences, corresponding depth map sequences, and motion-captured posture sequences by combining a full-body inertial motion capture device, a binocular depth camera, and an RGB camera; S2: performing cross-device gait feature alignment preprocessing on the multimodal dataset to obtain spatiotemporally aligned motion-captured postures and video postures; S3: constructing a three-dimensional posture induction module, using the spatiotemporally aligned motion-captured postures as a reference, and extracting induced posture information from the video postures through supervised training with graph feature alignment loss; S4: constructing a multimodal recognition model that integrates induced posture information and gait contour features to achieve gait recognition.

[0007] In step S1, RGB gait videos of the subject are acquired through an RGB camera at multiple viewpoints from 0° to 180°, depth map sequences are acquired through a binocular depth camera at preset viewpoints, and motion capture posture sequences are acquired simultaneously through a full-body inertial motion capture device; the gait videos include gait data under at least three conditions: normal walking, walking with a backpack, and walking while wearing a coat.

[0008] The multimodal dataset contains gait data from multiple subjects. Each subject's gait data corresponds to video sequences from multiple viewpoints, and each video sequence is associated with a depth map sequence and a motion capture pose sequence. The dataset is divided into a training set and a test set.

[0009] Cross-device gait feature alignment includes temporal alignment and spatial alignment. Temporal alignment is achieved by using the maximum stride distance at the initial walking stage of the gait, while spatial alignment is achieved by adapting the coordinate system of the motion capture pose to the video pose through frame-by-frame translation and sequence-level rotation.

[0010] The time alignment specifically includes: calculating the three-dimensional Euclidean distance time sequence of the two feet after the start of walking in the motion capture posture data collected by the whole-body inertial motion capture device and the posture data corresponding to the RGB gait video collected by the RGB camera, respectively, and using the local minimum value immediately following the first local maximum value in the time sequence as the time alignment benchmark to achieve a one-to-one correspondence between the two sets of device data.

[0011] Spatial alignment specifically includes: using the pelvic joint as the origin of the global coordinate system, aligning the motion capture pelvic joint with the video pelvic joint through frame-by-frame translation, and then aligning the motion capture coordinate system with the video coordinate system through sequence-level rotation.

[0012] In step S3, the same graph convolution operation is performed on the spatiotemporally aligned motion capture pose and video pose respectively, keeping the graph convolution features of the motion capture pose unchanged. The video pose branch is trained in reverse through graph feature alignment loss to extract the semantic information of the induced pose.

[0013] The graph feature alignment loss includes feature alignment loss, limb node weighted loss, and temporal smoothing loss. The feature alignment loss measures the difference between the graph convolutional features of motion capture pose and video pose using mean square error.

[0014] The multimodal recognition model includes a dual-branch structure; the induced posture branch is used to extract dynamic features from the induced posture information, and the contour branch is used to capture the appearance and motion features of the gait contour. The features of the two branches are fused through a multi-stage feature fusion strategy to achieve gait recognition.

[0015] The multi-stage feature fusion strategy includes frame-level fusion, spatiotemporal fusion, and global fusion. The model is trained using a triplet loss function to reduce the feature distance between samples with the same identity and increase the feature distance between samples with different identities.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0017] 1. The multimodal gait recognition method based on posture induction provided in this application provides richer raw information support for gait recognition by constructing a multimodal dataset containing RGB gait videos, depth maps, and motion-captured postures. In particular, it supplements real three-dimensional posture data, laying the foundation for subsequent feature fusion. The cross-device gait feature alignment preprocessing solves the spatiotemporal heterogeneity problem of data from different devices through temporal and spatial alignment, ensuring accurate matching between motion-captured postures and video postures, and improving data consistency.

[0018] 2. The 3D pose induction module uses real motion-captured poses as a reference and is trained using graph feature alignment loss to extract more realistic induced pose information from video poses, overcoming the limitations of single video pose features. The multimodal recognition model adopts a dual-branch structure to fuse induced pose and contour features, combining a multi-stage fusion strategy and triplet loss to fully leverage the complementarity of different modal features. This effectively improves the model's fine-grained feature extraction capability and recognition robustness under complex conditions such as changing viewpoints and clothing occlusion, ultimately achieving higher-precision gait recognition. Attached Figure Description

[0019] Figure 1 This is a flowchart of a posture-induced multimodal gait recognition method provided in an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of a data acquisition site provided in an embodiment of this application;

[0021] Figure 3 This is a flowchart of a CASIA_V dataset preprocessing method provided in an embodiment of this application;

[0022] Figure 4 This is a structural diagram of a three-dimensional pose guidance module (3D-SIM) in a SIGait model provided in an embodiment of this application;

[0023] Figure 5 This is an overall architecture diagram of a SIGAI model provided in an embodiment of this application;

[0024] Figure 6 This application provides an adaptive feature fusion module (AFFM) in a SIGAit model.

[0025] Figure 7 This application provides a multi-scale spatiotemporal feature extractor module (MSSTFE) in a SIGAit model.

[0026] Figure 8 This is a performance evaluation experiment result diagram provided in an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0028] To verify the assistance of real 3D pose in gait recognition, this application constructs the CASIA_V dataset. Compared to the CASIA_B dataset, the CASIA_V dataset includes motion-captured pose data and a depth map from a frontal view. Preprocessing the original data yields motion-captured pose data time-aligned with the video data, thus forming the CASIA_V dataset. The video data from the CASIA_V dataset is then used for VideoPose3D prediction to obtain video poses. Before using the video pose and motion-captured pose as input to the 3D-SIM module, the motion-captured pose space needs to be aligned with the video pose. The 3D-SIM module is then trained to extract induced pose semantic information from the video pose.

[0029] By combining a 3D-SIM module, a multimodal gait recognition model SIGait is constructed, which integrates pose guidance information and contour information. The induced pose is extracted from the video using the 3D-SIM module, and its semantic information, along with contour information, serves as the input to the SIGait model. A multi-stage feature fusion strategy is employed, combined with triplet loss, to construct the gait recognition model SIGait.

[0030] For example, such as Figure 1 As shown, this application provides a method for multimodal gait recognition based on posture-induced gait, including:

[0031] S1: Combining a full-body inertial motion capture device, a binocular depth camera, and an RGB camera, construct a multimodal dataset containing gait video sequences, corresponding depth map sequences, and motion capture posture sequences.

[0032] For example, in order to explore the assistance provided by real three-dimensional posture in multi-view gait recognition, this application embodiment constructs the CASIA_V dataset by collecting multi-dimensional gait sequences of the same subject from different perspectives, in combination with hardware devices such as the VDSuit whole-body inertial motion capture device, the D455 binocular depth camera, and the Hikvision camera. That is, it constructs a multimodal dataset containing gait video sequences, corresponding depth map sequences, and motion capture posture sequences.

[0033] The CASIA_V dataset was constructed to explore the assistance that real 3D pose can provide in multi-view gait recognition, by acquiring multiple gait sequences of the same subject from different viewpoints. Its construction method can be seen as adding motion-captured pose data synchronized with video data to the CASIA_B dataset.

[0034] As one possible approach, RGB gait videos of the subject are captured by an RGB camera from multiple perspectives ranging from 0° to 180°. Depth map sequences are captured by a binocular depth camera at preset perspectives, and motion capture posture sequences are captured simultaneously by a full-body inertial motion capture device. The gait videos include gait data under at least three conditions: normal walking, walking with a backpack, and walking while wearing a coat.

[0035] For example, refer to Figure 2 , Figure 2 The data collection setup is shown. Each subject wore a VDSuit and recorded RGB gait using a Hikvision camera from 11 viewpoints between 0° and 180°. Depth map gait was recorded using a binocular depth camera from the 0° viewpoint. Each viewpoint included three conditions: normal walking, walking with a backpack, and walking while wearing a coat. During data collection, subjects were required to walk naturally in a semi-indoor environment to ensure data diversity and authenticity. The dataset contains gait data from 103 subjects, comprising 103×11×11 video sequences, each with a corresponding depth map sequence and motion capture posture sequence. The training set includes 70 subjects, and the test set includes 33 subjects.

[0036] S2: Perform cross-device gait feature alignment preprocessing on the multimodal dataset to obtain spatiotemporally aligned motion capture poses and video poses.

[0037] For example, such as Figure 3 As shown, the multimodal dataset contains gait data from multiple subjects. Each subject's gait data corresponds to video sequences from multiple viewpoints, and each video sequence is associated with a depth map sequence and a motion capture pose sequence.

[0038] Cross-device gait feature alignment includes temporal alignment and spatial alignment. Temporal alignment is achieved by using the maximum stride distance at the initial walking stage of the gait, while spatial alignment is achieved by adapting the coordinate system of the motion capture pose to the video pose through frame-by-frame translation and sequence-level rotation.

[0039] Because the motion capture device VDsuit lacks the function of capturing absolute timestamps, the acquired motion capture data cannot be aligned with the timestamps of the RGB images captured by Hikvision cameras. Time alignment is required to achieve a one-to-one correspondence between the data collected by the devices.

[0040] The time alignment specifically includes: calculating the three-dimensional Euclidean distance time sequence of the two feet after the start of walking in the motion capture posture data collected by the whole-body inertial motion capture device and the posture data corresponding to the RGB gait video collected by the RGB camera, respectively, and using the local minimum value immediately following the first local maximum value in the time sequence as the time alignment benchmark to achieve a one-to-one correspondence between the two sets of device data.

[0041] For example, in the preprocessing of the CASIA_V dataset, gait feature alignment is used for time alignment, specifically by using the maximum and minimum stride distances at the start of walking: First, the three-dimensional Euclidean distance time series of the two feet after the start of walking is calculated for both sets of device data; then, the local minimum (minimum stride distance) immediately following the first local maximum (maximum stride distance) in the time series is used as the time alignment standard to complete synchronization. Results show that this method has a significant time alignment effect. The mathematical expression of this method is:

[0042]

[0043] in Indicates the time alignment reference. , These represent the temporal information of the local minimum (minimum step size) immediately following the first local maximum (maximum step size) in the temporal sequence corresponding to the motion capture pose and the video pose, respectively.

[0044] Spatial alignment specifically includes: using the pelvic joint as the origin of the global coordinate system, aligning the motion capture pelvic joint with the video pelvic joint through frame-by-frame translation, and then aligning the motion capture coordinate system with the video coordinate system through sequence-level rotation.

[0045] In 3D pose data processing, establishing a coordinate system is a crucial step in ensuring consistent spatial reference for pose data from different sources (motion-captured poses and video poses predicted by VideoPose3D). For pose sequences, frame-by-frame translation alignment is used to align the motion-captured pelvis with the video pelvis. Combined with global rotation of the sequence, this addresses the overall spatial differences that single-frame alignment cannot cover, ensuring spatial consistency throughout the entire motion sequence and ultimately adapting the coordinate system of the motion-captured pose to the video pose. The following is the specific implementation method and mathematical derivation for aligning the motion-captured pose with the video pose:

[0046] First, determine the global origin point, the pelvic joint. The human pelvic joint is selected as the origin of the global coordinate system because this joint is relatively stable during movement and is the central hub of limb movement.

[0047] For motion capture posture data, the coordinates of the pelvic joint in each frame are ( ).

[0048] For VideoPose3D data, the 2D coordinates of the pelvic joints need to be located first using HRNet 2D detection. , , Then, its 3D coordinates are estimated using a weak perspective projection model:

[0049]

[0050] in, The average width of the human body For camera focal length, Based on the actual height. The final predicted 3D coordinates of the pelvis are obtained. , , ).

[0051] Where t=1,2,…,T (T is the number of frames in the sequence).

[0052] Then, the motion capture joint coordinates of each frame in the sequence are translated to align the motion capture pelvic joint coordinates with the video pelvic joint coordinates.

[0053]

[0054] Results: The pelvic joints of the motion capture pose sequence were aligned to the video pelvic joint positions in each frame.

[0055] Then, the sequence-level rotation matrix R is calculated using Procrustes analysis to rotate the motion capture coordinate system to the video coordinate system orientation:

[0056] First, mean removal processing is performed. For both the translated motion capture pose sequence and the original video pose sequence, the mean of all joints is calculated to eliminate the residual effects of translation.

[0057] ,

[0058] Where J represents the number of joints. This represents the coordinates of the j-th joint in the t-th frame of the motion capture sequence after translation.

[0059] The pose sequence after removing the mean:

[0060] ,

[0061] Then, the covariance matrix H between the motion capture and the video sequence is constructed:

[0062]

[0063] Perform singular value decomposition (SVD) on H:

[0064] The rotation matrix R is obtained by the following formula (ensuring orthogonality):

[0065] If the determinant R is negative, there is a mirror flip, so after correcting the sign of the last column of V, R is recalculated.

[0066] Finally, a rotation matrix R is applied to the mean-removed motion capture pose sequence:

[0067]

[0068] Then, by restoring the mean (mean of the video pose), we obtain the rotated and aligned motion capture pose sequence:

[0069]

[0070] Through the above operations, the overall coordinate axis direction of the motion capture posture sequence is completely consistent with the video posture sequence.

[0071] S3: Construct a 3D pose induction module. Using the spatiotemporally aligned motion capture pose as a reference, extract induced pose information from the video pose through supervised training with graph feature alignment loss.

[0072] In step S3, the same graph convolution operation is performed on the spatiotemporally aligned motion capture pose and video pose respectively, keeping the graph convolution features of the motion capture pose unchanged. The video pose branch is trained in reverse through graph feature alignment loss to extract the semantic information of the induced pose.

[0073] For example, the 3D-SIM module is trained under the supervision of graph feature alignment loss. The same graph convolution operation is performed on both the preprocessed motion-captured pose and the video pose, and the training is supervised by graph feature alignment loss. During training, the motion-captured features are kept unchanged, and the video pose branch is trained in reverse using the motion-captured pose as a reference to obtain induced pose semantic information from the video pose. The significance of the 3D-SIM module is that during the testing phase, the induced pose features can be directly extracted from the video pose using the pre-trained graph convolution module, thereby obtaining more effective 3D pose features.

[0074] The graph feature alignment loss includes feature alignment loss, limb node weighted loss, and temporal smoothing loss. The feature alignment loss measures the difference between the graph convolutional features of motion capture pose and video pose using mean square error.

[0075] For example, the detailed structure of a 3D-SIM module is as follows: Figure 4 As shown. Figure 4The two input branches are the motion-captured pose aligned with the video pose space and the video pose obtained by VideoPose3D processing, respectively. Spatial alignment is achieved by aligning the motion-captured pelvis to the video pelvis through frame-by-frame translation, combined with global rotation of the sequence to ensure spatial consistency throughout the motion sequence, ultimately adapting the motion-captured pose to the coordinate system of the video pose. Both branches perform the same multiple graph convolution operation to extract graph convolution features. The graph convolution features of the motion-captured pose remain unchanged, and the graph convolution module corresponding to the video pose is back-trained using the graph convolution feature loss function to obtain the semantic information of the induced pose.

[0076] S4: Construct a multimodal recognition model that integrates induced posture information and gait contour features to achieve gait recognition.

[0077] The multimodal recognition model includes a dual-branch structure; the induced posture branch is used to extract dynamic features from the induced posture information, and the contour branch is used to capture the appearance and motion features of the gait contour. The features of the two branches are fused through a multi-stage feature fusion strategy to achieve gait recognition.

[0078] For example, refer to Figure 5 , Figure 5 The detailed structure of the SIGait model is shown. The method for constructing the SIGait model using the 3D-SIM module is as follows: induced pose is extracted from the video pose using the 3D-SIM module, and semantic information of the induced pose is obtained. This semantic information, along with contour information, serves as the input to the SIGait model. Therefore, the model constructs two branches: the induced pose branch focuses on extracting dynamic human pose information, and the contour branch is responsible for capturing the appearance and motion features in the contour sequence. Through this dual-branch structure, the model can fully utilize the complementary information provided by each branch, effectively improving the model's fine-grained pose feature extraction and multi-view sensitivity, and enhancing recognition robustness.

[0079] The multi-stage feature fusion strategy includes frame-level fusion, spatiotemporal fusion, and global fusion. The model is trained using a triplet loss function to reduce the feature distance between samples with the same identity and increase the feature distance between samples with different identities. Through hierarchical fusion at the frame, spatiotemporal, and global levels, the multi-stage feature fusion strategy can fully exploit the complementarity between induced pose information and gait contour features in different dimensions: frame-level fusion can capture the correlation of local detailed features early on; spatiotemporal fusion can effectively extract the dynamic correlation between time series and spatial structure; and global fusion achieves efficient aggregation of overall features. The combination of these three approaches significantly improves the model's sensitivity to fine-grained pose features and multi-viewpoint differences, enhancing the completeness of feature representation in complex scenes.

[0080] The SIGAit model provided in this application is based on a multi-stage adaptive feature fusion strategy. For the input silhouette sequence and pose sequence (replacing the skeleton sequence of the original model), it achieves efficient and robust gait recognition through three-level fusion: frame-level, spatiotemporal-level, and global-level, combined with multi-scale spatiotemporal feature extraction and feature dimension pooling. The complete network structure and process are detailed below:

[0081] SIGait's input includes two modal sequences: silhouette sequences and pose sequences. Silhouette sequence X img ∈R N×H×W Where N is the number of frames, H and W are the image height and width, respectively, and each frame is a binary human contour image; pose sequence X pose ∈R N ×3×Z Where N is the number of frames, 3 is the number of channels, and Z is the number of keypoints. Feature extraction networks are designed to capture spatial features for the two modalities, taking into account their structural characteristics. For silhouette sequences, four 2D convolutional layers plus one 2D max-pooling layer are used to obtain the silhouette frame-level features S. img ∈R N×C×H / 2×W / 2 The convolutional layers have a kernel size of 3×3 and a stride of 1, with output channels of 32, 64, 128, and 128 respectively. Each layer is followed by a ReLU activation function to extract local shape and spatial structure features of the silhouette. Pooling layers are placed after the second convolutional layer, with a kernel size of 2×2 and a stride of 2, downsampling the feature maps and outputting features of N×C×H / 2×W / 2 dimensions. For pose sequences, pose frame-level features S are obtained using the 3D-SIM module. pose ∈R N×C×Z .

[0082] For example, such as Figure 6 As shown, to capture the spatiotemporal correlation of gait at different spatial scales, a multi-scale spatiotemporal feature extractor (MSSTFE) is designed to process silhouette, pose, and fused features respectively. MSSTFE extracts spatiotemporal correlation features at three spatial scales: partial, local, and global. For partial and local feature extraction, a stacked structure of spatial 1D convolution and temporal 1D convolution is used. The kernel sizes of the spatial convolutions are 1 and 3, respectively, to extract local or adjacent features in the spatial dimension; the temporal convolution kernel size is 3 to capture continuous frame dependencies in the temporal dimension. The final output is the partial-level feature ST. p and local-level features ST l For global feature extraction, channel dimensions are compressed using GAP and GMP, and global spatial attention weights are generated using a self-attention mechanism to highlight key parts with large motion amplitudes. Temporal 1D convolution is applied to capture global temporal correlations, outputting global feature ST. g Then feature aggregation is performed, and ST is concatenated. p STl and ST g By aggregating the time dimension through max pooling, multi-scale spatiotemporal features ST are output. m ∈R C×3M .

[0083] Reference Figure 7 The multi-stage adaptive feature fusion module (AFFM) fully leverages the complementary information between silhouettes and poses through frame-level and spatiotemporal fusion strategies. For different input branches, AFFM employs multi-head attention to calculate the association weights of the input branches, highlighting strongly correlated parts, and combines the bias term generated by global pooling to compensate for features in weakly correlated regions.

[0084] The SIGait model employs an All (BA+) triplet loss, with each batch containing p subjects and each subject sampling k sequences. The initial learning rate is 1e-4, decaying to 0.1 times at 30K and 60K iterations; the number of iterations is 100K. During model training, the globally fused feature Out is used to calculate the loss. During testing, low-dimensional features OutFDP are output through FD Pooling, and Euclidean distance is used to calculate sample similarity to achieve gait recognition.

[0085] Model training is divided into two stages: the 3D-SIM module training stage and the SIGait model training stage.

[0086] As one possible implementation, the 3D-SIM module uses graph convolutional feature loss as the training function, and iteratively updates the model's weight parameters through backpropagation. The graph convolutional feature loss function is defined as follows:

[0087] The graph convolution loss function consists of three parts: feature alignment loss, limb node weighted loss, and temporal smoothing loss.

[0088]

[0089] in This represents the weighting coefficient for limb nodes. This represents the time-series smoothing coefficient.

[0090] Feature alignment loss

[0091] Mean squared error (MSE) is used to align the graph convolutional features of the true pose and the predicted pose, ensuring that they are close in the feature space:

[0092]

[0093] Where T represents the number of frames in the video sequence; This indicates the number of nodes in the limb being monitored; Indicates the feature dimension of the graph convolution; , The graph node features represent the input of GCN, corresponding to the input of motion capture pose graph convolutional features and video pose graph convolutional features, respectively, with dimensions [batch_size, num_joints, feature_dim]; A represents the adjacency matrix, used to define joint connection relationships; is the Frobenius norm, which measures element-wise differences in a matrix.

[0094] Limb node weighted loss

[0095] Give higher weight to the feature errors of limb nodes (denoted as ). Strengthen the optimization of key nodes:

[0096]

[0097] Where T represents the number of frames in the video sequence; Indicates the number of limb nodes of interest; Indicates the feature dimension of the graph convolution; Indicates joint Weighting coefficients; and These represent the joints corresponding to the motion capture pose and the video pose, respectively. The feature vector has a dimension of . ; is the Frobenius norm, which measures element-wise differences in a matrix.

[0098] Limb node weighted loss

[0099]

[0100] Where T represents the number of frames in the video sequence; Indicates the number of limb nodes of interest; Indicates the feature dimension of the graph convolution; and These represent the joints corresponding to the motion capture pose and the video pose, respectively. The feature vector has a dimension of . ; is the Frobenius norm, which measures element-wise differences in a matrix.

[0101] As one possible implementation, the SIGait model is trained using a triplet loss function. During training, the output `Out` after global fusion is input into the `All(BA+) tripletloss`. By minimizing the feature distance between samples with the same identity and maximizing the feature distance between samples with different identities, the model's ability to learn the uniqueness of individual gait is enhanced. This makes samples with the same identity closer together in the feature space, and samples with different identities farther apart, thereby improving the model's recognition ability. This results in extracted gait features with better discriminative power in the metric space, especially under complex conditions with changes in clothing (such as wearing a coat or backpack) and significant differences in perspective. It effectively reduces interference from confused samples, significantly improving the accuracy and robustness of gait recognition.

[0102] Specifically, triplet loss :

[0103]

[0104] in, This represents the total number of triples contained in a batch; , and represents the gait characteristics of the anchor sample, the positive sample with the same identity as the anchor sample, and the negative sample with a different identity from the anchor sample in the i-th triplet of this batch, respectively; margin represents the allowance.

[0105] During the training phase, the contour and pose sequences of the training set are input into the SIGait model to extract features at each stage (such as frame-level, spatiotemporal-level, and global-level features). The final gait feature vector is obtained through global-level fusion. The feature vector is then input into All(BA+) tripletloss, and the model parameters are optimized by minimizing the loss, so that features of samples with the same identity are closer together in the metric space, while features of samples with different identities are farther apart.

[0106] During the testing phase, the test set is divided into a registration set (Gallery) and a validation set (Problem). The contour, skeleton, and pose sequences from the registration and validation sets are input into the trained model to extract global gait features. The Euclidean distance between the features of the registration and validation sets is calculated, and the features are sorted in ascending order of distance. The Rank1 recognition accuracy (i.e., the proportion of correct matches in the validation set that rank first in the sorted results) is then calculated.

[0107] This invention adds induced pose feature input to the SIGait model through a 3D-SIM module, fully integrating more realistic motion-captured pose data to improve the extraction of fine-grained features and multi-view modal sensitivity. Ablation experiments on the SIGait model with the 3D-SIM module added show that using the 3D pose obtained directly from VideoPose3D as input for multi-modal fusion does not improve the Rank-1 metric.

[0108] To evaluate the pose-induced multimodal gait recognition method proposed in this invention, a systematic comparison was conducted on the CASIA_V dataset with five contour-based methods, two skeleton-based methods, and three multimodal methods based on both contour and skeleton models. Rank-1 accuracy was used to evaluate the model's performance. Experimental results are as follows: Figure 8 As shown. According to Figure 8 The following conclusions can be drawn: 1) SIGAI achieved better results under all conditions, indicating that the gait features extracted by SIGAI are more robust. 2) Under difficult conditions, such as CL, SIGAI's performance is 20.98% better than GaitSet, 10.58% better than GaitPart, 10.58% better than GaitGL, 7.68% better than CSTL, 5.48% better than the multimodal method TransGait, and 7.08% better than the multimodal method MSAFF. This shows that SIGAI is more effective in complex environments. 3) As Figure 2 As shown, SIGait achieved better results in multi-view gait sequences, demonstrating the improvement in fine-grained feature extraction and multi-view modality sensitivity of the model.

[0109] In the description of this specification, specific features, structures, materials, or characteristics may be combined in any suitable manner in one or more embodiments or examples.

[0110] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for multimodal gait recognition based on posture-induced gait, characterized in that, include: S1: Construct a multimodal dataset containing gait video sequences, corresponding depth map sequences, and motion capture posture sequences by combining a full-body inertial motion capture device, a binocular depth camera, and an RGB camera; S2: Perform cross-device gait feature alignment preprocessing on the multimodal dataset to obtain spatiotemporally aligned motion capture postures and video postures; wherein, the cross-device gait feature alignment includes temporal alignment and spatial alignment. Temporal alignment is performed by using the maximum stride distance at the initial walking position, and spatial alignment is performed by adapting the coordinate systems of the motion capture posture and video posture through frame-by-frame translation and sequence-level rotation; the temporal alignment specifically includes: calculating the three-dimensional Euclidean distance time series of the two feet after the start of walking in the motion capture posture data collected by the full-body inertial motion capture device and the posture data corresponding to the RGB gait video collected by the RGB camera, respectively, and using the local minimum value immediately following the first local maximum value in the time series as the time alignment benchmark to realize the alignment of the two sets of device data. The one-to-one correspondence; the spatial alignment specifically includes: taking the pelvic joint as the origin of the global coordinate system, aligning the motion-captured pelvic joint with the video pelvic joint through frame-by-frame translation, and then aligning the motion-captured coordinate system with the video coordinate system through sequence-level rotation; S3: constructing a three-dimensional pose induction module, taking the spatiotemporally aligned motion-captured pose as a reference, and extracting induced pose information from the video pose through supervised training with graph feature alignment loss; S4: constructing a multimodal recognition model that integrates the induced pose information and gait contour features to achieve gait recognition; in step S3, taking the spatiotemporally aligned motion-captured pose as a reference, extracting induced pose information from the video pose through supervised training with graph feature alignment loss specifically includes performing the same graph convolution operation on the spatiotemporally aligned motion-captured pose and the video pose respectively, keeping the graph convolution features of the motion-captured pose unchanged, and training the video pose branch in reverse through graph feature alignment loss to extract induced pose semantic information.

2. The method for multimodal gait recognition based on posture induction according to claim 1, characterized in that: In step S1, RGB gait videos of the subject are captured by an RGB camera at multiple viewpoints from 0° to 180°, and depth map sequences are captured by a binocular depth camera at preset viewpoints. At the same time, motion capture posture sequences are captured synchronously by a whole-body inertial motion capture device. The gait videos include gait data under at least three conditions: normal walking, walking with a backpack, and walking while wearing a coat.

3. The method for multimodal gait recognition based on posture induction according to claim 2, characterized in that: The multimodal dataset contains gait data of multiple subjects. Each subject's gait data corresponds to video sequences from multiple viewpoints, and each video sequence is associated with a depth map sequence and a motion capture pose sequence. The dataset is divided into a training set and a test set.

4. The method for multimodal gait recognition based on posture induction according to claim 1, characterized in that, The graph feature alignment loss includes feature alignment loss, limb node weighted loss, and temporal smoothing loss. The feature alignment loss measures the difference in graph convolutional features between motion capture pose and video pose using mean square error.

5. The method for multimodal gait recognition based on posture induction according to claim 1, characterized in that, The multimodal recognition model includes a dual-branch structure; the induced posture branch is used to extract dynamic features from the induced posture information, and the contour branch is used to capture the appearance and motion features of the gait contour. The features of the two branches are fused through a multi-stage feature fusion strategy to achieve gait recognition.

6. The method for multimodal gait recognition based on posture induction according to claim 5, characterized in that, The multi-stage feature fusion strategy includes frame-level fusion, spatiotemporal fusion, and global fusion. The model is trained using a triplet loss function to reduce the feature distance between samples with the same identity and increase the feature distance between samples with different identities.

Images