Gait recognition method based on contour and skeleton mixed attention feature fusion
By combining 2D and 3D skeleton data with a dual-branch parallel architecture and a hybrid attention fusion module, this method addresses the lack of robustness of existing gait recognition methods to changes in clothing and carried items in complex scenes, achieving efficient and robust gait recognition.
Patent Information
- Application Number
- CN202511371258.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-24
- Publication Date
- 2025-12-26
AI Technical Summary
Existing gait recognition methods are sensitive to factors such as clothing changes, carried items, and occlusion in complex scenarios. Furthermore, the skeleton data is sparse and computationally complex, making it difficult to achieve efficient and robust identity recognition.
A dual-branch parallel architecture is adopted, combining 2D and 3D keypoint skeleton data. Through an efficient channel attention mechanism and a hybrid attention fusion module, skeleton features are extracted, and contour features are processed by convolution and spatial transformer. Multiple loss functions are designed for feature fusion.
It improves the accuracy of gait recognition under complex conditions, realizes long-distance, non-contact identity recognition, is suitable for resource-constrained devices, and has good generalization ability and robustness.
Smart Images

Figure CN121214547A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a gait recognition method based on contour and skeleton hybrid attention feature fusion, belonging to the fields of biometric recognition, computer vision and pattern recognition technology. Background Technology
[0002] Gait recognition, as an important biometric identification technology, can identify individuals by analyzing their walking patterns, offering unique advantages such as long-distance, non-contact, and non-cooperative operation. Unlike other biometric identification technologies that require the identified individual to be close to the data collection system (such as face, iris, and fingerprint recognition), gait recognition can identify individuals from a greater distance, thus having broad application prospects in fields such as crime prevention, forensic identification, and social security.
[0003] Currently, gait recognition methods are mainly divided into two categories: contour-based methods and skeleton-based methods. Contour-based gait recognition methods use human body contours as input and are the most widely used technical approach. These include set-based and sequence-based methods. However, these methods are highly sensitive to factors such as clothing changes, carried items, and occlusion, and are prone to losing fine-grained spatial information in complex scenes. Skeleton-based gait recognition methods use human body key points as input and are more robust to appearance changes. However, although skeleton data has viewpoint invariance, existing methods still face problems such as limited pose estimation accuracy, high computational complexity, and sparse skeleton data information. They also cannot effectively cope with the challenges of carried items and clothing changes.
[0004] The technical differences compared to existing technologies are as follows:
[0005] Technical Comparison with Patent CN115050101B "A Gait Recognition Method Based on Skeletal and Contour Feature Fusion"
[0006] 1. Patent CN115050101B proposes a gait recognition method based on the fusion of skeletal and contour features. It utilizes information such as the position, velocity, angle, and length of key points on a two-dimensional skeleton, extracting skeletal features via a graph convolutional network. Simultaneously, it employs instance segmentation and a local contour feature extractor to obtain contour features, which are then fused through intermediate layer concatenation and a local attention mechanism to ultimately achieve gait recognition. The core of this method lies in leveraging the complementarity of multimodal features to alleviate the insufficient robustness of single features under changes in clothing, viewing angle, and the interference of carried objects.
[0007] This patent features innovative designs for a dual-source skeleton (fusion of 2D and 3D keypoints) and a hybrid attention fusion mechanism. Specifically, an efficient channel attention mechanism (ECA-ResGCN) is introduced into the skeleton branch to enhance the ability to extract spatiotemporal features of the skeleton; in the contour branch, convolution and spatial transform (CST) and micro-motion capture module (MCM) are combined to enhance the ability to model cross-viewpoint and fine-grained motion; and a dual fusion module based on attention (SEFFM) and Transformer (FFSCT) is designed in the fusion layer, which can be adaptively selected according to the size of the dataset.
[0008] The two differ fundamentally in their feature extraction methods and fusion mechanisms. The former uses traditional splicing and fusion, while the latter achieves more intelligent and flexible feature integration through an adaptive hybrid attention mechanism.
[0009] 2. Patent CN115050101B mainly relies on Euclidean distance metric to calculate the similarity between gait features and samples in the database during the training and recognition stages. It does not perform special optimization on the loss function, and the overall training objective is relatively simple.
[0010] This patent designs multiple loss functions, fusing batch all triplet loss and cross-entropy loss, to enhance intra-class compactness and inter-class separability through joint optimization. Simultaneously, a hybrid pooling strategy is introduced in the contour branch to effectively suppress overfitting and improve fine-grained feature representation.
[0011] The two differ significantly in their training optimization objectives and loss function design. This patent further improves the model's generalization ability and robustness while ensuring discriminability.
[0012] Technical Comparison with Patent CN117809372A "Gait Recognition System Guided by Human Posture Information"
[0013] 1. Patent CN117809372A proposes a gait recognition system guided by human posture information. It employs a single processing flow with contour image sequences as the primary input. It acquires contour spatial features through a hybrid attention mechanism spatial feature extraction module (CBAM-SFE), models temporal features using a multi-scale temporal attention module (MS-TA), and performs simple feature concatenation through a posture and contour feature fusion module (PSFF) to ultimately achieve gait recognition. The core of this method lies in using human skeletal information to guide the learning of contour features, thereby handling the relationship between covariates and non-covariates.
[0014] This patent innovatively proposes the GaitFusion dual-branch parallel architecture, featuring a more advanced technical design. Specifically, it constructs a skeleton branch, GaitGCN, employing a graph convolutional network with an efficient channel attention mechanism (ECA-ResGCN). This enhances skeleton feature extraction capabilities through three input branches (joints, velocity, and skeleton) and a 2D+3D keypoint fusion strategy. A contour branch, GaitContour, is constructed, integrating a Convolutional Spatial Transformer (CST) and a Micro-Motion Capture (MCM) module to achieve robust modeling of cross-view images and fine-grained motion. At the feature fusion level, a dual-fusion architecture is designed, consisting of an attention-based SEFFM module and a Transformer-based FFSCT module, which can adaptively select the optimal fusion strategy based on the dataset size.
[0015] The two differ fundamentally in their overall architecture design and technical implementation. The former adopts a traditional architecture with a single contour main line and skeleton assistance, and the fusion method is simple feature splicing; the latter achieves full mining of contour and skeleton features through dual-branch parallel processing, and achieves more accurate and flexible multimodal feature integration through an intelligent adaptive fusion mechanism.
[0016] 2. Patent CN117809372A employs a standard ST-GCN network architecture for skeleton feature extraction. It uses the OpenPose algorithm to extract 18 2D human keypoints as input, employing a 6-layer ST-GCN structure. Each layer includes a Graph Convolutional Network (GCN) and a Temporal Convolutional Network (TCN) to process spatial and temporal features respectively, and a dropout mechanism is implemented after all layers to prevent overfitting. This method follows the traditional spatiotemporal graph convolution processing flow and has basic processing capabilities for spatiotemporal modeling of skeleton data, but it has certain limitations in feature representation capabilities and the application of attention mechanisms.
[0017] This patent features multi-layered technological innovations in skeleton feature extraction. First, it employs a dual-source skeleton data fusion strategy, combining 2D keypoints extracted using the OpenPose algorithm with 3D keypoints acquired via SMPL-X or Azure Kinect to form a richer skeleton representation. Second, it designs a GaitGCN network architecture, including three parallel input channels: joint branch, velocity branch, and skeleton branch, which respectively process keypoint coordinate information, inter-frame motion velocity, and human skeletal connection structure information. Most importantly, it proposes the ECA-ResGCN module, introducing an efficient channel attention mechanism (ECA) on top of the residual map convolutional network. Through one-dimensional convolution modeling of local cross-channel interactions, it minimizes the number of parameters while significantly improving recognition accuracy.
[0018] The two approaches differ significantly in their technical approaches to skeleton feature extraction. The former employs a traditional method using a single data source and a standard network structure, lacking in-depth optimization of skeleton feature representation capabilities; the latter, through multi-source data fusion, multi-branch parallel processing, and attention mechanism integration, constructs a more efficient and accurate skeleton feature extraction framework, achieving a comprehensive improvement in feature extraction capabilities while maintaining a lightweight design.
[0019] Technical Comparison with Patent CN20340134A "A Multimodal Gait Recognition Method and System"
[0020] 1. Patent CN120340134A proposes a multimodal gait recognition method. It employs the YOLO image segmentation algorithm to extract gait contour maps, simultaneously utilizing a joint prediction algorithm to obtain skeleton point positions, and converting the skeleton points into Gaussian heatmap representations using the K-Gaussian method. In the data preprocessing stage, the method centers and scales the skeleton coordinates to eliminate interference from irrelevant information such as walking trajectory and shooting distance. Then, a dual-branch shallow feature extraction module is constructed to process both the contour map and the skeleton Gaussian heatmap data separately, and feature integration is achieved through a cross-modal attention fusion mechanism. Its core lies in converting skeleton data into an image representation, facilitating processing using a unified convolutional network architecture.
[0021] This patent achieves technological breakthroughs in multi-source skeleton data fusion and dual-branch network architecture design. Specifically, during the data acquisition stage, both 2D pose estimation algorithms (OpenPose) and 3D pose estimation models are used to obtain enhanced skeleton representations, constructing richer skeleton feature descriptions. In the network architecture design, the skeleton branch uses a dedicated graph convolutional network, GaitGCN, to process the topology of the skeleton data, including three input channels: joint branch, velocity branch, and bone branch, and introduces an ECA channel attention mechanism to improve feature representation capabilities. The contour branch integrates a spatial transformer (CST) to handle viewpoint changes and optimizes the training process through a hybrid pooling strategy and multiple loss functions. In terms of fusion strategy, a dual adaptive fusion module of SEFFM and FFSCT is designed, which can automatically select the optimal fusion method according to different dataset sizes.
[0022] The two differ fundamentally in their data representation and processing strategies: the former converts skeleton data into Gaussian heatmaps for unified image processing, while the latter preserves the original topological structure of the skeleton data and uses specialized graph networks for processing, while also having significant advantages in the diversity of data sources and the adaptability of the fusion mechanism.
[0023] 2. Patent CN120340134A employs a layered processing strategy combining a dual-branch shallow feature extraction module with a deep Transformer in its feature extraction network architecture design. In the shallow feature extraction stage, this method constructs two structurally identical shallow feature encoders. Each encoder contains three components: a global feature sub-encoder, a local feature sub-encoder, and a spatiotemporal feature fusion unit. A pseudo-3D convolutional operation group (consisting of two 1×n×n convolutions and one m×1×1 convolution) replaces the traditional convolutional layer, achieving a balance between computational efficiency and feature representation capability. In the deep feature extraction stage, a sliding window Transformer module is introduced. This module divides the feature map into 3D windows through a block-based mechanism, utilizes the self-attention mechanism of the Swing Transformer to model long-range spatiotemporal dependencies, and adds a Linear Embedding layer after the fourth layer for dimension mapping. Its core advantage lies in combining the local feature extraction capability of CNNs with the global modeling capability of Transformers.
[0024] This patent achieves technological innovation in specialized network architecture design and the integration of hybrid attention mechanisms. Specifically, the skeleton branch employs a specially designed graph convolutional network, GaitGCN, which includes three parallel input branches (joint branch for processing keypoint coordinates, velocity branch for calculating inter-frame motion velocity, and skeleton branch for extracting structural connection information). It also innovatively introduces the ECA-ResGCN module, which enhances the spatiotemporal feature learning capability of the skeleton through an efficient channel attention mechanism (ECA) and residual connections. In the contour branch, a convolutional network, GaitContour, integrating a spatial transformer is constructed. This network fuses a frame-level part feature extractor (FPFE), a convolutional spatial transformer (CST), and a micro-motion capture module (MCM). The CST simultaneously handles viewpoint registration and phase registration through a cross-attention fusion mechanism, while the MCM uses a hybrid pooling strategy to aggregate spatiotemporal features. A dual adaptive fusion mechanism is designed in the feature fusion layer, including a SEFFM module based on SE blocks and an FFSCT module based on Transformer, which can dynamically select the optimal fusion strategy according to the dataset size.
[0025] The two differ significantly in their network architecture design philosophy: the former adopts a unified layered processing strategy, extracting basic features through shallow CNNs and then using Transformers to model global dependencies; the latter adopts a specialized branch design, selecting the most suitable network structure for the characteristics of different modal data (graph convolution to process skeleton topology, CNN to process contour images), and integrating multiple attention mechanisms within each branch to achieve more refined and adaptive feature extraction and fusion. Summary of the Invention
[0026] This invention aims to provide a gait recognition method based on contour and skeleton hybrid attention feature fusion. Gait features are extracted through contour-based and skeleton-based branches: in the skeleton-based branch, GaitGCN, a graph convolutional network with channel attention mechanism, efficiently extracts skeleton features; in the contour-based branch, a GaitContour convolutional neural network model, which integrates a spatial transformer, achieves robust contour feature extraction. Furthermore, intelligent fusion of the two feature types is achieved through an attention-based SEFFM module and a Transformer-based FFSCT module.
[0027] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0028] A gait recognition method based on contour and skeletal hybrid attention feature fusion, characterized by the following steps:
[0029] S1. Gait data acquisition and preprocessing;
[0030] S2. Construct the dual-branch gait recognition framework GaitFusion;
[0031] S3. In the skeleton branch, skeleton features are extracted based on the graph convolutional network GaitGCN.
[0032] S4. In the contour branch, the GaitContour convolutional neural network with integrated spatial transformer is used to extract contour features;
[0033] S5. Adaptive fusion of skeleton features and contour features is performed through the feature fusion module;
[0034] S6. Generate the final gait feature representation for identity recognition;
[0035] S7. Model Optimization and Deployment.
[0036] As a preferred technical solution of the present invention, step S1 is specifically as follows:
[0037] S11. Extract human contour images from gait video sequences.
[0038] First, the foreground human body region is separated from the original RGB image using a background subtraction algorithm. Then, binarization is performed to obtain a silhouette sequence. The silhouette image is then normalized in size, the pixel size is adjusted uniformly, and center alignment is performed.
[0039] S12. Extract key point data of the human skeleton.
[0040] The OpenPose algorithm is used to extract human key points and obtain the (x,y) coordinates of the key points; at the same time, the SMPL-X-based 3D pose estimation model or Azure Kinect depth camera is used to obtain the (x,y,z) coordinates of 3D key points. The 2D and 3D key point data are fused to form an enhanced skeleton sequence.
[0041] S13, Data Synchronization and Alignment
[0042] By strictly aligning the silhouette sequence and skeleton sequence in time, with each frame of silhouette image corresponding to the skeleton data at the same moment, a paired multimodal gait sequence is constructed.
[0043] As a preferred technical solution of the present invention, step S2 is specifically as follows:
[0044] The constructed dual-branch gait recognition framework GaitFusion includes two parallel branches: a skeleton branch and a contour branch, as well as a feature fusion module.
[0045] As a preferred technical solution of the present invention, step S3 is as follows:
[0046] S31. The graph convolutional network GaitGCN with channel attention mechanism is used to extract skeleton features. The graph convolutional network GaitGCN contains three input branches: joint branch, velocity branch and skeleton branch. Among them, the joint branch directly processes the key point coordinate information, the velocity branch calculates the motion velocity of key points between adjacent frames, and the skeleton branch extracts the structural information of human skeleton connection.
[0047] S32. Residual graph convolutional network blocks and channel attention mechanisms are used to process skeleton sequence data. The GaitGCN graph convolutional network adopts the ECA-ResGCN module. The ECA-ResGCN module introduces the efficient channel attention mechanism ECA on the basis of residual graph convolutional networks.
[0048] The complete calculation process of the ECA-ResGCN module is as follows:
[0049] (1);
[0050] (2);
[0051] in Represents a block of a graph convolutional network. This represents a high-efficiency channel attention block. and These represent the (x, y) coordinates of the 2D keypoints;
[0052] S33. After obtaining the output features, the output features of the three input branches are fused, and deep feature learning is performed through stacked ECA-ResGCN modules. Finally, a skeleton feature vector with a dimension of 128 is generated through global average pooling and fully connected layers. .
[0053] As a preferred technical solution of the present invention, step S4 is specifically as follows:
[0054] The GaitContour convolutional neural network, which integrates a spatial transformer, processes silhouette sequences through a frame-level part feature extractor and a micro-motion capture module, as follows:
[0055] S41, Frame-level component feature extraction:
[0056] Instead of conventional convolution, a focal convolutional layer is used to extract the component-level spatial features of each frame's contour. The human body is divided into different horizontal strips, and local features of the head, torso, hands, legs, and feet are extracted separately.
[0057] S42. Convolutional Space Transformation:
[0058] By combining convolution and spatial transform through a cross-attention fusion mechanism, both viewpoint registration and phase registration are processed simultaneously, capturing local and global contextual information of cross-viewpoint images.
[0059] S43, Micro-motion capture:
[0060] Parallel MCM modules are used to aggregate the spatiotemporal features of each component. Each MCM includes hybrid pooling and temporal aggregation, specifically for the hybrid pooling of the input gait sequence x. The calculation method is as follows:
[0061] (3);
[0062] (4);
[0063] In the formula, Used to determine max pooling With average pooling The mixing ratio;
[0064] S44, Multiple Loss Optimization:
[0065] The training process combines batch full triplet loss and cross-entropy loss, as shown in the following formula:
[0066] (5);
[0067] in It varies between 0 and 1. For triple loss, This represents the cross-entropy loss.
[0068] As a preferred technical solution of the present invention, step S5 is as follows:
[0069] The feature fusion module includes two fusion methods: an attention-based SEFFM module and a Transformer-based FFSCT module. Different feature fusion methods are adaptively selected based on the size of different datasets. The feature fusion formula is expressed as:
[0070] (6);
[0071] in, This represents the feature integration operation implemented through the SEFFM and FFSCT modules. Represents skeleton feature vector With contour feature vector The proportionality coefficient is adaptively adjustable.
[0072] As a preferred technical solution of the present invention, step S6 is specifically as follows:
[0073] The fused feature vector The input is fed into a classifier, which calculates the similarity to each identity template in the image library and outputs the recognition result.
[0074] As a preferred technical solution of the present invention, step S7 is as follows:
[0075] S71. Training parameter settings:
[0076] The skeleton branch batch size is 128, with 600 training rounds and an initial learning rate of 0.01; the contour branch batch size is (8, 16), with a maximum of 20,000 iterations and a learning rate of 1×10^-3.
[0077] S72, Model Lightweighting:
[0078] The entire dual-branch gait recognition framework GaitFusion contains only 1.90M parameters, with the feature fusion module containing only 0.11M parameters, making it suitable for practical deployment applications.
[0079] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0080] This invention effectively improves gait recognition accuracy under complex conditions such as clothing changes and carrying items by fusing the complementary advantages of contours and skeletons. It excels in addressing the challenges of gait recognition in unconstrained environments, particularly in scenarios involving carrying items and changing clothing conditions. Unlike existing methods that manually set fusion ratios, the proposed SEFFM and FFSCT fusion module automatically learns the optimal fusion weight λ based on input features. Attention-based SEFFM is suitable for medium-sized datasets, while Transformer-based FFSCT performs better on large-scale datasets. The combination of these two fusion strategies enhances the system's adaptability. This invention achieves long-distance, contactless identity recognition while maintaining a lightweight model and computational efficiency, making it suitable for deployment on resource-constrained devices. Attached Figure Description
[0081] Figure 1 This is a general framework diagram of the present invention;
[0082] Figure 2 This is a structural diagram of the skeleton branch model of the present invention;
[0083] Figure 3 This is a schematic diagram of the architecture of the ResGCN module, SE-ResGCN module, and ECA-ResGCN module in the GaitGCN convolutional network of this invention;
[0084] Figure 4 This is a structural diagram of the contour branch model of the present invention;
[0085] Figure 5 This is a block diagram of the feature fusion modules SEFFM and FFSCT of the present invention. Detailed Implementation
[0086] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:
[0087] like Figure 1 As shown, the gait recognition method based on contour and skeleton hybrid attention feature fusion of the present invention includes the following steps:
[0088] Step S1: Gait data acquisition and preprocessing;
[0089] The CASIA-B large-scale multi-view gait dataset was used, containing gait data from 124 subjects in 11 viewpoints (0°, 18°, 36°, ..., 180°). The dataset considers three walking conditions: normal walking (NM) with 6 sequences per person, carrying a backpack (BG) with 2 sequences per person, and wearing a jacket (CL) with 2 sequences per person.
[0090] Step S11: Extract human contour images from gait video sequences.
[0091] First, the foreground human body region is separated from the original RGB image using a background subtraction algorithm. Then, the image is binarized to obtain a silhouette sequence. The silhouette image is then normalized to a uniform pixel size of 64×44 pixels and centered.
[0092] Step S12: Extract key point data of the human skeleton
[0093] A dual-source skeleton data fusion strategy was adopted: 17 2D human keypoints were extracted using the OpenPose algorithm to obtain the (x,y) coordinates of the keypoints; simultaneously, the (x,y,z) coordinates of 3D keypoints were obtained using an Azure Kinect depth camera. The 2D and 3D keypoint data were then fused to form an enhanced skeleton sequence.
[0094] Step S13, Data Synchronization and Alignment
[0095] By strictly aligning the silhouette sequence and skeleton sequence in time, with each frame of silhouette image corresponding to the skeleton data at the same moment, a paired multimodal gait sequence is constructed.
[0096] Step S2: Construct the dual-branch gait recognition framework GaitFusion;
[0097] The constructed dual-branch gait recognition framework GaitFusion includes two parallel branches: a skeleton branch and a contour branch, as well as a feature fusion module.
[0098] like Figure 2 The process begins with an RGB image, acquires skeleton data via Azure Kinect and Openpose, extracts features through multiple input branches and feeds them into the GaitGCN graph convolutional network, and finally outputs the result after global average pooling and fully connected layers.
[0099] like Figure 3 As shown, the design inspiration for the ECA-ResGCN model comes from the Efficient Channel Attention (ECA) mechanism based on the Squeeze Excitation (SE) mechanism. Compared with the SE block, the ECA block has an improved activation module and can be more efficiently integrated into various convolutional neural networks.
[0100] Step S3: In the skeleton branch, extract skeleton features based on the graph convolutional network GaitGCN;
[0101] S31. The graph convolutional network GaitGCN with channel attention mechanism is used to extract skeleton features. The graph convolutional network GaitGCN contains three input branches: joint branch, velocity branch and skeleton branch. Among them, the joint branch directly processes the key point coordinate information, the velocity branch calculates the motion velocity of key points between adjacent frames, and the skeleton branch extracts the structural information of human skeleton connection.
[0102] S32. Residual graph convolutional network blocks and channel attention mechanisms are used to process skeleton sequence data. The GaitGCN graph convolutional network adopts the ECA-ResGCN module. The ECA-ResGCN module introduces the efficient channel attention mechanism ECA on the basis of residual graph convolutional networks.
[0103] The complete calculation process of the ECA-ResGCN module is as follows:
[0104] (1);
[0105] (2);
[0106] in Represents a block of a graph convolutional network. This represents a high-efficiency channel attention block. and These represent the (x, y) coordinates of the 2D keypoints;
[0107] S33. After obtaining the output features, the output features of the three input branches are fused, and deep feature learning is performed through stacked ECA-ResGCN modules. Finally, a skeleton feature vector with a dimension of 128 is generated through global average pooling and fully connected layers. .
[0108] like Figure 4 As shown, the GaitContour convolutional neural network combines a convolutional neural network (CNN) with a spatial Transformer. Furthermore, GaitContour employs multiple loss functions and a hybrid pooling strategy. The combination of multiple loss functions can compensate for the shortcomings of a single loss function, thereby improving performance. The advantage of the hybrid pooling strategy lies in its excellent ability to effectively suppress overfitting that occurs during training with multiple loss functions.
[0109] Step S4: In the contour branch, the GaitContour convolutional neural network with integrated spatial transformer is used to extract contour features;
[0110] The GaitContou convolutional neural network processes contour sequences through a frame-level part feature extractor and a micro-motion capture module, as follows:
[0111] Step S41, Frame-Level Part Feature Extraction (FPFE):
[0112] Instead of conventional convolution, a focal convolutional layer is used to extract the component-level spatial features of the contour in each frame. The human body is divided into multiple different horizontal strips, and local features of the head, torso, hands, legs, and feet are extracted respectively.
[0113] Step S42, Convolution Space Transformation:
[0114] By combining convolution and spatial transform (CST) through a cross-attention fusion mechanism, both viewpoint registration and phase registration are processed simultaneously, capturing local and global contextual information of cross-viewpoint images;
[0115] Step S43, Micro-motion capture (MCM):
[0116] Parallel MCM modules are used to aggregate the spatiotemporal features of each component. Each MCM includes hybrid pooling and temporal aggregation, specifically for the hybrid pooling of the input gait sequence x. The calculation method is as follows:
[0117] (3);
[0118] (4);
[0119] In the formula, Used to determine max pooling With average pooling The mixing ratio;
[0120] Step S44, Multiple Loss Optimization:
[0121] The training process combines the batch full triplet loss (BA₊ triplet loss) and the cross-entropy loss, as shown in the following equation:
[0122] (5);
[0123] in It varies between 0 and 1. For triple loss, This represents the cross-entropy loss.
[0124] Step S5: Adaptively fuse skeleton features and contour features using the feature fusion module;
[0125] The feature fusion module includes two fusion methods: the attention-based SEFFM module and the Transformer-based FFSCT module.
[0126] The SEFFM module contains an SE block and a feature fusion block, which adaptively fuses contour and skeleton features through two stages: squeezing and excitation.
[0127] The FFSCT module contains an inner T-Block and an outer T-Block, which act as an encoder and decoder to process shallow and deep features, respectively, achieving feature fusion in spatial and channel dimensions.
[0128] The feature fusion module is adaptively selected based on different dataset sizes: for medium-sized datasets, the attention-based SEFFM module is preferred, and for large-scale datasets, the Transformer-based FFSCT module is preferred to achieve the best recognition performance.
[0129] The feature fusion formula is expressed as:
[0130] (6);
[0131] Among them, symbols This represents the feature integration operation implemented through SEFFM and FFSCT. Representing skeleton features Contour features The proportionality coefficient is adaptively adjustable.
[0132] Step S6: Generate the final gait feature representation for identity recognition;
[0133] The fused feature vector The input is fed into a classifier, which calculates the similarity to each identity template in the image library and outputs the recognition result.
[0134] Step S7: Model optimization and deployment;
[0135] Step S71: Setting training parameters
[0136] The skeleton branch batch size is 128, with 600 training rounds and an initial learning rate of 0.01; the contour branch batch size is (8, 16), with a maximum of 20,000 iterations and a learning rate of 1×10^-3.
[0137] Step S72, Model Lightweighting
[0138] The entire dual-branch gait recognition framework GaitFusion contains only 1.90M parameters, with the feature fusion module containing only 0.11M parameters, making it suitable for practical deployment applications.
[0139] This invention effectively improves gait recognition accuracy under complex conditions such as clothing changes and carrying items by integrating the complementary advantages of contour and skeleton. It excels in addressing the challenges of gait recognition in unconstrained environments, particularly in scenarios involving carrying items and changes in clothing status. This invention achieves long-distance, non-contact identity recognition while maintaining a lightweight model and computational efficiency, making it suitable for deployment on resource-constrained devices.
[0140] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.
Claims
1. A gait recognition method based on contour and skeleton mixed attention feature fusion, characterized in that, Comprising the following steps: S1, gait data acquisition and preprocessing; S2, constructing a double-branch gait recognition framework GaitFusion; S3, in the skeleton branch, extracting skeleton features based on a graph convolutional network GaitGCN; S4, in the contour branch, extracting contour features using a convolutional neural network GaitContour integrated with a spatial transformer; S5, adaptively fusing the skeleton features and the contour features through a feature fusion module; S6, generating a final gait feature representation for identity recognition; S7, model optimization and deployment.
2. The gait recognition method based on contour and skeleton mixed attention feature fusion according to claim 1, characterized in that, Step S1 is specifically as follows: S11, extracting human contour images from gait video sequences First, use the background subtraction algorithm to separate the foreground human region from the original RGB image, then perform binaryzation to obtain the contour silhouette sequence, normalize the size of the contour image, uniformly adjust the pixel size, and perform center alignment processing; S12, extracting human skeleton key point data Use the OpenPose algorithm to extract human key points to obtain the (x, y) coordinates of the key points; at the same time, use a 3D pose estimation model based on SMPL-X or an Azure Kinect depth camera to obtain the (x, y, z) coordinates of the 3D key points, and fuse the 2D and 3D key point data to form an enhanced skeleton sequence; S13, data synchronization and alignment Make the contour silhouette sequence and the skeleton sequence strictly aligned in time, so that each contour image corresponds to the skeleton data at the same time, and construct a paired multi-modal gait sequence.
3. The gait recognition method based on contour and skeleton mixed attention feature fusion according to claim 1, characterized in that, Step S2 is specifically as follows: The constructed double-branch gait recognition framework GaitFusion includes two parallel branches of a skeleton branch and a contour branch, and a feature fusion module.
4. The gait recognition method based on contour and skeleton mixed attention feature fusion according to claim 1, characterized in that, Step S3 is specifically as follows: S31, use a graph convolutional network GaitGCN with a channel attention mechanism to extract skeleton features, the graph convolutional network GaitGCN includes three input branches of a joint branch, a velocity branch and a skeleton branch, wherein the joint branch directly processes key point coordinate information, the velocity branch calculates the motion velocity of key points between adjacent frames, and the skeleton branch extracts structural information of human body skeleton connection; S32, process the skeleton sequence data through a residual graph convolutional network block and a channel attention mechanism, the graph convolutional network GaitGCN adopts an ECA-ResGCN module, the ECA-ResGCN module introduces an efficient channel attention mechanism ECA based on a residual graph convolutional network, The complete calculation process of the ECA-ResGCN module is as follows: (1); (2); wherein, denotes the generated feature enhanced by the ECA mechanism, denotes a graph convolution network block, denotes an efficient channel attention block, and denote the (x, y) coordinates of 2D key points, respectively, denotes a feature fusion operation; S33, after obtaining the output features, the output features of the three input branches are fused, deep feature learning is performed through the stacked ECA-ResGCN modules, and finally a skeleton feature vector with a dimension of 128 is generated through global average pooling and a fully connected layer .
5. The gait recognition method based on contour and skeleton mixed attention feature fusion according to claim 1, characterized in that, Step S4 is specifically as follows: The convolutional neural network GaitContour integrated with a spatial transformer processes the contour silhouette sequence through a frame-level part feature extractor and a micro-motion capture module, specifically as follows: S41, frame-level part feature extraction: Use a focal convolution layer to replace a regular convolution to extract part-level spatial features of each contour, divide the human body into different horizontal strips, and extract local features of head, torso, hand, leg and foot parts respectively; S42, convolution spatial transformation: By combining convolution and spatial transformer through cross-attention fusion mechanism, local and global context information of cross-view images is captured while processing view registration and phase registration; S43, micro-motion capture: The spatio-temporal features of each component are aggregated using a parallel MCM module, each MCM comprising hybrid pooling and temporal aggregation, the hybrid pooling being for an input gait sequence x The computation is as follows: (3); (4); wherein denotes a characteristic sequence to a pooling operation that takes the maximum value, is a mixing ratio coefficient, for determining the maximum value pooling with the average value pooling the mixing ratio; S44, multiple loss optimization: The batch full-triplet loss and cross-entropy loss are fused for training, as follows: (5); wherein is a weight coefficient for the loss function and varies between 0 and 1, is a triplet loss, is a cross-entropy loss.
6. The gait recognition method based on contour and skeleton mixed attention feature fusion according to claim 1, characterized in that, Step S5 is specifically as follows: The feature fusion module includes two fusion modes, namely SEFFM module based on attention and FFSCT module based on Transformer. Different feature fusion modes are adaptively selected according to different data set scales. The feature fusion formula is represented as: (6); Wherein, FFM is a feature fusion module, represents a feature integration operation implemented through the SEFFM module and the FFSCT module, represents a skeleton feature vector represents a contour feature vector a proportional coefficient of the skeleton feature vector and the contour feature vector, which can be adaptively adjusted.
7. The gait recognition method based on contour and skeleton mixed attention feature fusion according to claim 1 or 6, characterized in that, Step S6 is specifically as follows: The fused feature vector is input to a classifier, the similarity with each identity template in the gallery is calculated, and the recognition result is output.
8. The gait recognition method based on contour and skeleton mixed attention feature fusion according to claim 1, characterized in that, Step S7 is specifically as follows: S71, training parameter setting: The skeleton branch batch size is 128, the training is 600 rounds, the initial learning rate is 0.01; the contour branch batch size is (8, 16), the maximum iteration is 20000 times; the learning rate is 1×10^-3; S72, model lightening: The whole dual-branch gait recognition framework GaitFusion only contains 1.90M parameters, Among them, the feature fusion module only contains 0.11M parameters, which is suitable for actual deployment application.
Citation Information
Patent Citations
Gait recognition system guided by human body posture information
CN117809372A
Multi-modal gait recognition method and system
CN120340134A
Cited By
Multi-modal gait recognition method based on three-domain fusion and frequency domain guided learning
CN122049990A
