Gait recognition method and system based on profile and skeleton fusion of bidirectional cross attention

Abstract

The present application relates to a gait recognition method and system based on bidirectional cross attention profile and skeleton fusion, comprising: A, gait multi-modal data acquisition and alignment processing; B, profile branch hierarchical spatio-temporal feature extraction; extracting multi-level appearance spatio-temporal features of gait profile sequence; C, skeleton branch hierarchical spatio-temporal feature extraction; modeling the spatial dependence relationship between human body joints and the long-distance time dependence relationship of joints; D, bidirectional hierarchical cross attention collaborative fusion; multi-stage collaborative interaction and fusion of profile intermediate features and skeleton intermediate features; E, overall feature aggregation; the features output by the profile branch, skeleton branch and cross-modal collaborative fusion module are jointly aggregated to form the final identity representation; F, multi-modal gait recognition model training; G, multi-modal cross-view gait recognition. The present application can enhance the complementary use of profile information and skeleton information, and improve the stability and robustness of gait recognition under complex view, occlusion and appearance change conditions.

Description

Technical Field

[0001] This invention relates to a gait recognition method and system based on contour and skeleton fusion with bidirectional cross-attention, belonging to the fields of deep learning and biometric recognition technology. Background Technology

[0002] Gait recognition technology aims to identify individuals by analyzing their spatiotemporal movement patterns during walking. Its non-contact and long-range effectiveness makes it well-suited for applications in public safety. However, in real-world, unconstrained applications, the robustness of gait recognition faces significant challenges due to factors such as changes in camera perspective, pedestrian clothing, and obstructions from objects. These factors can significantly interfere with appearance and motion information, leading to a substantial decrease in model performance.

[0003] In gait recognition research, appearance-based methods excel at learning representations that preserve the overall body shape and contour, thus providing strong appearance priors and naturally suppressing texture interference. However, these representations lack explicit encoding of joint topology and kinematic constraints, making them susceptible to self-occlusion and covariate interference, exhibiting a "shape-strong, structure-weak" characteristic. In contrast, model-based methods generate representations that accurately model structure and motion, possessing strong spatiotemporal discriminative capabilities, but underutilize shape cues of the torso and limbs, resulting in a "structure-strong, shape-weak" characteristic. Therefore, the representations generated by each modality are inherently incomplete yet highly complementary, providing a clear theoretical necessity for deep fusion to create more comprehensive and robust identity representations.

[0004] Despite their immense potential, the advantages of multimodal fusion have not been fully explored. Existing multimodal research typically employs shallow aggregation mechanisms, such as simple feature connections or weighted averaging. These methods treat the two representations as static, independent sources of internal information, failing to foster deep, iterative dialogue between them. Furthermore, the skeleton branches of many fusion frameworks rely on graph convolutional networks. Graph convolutional networks, based on fixed physical topologies, inherently have local aggregation mechanisms that struggle to explicitly capture long-range dependencies between distant joints and time-varying global interactions, both crucial for gait recognition. Thus, these popular methods face a double limitation: they not only rely on constrained encoder architectures that produce suboptimal single-modal representations, but their shallow fusion mechanisms also hinder any deep, bidirectional dialogue between modalities. As a result, the final gait representation remains a static feature representation of its individual parts. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a gait recognition method and system based on contour and skeleton fusion using bidirectional cross-attention. Invention Overview: A gait recognition method and system based on bidirectional cross-attention contour and skeleton fusion includes gait multimodal data acquisition and alignment processing, contour branch hierarchical spatiotemporal feature extraction, skeleton branch hierarchical spatiotemporal feature extraction, bidirectional hierarchical cross-attention collaborative fusion, overall feature aggregation, model training, and cross-view gait recognition.

[0007] To ensure the temporal correspondence between contour modalities and skeleton modalities during feature interaction, gait contour sequences and skeleton sequences are synchronously sampled and aligned. To extract discriminative gait appearance information, a hierarchical convolutional contour encoding module is constructed to extract multi-stage convolutional spatiotemporal features from the contour sequences, combining horizontal pooling and temporal pooling to obtain contour branch feature representations. To model the spatial dependencies between human joints and the long-distance temporal dependencies of joints, a hierarchical spatiotemporal transformation encoding module is constructed to model the skeleton sequences using hierarchical spatiotemporal features, obtaining skeleton branch feature representations. To enhance the interaction between contour and skeleton branches at different feature levels, a bidirectional hierarchical cross-attention collaborative fusion module is constructed, performing spatiotemporal pooling, feature projection, and bidirectional cross-attention collaborative modeling on the intermediate layer features of the two branches, and obtaining multi-level fused features through adaptive feature fusion. To improve the completeness and discriminative ability of the final identity representation, an overall feature aggregation module is constructed, jointly aggregating contour single-modal features, skeleton single-modal features, intermediate projection features, and cross-modal fusion features. To improve the overall framework recognition performance, triplet loss and cross-entropy loss are used for joint training of the model. Finally, the trained model is used for multimodal cross-view gait recognition.

[0008] Terminology Explanation: 1. 3D-CNN-based backbone network: This is an adaptation and modification based on existing 3D convolutional networks (similar to 3D-ResNet). It refers to a convolutional neural network that uses 3D convolutional kernels (such as 3D-ResNet), which can extract the dynamic appearance and spatiotemporal feature changes of gait sequences in both spatial (width and height) and temporal (frame sequence) dimensions through a 3D sliding window.

[0009] 2. Generalized Mean Horizontal Pyramid Pooling (GeMHPP) Operator: This is a pooling operation for feature extraction that combines the adaptive properties of Generalized Mean Pooling (GeM) with the multi-scale local modeling capabilities of Horizontal Pyramid Pooling. This operator divides the input feature map into multiple horizontal strips (pyramid structure) of different scales along the height direction. Generalized Mean Pooling is applied independently within each strip, and the pooling results of all strips are concatenated into a global feature vector.

[0010] 3. Linear Embedding Layer: This is a linear projection operator, typically implemented by a fully connected layer.

[0011] 4. Initial Temporal Convolutional Network: TCN is a network structure that uses one-dimensional convolution to perform sliding window operations on sequence data along the time dimension, which can effectively capture the short-term motion dynamics and continuity between local adjacent frames of the skeleton sequence.

[0012] The technical solution of the present invention is as follows: A gait recognition method based on bidirectional cross-attention contour and skeleton fusion includes the following steps: A. Gait multimodal data acquisition and alignment processing; Gait contour sequence data and gait skeleton sequence data are acquired and preprocessed. The preprocessing includes contour sequence normalization, skeleton joint point sequence construction, and data alignment. Among them, the gait contour sequence and gait skeleton sequence are synchronously sampled and time-aligned. B. Contour branch hierarchical spatiotemporal feature extraction; Extract multi-level spatiotemporal features of gait contour sequences; for the gait contour sequences processed in step A, perform hierarchical spatiotemporal feature extraction through initial layers and multi-stage convolutional blocks to obtain intermediate contour features at different levels; among them, the final-level contour features are processed by horizontal pooling and temporal pooling to obtain the contour single-modal feature representation, i.e., contour features. C. Extraction of spatiotemporal features by skeleton branch hierarchical structure; Model the spatial dependencies between human joints and the long-distance temporal dependencies between joints; for the gait skeleton sequence processed in step A, perform spatiotemporal feature modeling of the skeleton to obtain intermediate skeleton features at different levels; and fuse the spatial branch output and temporal branch output through a dynamic weighted fusion mechanism; after the final-level skeleton features are processed by temporal pooling, the skeleton single-modal feature representation, i.e., skeleton features, is obtained. D. Two-way hierarchical cross-attention collaborative fusion; Multi-stage collaborative interaction and fusion are performed on the contour intermediate features obtained in step B and the skeleton intermediate features obtained in step C. For contour intermediate features and skeleton intermediate features at different levels, compact representations are obtained through contour branch symmetric spatiotemporal pooling and skeleton branch symmetric spatiotemporal pooling, respectively. Dimensional mapping is then performed through a parallel feature projector to obtain projected intermediate features, achieving a unified embedding representation of features at different levels and modalities. Subsequently, bidirectional cross-modal attention interaction is performed on contour features and skeleton features through cross-attention collaboration to obtain fused features at each level. Finally, an adaptive feature fusion mechanism is used to adaptively aggregate the fused features at each level to obtain the fused output features. E. Overall feature aggregation; The features output by the contour branch, skeleton branch, and cross-modal collaborative fusion module are jointly aggregated to form the final identity representation; the contour single-modal features obtained in step B, the skeleton single-modal features obtained in step C, the projection intermediate features in step D, and the fusion output features are spliced ​​or combined to obtain the final co-evolutionary feature representation, which is used for subsequent training and recognition. F. Training of multimodal gait recognition model; The final co-evolutionary feature representation obtained in step E is used as the pedestrian identity representation. The multimodal gait recognition model is jointly trained using triplet loss and cross-entropy loss. The triplet loss is used to enhance the feature measurement and discrimination ability, and the cross-entropy loss is used to enhance the identity classification and discrimination ability, so as to jointly supervise the training process of the multimodal gait recognition model. G. Multimodal cross-view gait recognition; The gait contour sequences and gait skeleton sequences of the registration set and query set are input into the trained multimodal gait recognition model to extract the corresponding final identity feature representations. By comparing the similarity between the features of the query sample and the features of the registration sample, the identity matching and confirmation of the query sample in the registration set is completed, thus realizing multimodal cross-view gait recognition.

[0013] According to a preferred embodiment of the present invention, in step A, the gait contour sequence is... With gait skeleton sequence The specific preprocessing process is as follows: Data alignment and sampling; acquiring the original gait walking sequence, and synchronously retrieving and loading the corresponding contour feature sequence and skeleton pose sequence through the "triple" information of identity label, walking conditions, and viewpoint; adopting a fixed-length sequential sampling strategy: first, calculating the target sampling frame number. With frame skipping threshold The sampling span constituted, if the original sequence length If insufficient, then proceed according to The original index is expanded cyclically by a multiple of the original index; then, a starting offset is randomly selected within the expanded index field, and samples are extracted in a non-repeating sampling manner within this field window. Arrange the indexes in ascending order to generate a uniformly aligned index set. Ultimately, utilizing Perform synchronous slicing and normalization on the two branches to extract the contents. Frame contour sequence and skeleton sequence .

[0014] According to a preferred embodiment of the present invention, the multimodal gait recognition model includes a contour branch hierarchical convolutional contour encoding module, a skeleton branch hierarchical spatiotemporal transformation encoding module, a bidirectional hierarchical cross-attention collaborative fusion module, and an overall feature aggregation module; The contour branch hierarchical convolutional contour encoding module is used to extract multi-level spatiotemporal features of gait contour sequences; the skeleton branch hierarchical spatiotemporal transformation encoding module is used to model the spatial dependencies between human joints and the long-distance temporal dependencies of joints; the bidirectional hierarchical cross-attention collaborative fusion module is used to perform multi-stage collaborative interaction and fusion of the contour intermediate features obtained in step B and the skeleton intermediate features obtained in step C; the overall feature aggregation module is used to jointly aggregate the features output by the contour branch, skeleton branch and cross-modal collaborative fusion module to form the final identity representation.

[0015] According to a preferred embodiment of the present invention, the specific implementation process of contour branch hierarchical spatiotemporal feature extraction in step B is as follows: b1. Multi-stage spatiotemporal convolutional coding of contours; the contour branch uses a 3D-CNN-based backbone network for hierarchical spatiotemporal feature extraction; for the input contour sequence First, basic spatiotemporal features are captured through an initial 3D convolutional layer, and then the code chain consists of four consecutive main stages. b2. Spatiotemporal co-sampling: Spatiotemporal downsampling operators are executed between stages to reduce spatial resolution and compress the temporal dimension. Construct a containing The feature pyramid has several levels, and the output is denoted as the set of intermediate layer feature maps. ; b3. Dual-path output mechanism; The contour branch hierarchical convolutional contour encoding module adopts a dual-path output strategy: Interaction path: Previous The set of intermediate layer feature maps generated at each stage It is directly transmitted to the bidirectional hierarchical cross-attention collaborative fusion module, providing unpooled multi-scale detail information; Single-modal aggregation path: The high-level semantic feature map output from the last stage Transform into a single-modal identity representation The process includes: First, horizontal pooling preserves the proportions of the human body structure; then, time pooling aggregates the motion features of the entire gait cycle. The calculation formula is as follows: (1); in, This indicates a time pooling operation. This indicates a horizontal pooling operation.

[0016] According to a preferred embodiment of the present invention, the specific implementation process of the skeleton branch hierarchical spatiotemporal feature extraction in step C is as follows: c1. Spatiotemporal embedding and local feature initialization; aligning the gait skeleton sequence... The input linear embedding layer is projected into a high-dimensional space, and a learnable spatial location embedding is superimposed. With time series embedding Subsequently, an initial temporal convolutional network is used to capture the local instantaneous dynamics of limb swinging, generating the initial input feature map for the two-stream spatiotemporal transformer. : (2); in, This represents a linear embedding layer mapping operation. Indicates temporal convolutional network operations; c2. Parallel timing and topological coding based on dual-stream space-time transformers; constructing a coding chain consisting of D layers of dual-stream space-time transformers, with each layer modeled using a dual-stream parallel structure: in each coding layer... In the middle, input features The data is fed into the spatial and temporal branches in parallel for processing. Spatial branching is achieved through joint dimensions Multi-head self-attention combined with layer normalization and multilayer perceptron is used to capture long-range topological relationships between key points in the human body, and the output spatial features are obtained. Represented as: (3); (4); This refers to the intermediate layer features of spatial branches; At the same time, time branches, through the time dimension A multi-head self-attention mechanism is implemented to capture the global motion dynamics features of the gait sequence during cross-frame evolution and output the temporal features. ; This refers to the time encoding function; The intermediate features of the time branch are calculated as follows: (5); The final output features of the time branch are calculated as follows: (6); Introducing a dynamic fusion gating mechanism to incorporate the aforementioned spatial features With time characteristics Channel concatenation is performed, and a set of adaptive weights is obtained through a linear mapping layer and a Softmax function prediction. : (7); Finally, adaptive weights are used. Element-wise weighted summation is performed on the spatial and temporal features to obtain the final output features of the current coding layer. : (8); c3. Multi-level feature pyramid and global representation generation: Intermediate features are extracted from different specified coding levels of the two-stream spatiotemporal transformer to construct a skeleton feature set with multi-scale semantic depth. This refers to the intermediate features of the skeleton at different levels; simultaneously, it refers to the output of the last layer of the encoding chain. Perform global pooling to eliminate spatiotemporal redundancy and extract feature vectors that reflect the global identity information of the skeleton unimodality. That is, skeleton single-modal feature representation.

[0017] According to a preferred embodiment of the present invention, the specific implementation process of bidirectional hierarchical cross-attention collaborative fusion in step D is as follows: d1. Symmetric Spatiotemporal Pooling and Feature Projection: First, symmetric spatiotemporal pooling and feature projection operations are used to unify the dimensions between modalities; specifically, this includes: for the intermediate tensors of each level of the skeleton branch. First, temporal max pooling is performed to aggregate motion cues. Then, global joint-level average pooling is performed to generate compact skeleton feature vectors. : ;(9); Meanwhile, the feature map for the contour branch Based on max pooling in the execution time dimension, a generalized mean-level pyramid pooling operator is combined to capture spatial layout features and generate corresponding contour feature vectors. : (10); A set of stage-specific linear projection operators are used to map the two modal features to a shared high-dimensional embedding space, resulting in projected feature representations. and : ; (11); The projection results from each level are stitched together to form an intermediate projection feature vector representing the information of a single-modal level. and ; d2. Bidirectional cross-modal attention interaction; after feature alignment is completed, at each sampling level... Construct a bidirectional cross-modal attention interaction flow, utilizing a shared weight matrix. This facilitates deep information exchange between two heterogeneous features; in the attention mechanism, , , These represent the query projection matrix, key projection matrix, and value projection matrix, respectively. The structure-guided appearance enhancement flow uses contour features as the query term and skeleton features as the key and value terms to calculate the appearance features incorporating structural constraints. : (12); in, This refers to the standard scaled dot product attention function, defined as: ;in, The dimension of the key vector is used for scaling to prevent the inner product from becoming too large and causing gradient saturation. It refers to the first Sampling level, the query matrix obtained by linear projection of contour features; and : in the The hierarchy is determined by the skeleton features sharing weights. and The key matrix and value matrix obtained by projection; The appearance-guided structure enhancement flow, which is symmetrical to it, reverses the roles of the two, correcting the motion trajectory of the skeleton through appearance priors to obtain the structural features corrected by appearance priors. : (13); Appearance features Structural features These two enhancement features are then concatenated along the channel dimension and transformed into preliminary fused features for the current layer via a linear mapping layer. : (14); d3. Adaptive Feature Fusion and Multi-Level Representation Aggregation: An adaptive feature fusion mechanism is introduced to perform channel-level dynamic calibration of the initially fused features; channel importance weights are predicted using a multilayer perceptron and activation function. : (15); in, The sigmoid function is represented by the mathematical form: ; Weight Element-wise multiplication is performed with the original features to enhance key feature information, resulting in refined fused features. : (16); Ultimately, executing the hierarchical aggregation strategy will result in the aggregation of elements belonging to the previous... Refined features at each fine-grained stage are aggregated into mid-level fused features. : (17); in, It is a multilayer perceptron; Indicates a splicing operation; The first Features from each high-level semantic stage are independently transformed into late-stage fusion features, i.e., fusion output features. : (18); in, It is a late-stage fusion sensor.

[0018] According to a preferred embodiment of the present invention, the specific implementation process of the overall feature aggregation in step E is as follows: The contour single-modal features obtained in step B The skeleton single-modal features obtained in step C and the intermediate projection features generated in step D. With fusion output features By splicing and aggregating the data, the final co-evolutionary feature representation is constructed. : (19).

[0019] According to a preferred embodiment of the present invention, the specific implementation process of training the multimodal gait recognition model in step F includes: During the training phase, the final co-evolutionary feature representation obtained in step E is used as the core supervision object. The multimodal gait recognition model is trained end-to-end through the joint drive of triplet loss and cross-entropy loss. Specifically, triplet loss enhances the clustering of samples with the same identity and the discriminative power of samples with different identities through metric learning, thereby improving the feature's metric discriminative ability. Cross-entropy loss enhances the model's classification discriminative ability by establishing identity mappings. Furthermore, the loss function is directly applied to the feature set. For each member, calculate the total loss. : (20); in, and To balance the hyperparameters; It refers to each of the following: single-modal global features, mid-level fusion features, late-stage fusion features, and intermediate projection features; This refers to cross-entropy loss; This refers to the loss of the triplet.

[0020] According to a preferred embodiment of the present invention, in step G, multimodal cross-view gait recognition includes: The gait contour sequences and gait skeleton sequences from the registration set and query set are respectively input into a pre-trained multimodal gait recognition model. Through contour branch hierarchical spatiotemporal feature extraction, skeleton branch hierarchical spatiotemporal feature extraction, bidirectional hierarchical cross-attention collaborative fusion, and overall feature aggregation, the corresponding final identity feature representations are extracted. ; Feature similarity is measured by calculating the cosine or Euclidean distance between the features of the query sample and the features of all samples in the registration set; the samples are then sorted according to their similarity scores, and the identity of the registration sample with the highest score is taken as the identification result.

[0021] A gait recognition method and system based on bidirectional cross-attention contour and skeleton fusion includes: The gait multimodal data acquisition and alignment processing module is configured to: acquire gait contour sequence data and gait skeleton sequence data, and perform preprocessing; the preprocessing includes contour sequence normalization, skeleton joint point sequence construction, and data alignment processing; wherein, the gait contour sequence and gait skeleton sequence are synchronously sampled and time-aligned. The contour branch hierarchical convolutional contour encoding module is configured to: extract multi-level spatiotemporal features of gait contour sequences; for the gait contour sequence processed by step A, hierarchical spatiotemporal features are extracted through the initial layer and multi-stage convolutional blocks to obtain intermediate contour features at different levels; among them, the final-level contour features are processed by horizontal pooling and temporal pooling to obtain the contour single-modal feature representation, i.e., contour features. The skeleton branch hierarchical spatiotemporal transformation coding module is configured to: model the spatial dependencies between human joints and the long-distance temporal dependencies between joints; for the processed gait skeleton sequence, perform skeleton spatiotemporal feature modeling to obtain intermediate skeleton features at different levels; and fuse the spatial branch output and temporal branch output through a dynamic weighted fusion mechanism; after the final-level skeleton features are processed by temporal pooling, the skeleton single-modal feature representation, i.e., skeleton features, is obtained. The bidirectional hierarchical cross-attention collaborative fusion module is configured to: perform multi-stage collaborative interaction and fusion on the obtained contour intermediate features and the obtained skeleton intermediate features; for contour intermediate features and skeleton intermediate features at different levels, compact representations are obtained through contour branch symmetric spatiotemporal pooling and skeleton branch symmetric spatiotemporal pooling, respectively, and then dimension mapping is performed through a parallel feature projector to obtain projected intermediate features, realizing a unified embedding representation of features at different levels and different modalities; then, bidirectional cross-modal attention interaction is performed on the contour features and skeleton features through cross-attention collaboration to obtain fused features at each level; finally, an adaptive feature fusion mechanism is used to adaptively aggregate the fused features at each level to obtain the fused output features; The overall feature aggregation module is configured to: jointly aggregate the features output by the contour branch, skeleton branch, and cross-modal collaborative fusion module to form the final identity representation; and splice or combine the contour single-modal features, skeleton single-modal features, projection intermediate features, and fusion output features to obtain the final collaborative co-evolutionary feature representation for subsequent training and recognition. The multimodal gait recognition model training module is configured to: use the final co-evolutionary feature representation as the pedestrian identity representation, and jointly train the multimodal gait recognition model using triplet loss and cross-entropy loss; wherein, triplet loss is used to enhance the feature measurement and discrimination ability, and cross-entropy loss is used to enhance the identity classification and discrimination ability, so as to jointly supervise the training process of the multimodal gait recognition model; The multimodal cross-view gait recognition module is configured to: input the gait contour sequences and gait skeleton sequences of the registration set and query set into the trained multimodal gait recognition model, extract the corresponding final identity feature representation; and complete the identity matching and confirmation of the query sample in the registration set by comparing the similarity between the query sample features and the registration sample features, thereby realizing multimodal cross-view gait recognition.

[0022] The beneficial effects of this invention are as follows: 1. The multimodal gait recognition framework involved in this invention introduces gait contour modality and gait skeleton modality simultaneously, and jointly utilizes appearance shape information and human body structure motion information in the same identity representation process. Compared with single modality feature extraction methods, it can improve the integrity and complementarity of gait feature representation.

[0023] 2. The hierarchical convolutional contour encoding module involved in this invention performs hierarchical encoding of gait contour sequences through a multi-stage convolutional feature extraction structure, and combines horizontal pooling and temporal pooling to obtain contour discriminative representations. This allows for the extraction of gait appearance features with temporal statistical characteristics while preserving human body shape information. The hierarchical spatiotemporal transformation encoding module involved in this invention, by performing hierarchical spatiotemporal modeling of the skeleton sequence, extracts cross-joint spatial relationships and long-distance temporal dependencies between joints. This enhances the expressive ability of skeleton modalities to human structural constraints and movement patterns, and improves the stability of skeleton features under changing viewpoint conditions.

[0024] 3. The bidirectional hierarchical cross-attention collaborative module involved in this invention enables phased interaction of multi-level intermediate features between contour branches and skeleton branches, and achieves bidirectional information transmission through the cross-attention collaborative module. This allows the two modalities to continuously collaborate during feature extraction, thereby improving the efficiency of cross-modal information utilization. It can gradually build the association between contour features and skeleton features at different semantic levels, allowing complementary features with different information to enhance each other, which is beneficial for forming a more discriminative representation. Attached Figure Description

[0025] Figure 1 The overall framework diagram of the gait recognition method based on bidirectional cross-attention contour and skeleton fusion proposed in this invention is shown below. Figure 2 This is a schematic diagram of the dual-stream space-time converter in this invention; Figure 3 This is a schematic diagram of cross-attention collaboration in the present invention; Figure 4 This is a schematic diagram of bidirectional cross-modal attention interaction in this invention; Figure 5 This is a schematic diagram of the adaptive feature fusion mechanism in this invention. Detailed Implementation

[0026] To facilitate understanding of the present invention, the present invention will be further described below through embodiments and in conjunction with the accompanying drawings, but these are not intended to limit the present invention.

[0027] Example 1 Gait recognition methods based on bidirectional cross-attention contour and skeleton fusion, such as Figure 1 As shown, it includes the following steps: A. Gait multimodal data acquisition and alignment processing; Gait contour sequence data and gait skeleton sequence data are acquired and preprocessed. The preprocessing includes contour sequence normalization, skeleton joint point sequence construction, and data alignment. Among them, the gait contour sequence and gait skeleton sequence are synchronously sampled and time-aligned to ensure the correspondence between the two modalities in the time dimension and to provide aligned input for subsequent cross-modal hierarchical interactive fusion.

[0028] B. Contour branch hierarchical spatiotemporal feature extraction; like Figure 1 As shown, multi-level spatiotemporal features of gait contour sequences are extracted. For the gait contour sequence processed in step A, hierarchical spatiotemporal features are extracted through the initial layer and multi-stage convolutional blocks to obtain intermediate contour features at different levels. Among them, the final contour features are processed by horizontal pooling and temporal pooling to obtain the contour single-modal feature representation, i.e., contour features. C. Extraction of spatiotemporal features by skeleton branch hierarchical structure; like Figure 1 As shown, the spatial dependencies between human joints and the long-distance temporal dependencies between joints are modeled; for the gait skeleton sequence processed in step A, the spatiotemporal features of the skeleton are modeled to obtain intermediate skeleton features at different levels; and the spatial branch output and the temporal branch output are fused through a dynamic weighted fusion mechanism; after the final-level skeleton features are processed by temporal pooling, the skeleton single-modal feature representation, i.e., skeleton features, is obtained. D. Two-way hierarchical cross-attention collaborative fusion; Multi-stage collaborative interaction and fusion are performed on the contour intermediate features obtained in step B and the skeleton intermediate features obtained in step C. For contour intermediate features and skeleton intermediate features at different levels, compact representations are obtained through contour branch symmetric spatiotemporal pooling and skeleton branch symmetric spatiotemporal pooling, respectively. Dimensional mapping is then performed through a parallel feature projector to obtain projected intermediate features, achieving a unified embedding representation of features at different levels and modalities. Subsequently, bidirectional cross-modal attention interaction is performed on contour features and skeleton features through cross-attention collaboration to obtain fused features at each level. Finally, an adaptive feature fusion mechanism is used to adaptively aggregate the fused features at each level to obtain the fused output features. E. Overall feature aggregation; The features output by the contour branch, skeleton branch, and cross-modal collaborative fusion module are jointly aggregated to form the final identity representation; the contour single-modal features obtained in step B, the skeleton single-modal features obtained in step C, the projection intermediate features in step D, and the fusion output features are spliced ​​or combined to obtain the final co-evolutionary feature representation, which is used for subsequent training and recognition. F. Training of multimodal gait recognition model; The final co-evolutionary feature representation obtained in step E is used as the pedestrian identity representation. The multimodal gait recognition model is jointly trained using triplet loss and cross-entropy loss. The triplet loss is used to enhance the feature measurement and discrimination ability, and the cross-entropy loss is used to enhance the identity classification and discrimination ability, so as to jointly supervise the training process of the multimodal gait recognition model. G. Multimodal cross-view gait recognition; The gait contour sequences and gait skeleton sequences of the registration set and query set are input into the trained multimodal gait recognition model to extract the corresponding final identity feature representations. By comparing the similarity between the features of the query sample and the features of the registration sample, the identity matching and confirmation of the query sample in the registration set is completed, thus realizing multimodal cross-view gait recognition.

[0029] Example 2 The difference between the gait recognition method based on bidirectional cross-attention contour and skeleton fusion described in Example 1 and the method described in Example 1 is as follows: In step A, the gait contour sequence is compared. With gait skeleton sequence The specific preprocessing process is as follows: Data alignment and sampling: The original gait walking sequence is acquired, and the corresponding contour feature sequence and skeleton pose sequence are simultaneously retrieved and loaded using the "triple" information of identity label, walking conditions, and viewpoint. To ensure a one-to-one correspondence between the two modalities in the frame index and to unify the input dimension, a fixed-length sequential sampling strategy is adopted: First, the target sampling frame number is calculated. With frame skipping threshold The sampling span constituted, if the original sequence length If insufficient, then proceed according to The original index is expanded cyclically by a multiple of the original index; then, a starting offset is randomly selected within the expanded index field, and samples are extracted in a non-repeating sampling manner within this field window. Arrange the indexes in ascending order to generate a uniformly aligned index set. Ultimately, utilizing Perform synchronous slicing and normalization on the two branches to extract the contents. Frame contour sequence (After central normalization calibration) and skeleton sequence (Maintain strict time step correspondence). Provide aligned tensor inputs for subsequent cross-modal hierarchical interactive fusion.

[0030] The multimodal gait recognition model includes a contour branch hierarchical convolutional contour encoding module, a skeleton branch hierarchical spatiotemporal transformation encoding module, a bidirectional hierarchical cross-attention collaborative fusion module, and an overall feature aggregation module; The contour branch hierarchical convolutional contour encoding module is used to extract multi-level spatiotemporal features of gait contour sequences; the skeleton branch hierarchical spatiotemporal transformation encoding module is used to model the spatial dependencies between human joints and the long-distance temporal dependencies of joints; the bidirectional hierarchical cross-attention collaborative fusion module is used to perform multi-stage collaborative interaction and fusion of the contour intermediate features obtained in step B and the skeleton intermediate features obtained in step C; the overall feature aggregation module is used to jointly aggregate the features output by the contour branch, skeleton branch and cross-modal collaborative fusion module to form the final identity representation.

[0031] In step B, the specific implementation process of contour branch hierarchical spatiotemporal feature extraction is as follows: b1. Multi-stage spatiotemporal convolutional coding of contours; the contour branch uses a 3D-CNN-based backbone network for hierarchical spatiotemporal feature extraction; for the input contour sequence First, basic spatiotemporal features are captured through an initial 3D convolutional layer, and then the encoding chain consists of four consecutive main stages (Stage 1 to Stage 4). b2. Spatiotemporal co-sampling: Spatiotemporal downsampling operators are executed between stages to reduce spatial resolution (height). With width And compress the time dimension This allows the network to gradually expand its receptive field. As the stages deepen, the semantic density of the feature maps gradually increases, thus constructing a network that includes... The feature pyramid has several levels, and the output is denoted as the set of intermediate layer feature maps. Used for subsequent cross-modal interactions.

[0032] b3. Dual-path output mechanism; In order to simultaneously consider fine-grained feature interaction and global identity representation, the contour branch hierarchical convolutional contour encoding module adopts a dual-path output strategy: The contour branch hierarchical convolutional contour encoding module adopts a backbone network architecture based on 3D-CNN (three-dimensional convolutional neural network, such as 3D-ResNet). Its specific function is as follows:

[0033] Initial layer: The input gait contour sequence first passes through an initial 3D convolutional layer (containing convolution, batch normalization, and activation functions) to capture the most basic low-level spatiotemporal dynamic features.

[0034] Multi-stage encoding chain: Subsequently, the feature map enters a hierarchical encoding chain consisting of four consecutive main stages (Stage 1 to Stage 4, i.e., the N stages mentioned above). Each stage contains multiple stacked 3D convolutional blocks.

[0035] Layered downsampling: Two downsampling operations (with convolutions with a stride of 2) are performed between the stages to gradually reduce the spatial and temporal resolution of the feature maps while increasing the channel dimension. This architecture can extract spatiotemporal features of multiple scales and different receptive fields from shallow to deep, thus taking into account both fine-grained details (shallow stages) and global abstract semantics (deep stages).

[0036] Interaction path: Previous The set of intermediate layer feature maps generated at each stage It is directly transmitted to the bidirectional hierarchical cross-attention collaborative fusion module, providing unpooled multi-scale detail information; Single-modal aggregation path: The high-level semantic feature map output from the last stage Transform into a single-modal identity representation intermediate layer feature map set The deepest feature map output after the forward propagation of the backbone network based on 3D-CNN in the last stage (i.e., the 4th stage) (which can be denoted as...) It contains the highest level of abstract global semantic information. This includes: first, preserving human body structural proportions through horizontal pooling; then, aggregating motion features across the entire gait cycle through temporal pooling, calculated using the following formula:

[0037] (1); in, This refers to the temporal pooling operation, which globally aggregates the entire sequence in the time dimension to extract the overall motion features that are not affected by the gait cycle length. This represents the horizontal pooling operation, which is used to divide and pool feature maps horizontally in the spatial dimension to preserve the structural proportion information of different parts of the human body.

[0038] Through the above layered design, the layered convolutional contour coding module can not only extract robust appearance prior features, but also provide a complete data foundation for subsequent multimodal interactions with skeleton branches by retaining intermediate layer features.

[0039] In step C, the specific implementation process of skeleton branch hierarchical spatiotemporal feature extraction is as follows: c1. Spatiotemporal embedding and local feature initialization; aligning the gait skeleton sequence... The input linear embedding layer is projected into a high-dimensional space, and a learnable spatial location embedding is superimposed. With time series embedding These are all learnable parameter tensors randomly generated during network initialization and continuously optimized through backpropagation during end-to-end training. Spatial location embedding Used to identify and supplement the spatial topological index of each key point in the skeleton; time series embedding This is used to identify the temporal order of the current frame in the sequence, compensating for the limitation of self-attention mechanisms in not being able to perceive order. Subsequently, an initial temporal convolutional network is used to capture the local instantaneous dynamics of limb swinging, generating the initial input feature map of the two-stream spatiotemporal transformer. :

[0040] (2); in, This represents a linear embedding layer mapping operation. Indicates temporal convolutional network operations; c2. Parallel timing and topological coding based on a dual-stream space-time converter; such as... Figure 2 As shown, a coding chain consisting of D layers of two-stream spatiotemporal transformers is constructed, with each layer modeled using a two-stream parallel structure: In each coding layer In the middle, input features The data is fed into the spatial and temporal branches in parallel for processing. In each coding layer of this invention, the input features are simultaneously and in parallel fed into two independent branches for feature extraction: Spatial Branch: This branch specifically performs multi-head self-attention (S-MHSA) on the "joint" dimension of the skeleton. Since the various key points of the human body have specific topological connections in physical space at the same time, this branch can effectively capture and establish long-range topological dependencies between different joints of the human body by combining S-MHSA with layer normalization (LN) and multilayer perceptron (MLP).

[0041] Temporal Branch: Simultaneously, this branch specifically performs multi-head self-attention (T-MHSA) on the "time" dimension of the gait sequence. Its role is to track the positional changes of each joint in a continuous frame sequence, capturing the global motion dynamics and temporal dependencies in the cross-frame evolution of the gait.

[0042] This parallel processing mechanism enables the network to extract structural and motion features of the skeleton simultaneously at the same feature level without interfering with each other.

[0043] Spatial branching is achieved through joint dimensions Multi-head self-attention (S-MHSA) is performed in conjunction with layer normalization (LN) and multilayer perceptron (MLP) to capture long-range topological relationships between key points in the human body, and the output spatial features are obtained. Represented as: (3); (4); This refers to the intermediate layer features of the spatial branch; it is the input feature. The result, processed by layer normalization (LN) and joint-dimensional multi-head self-attention (S-MHSA), and with residual connections, captures the long-range topological relationships (i.e., spatial structural information) between human joints, but has not yet undergone further abstraction through nonlinear mapping (MLP). It serves as a bridge for the spatial branch to transition from "raw attention computation" to "high-dimensional feature mapping".

[0044] At the same time, time branches, through the time dimension A multi-head self-attention mechanism is implemented to capture the global motion dynamics features of the gait sequence during cross-frame evolution and output the temporal features. ; This refers to the temporal coding function; it is a complete temporal processing module composed of layer normalization (LN), temporal multi-head self-attention (T-MHSA), multilayer perceptron (MLP), and residual connections (as shown in Equations (5) and (6)). Its function is to perform self-attention calculation on the temporal dimension T of the gait sequence, aiming to capture the global motion dynamics features and temporal dependencies of the gait skeleton in the evolution of continuous video frames.

[0045] The temporal branch has the same structure as the spatial branch, but its attention mechanism applies to the time dimension. ).

[0046] The intermediate features of the time branch are calculated as follows: (5); The final output features of the time branch are calculated as follows: (6); To achieve adaptive integration of spatiotemporal features, a dynamic fusion gating mechanism is introduced to integrate the aforementioned spatial features. With time characteristics Channel concatenation is performed, and a set of adaptive weights is obtained through a linear mapping layer and a Softmax function prediction. : (7); Finally, adaptive weights are used. Element-wise weighted summation is performed on the spatial and temporal features to obtain the final output features of the current coding layer. : (8); This allows for flexible adjustment of the weighting of structural configuration and motion patterns at each level based on the motion state of the input data.

[0047] c3. Multi-level feature pyramid and global representation generation: Intermediate features are extracted from different specified coding levels of the two-stream spatiotemporal transformer to construct a skeleton feature set with multi-scale semantic depth. These are intermediate features of the skeleton at different levels; used for subsequent intermodal collaborative fusion. It is a direct output of the skeleton branch hierarchical spatiotemporal transform coding module, and it is a series of high-dimensional intermediate feature tensors extracted from different stages of the skeleton branch hierarchical spatiotemporal transform coding module. Construction process: During the encoding process, the skeleton branch hierarchical spatiotemporal transform coding module does not only output the final result, but also... Each specified encoding stage will generate intermediate features at each stage. Extracted and summarized. Purpose: These features constitute a feature pyramid with multi-scale semantic depth, providing a hierarchical feature foundation for subsequent modality fusion in the bidirectional hierarchical cross-attention collaborative module. Simultaneously, the output of the last layer of the encoding chain... Perform global pooling to eliminate spatiotemporal redundancy and extract feature vectors that reflect the global identity information of the skeleton unimodality. That is, skeleton single-modal feature representation.

[0048] In step D, as Figure 3 As shown, the specific implementation process of bidirectional hierarchical cross-attention collaborative fusion is as follows: d1. Symmetric Spatiotemporal Pooling and Feature Projection: After obtaining the intermediate layer features of the two branches, the dimensionality between modalities is first unified through symmetric spatiotemporal pooling and feature projection operations; specifically, this includes: for the intermediate tensors of each level of the skeleton branch... intermediate tensors at each level of the skeleton branch It is a high-dimensional spatiotemporal feature encoding of skeleton branches at different stages, preserving the motion and topological details of the skeleton. First, temporal max pooling is performed to aggregate motion cues, and then global joint-dimensional average pooling is performed to generate compact skeleton feature vectors. : ;(9); Meanwhile, the feature map for the contour branch Based on max pooling in the execution time dimension, a generalized mean-level pyramid pooling operator is combined to capture spatial layout features and generate corresponding contour feature vectors. : (10); To eliminate channel dimension differences in subsequent interactions, a set of stage-specific linear projection operators are used to map the two modal features to a shared high-dimensional embedding space. and )Specific form: It is for each sampling level Independently learning linear fully connected layers; since the number of feature channels in the skeleton and contour branches may be inconsistent at different stages, these operators map them to a shared high-dimensional embedding space with uniform dimension, thereby eliminating channel differences between modalities and providing aligned input for subsequent attention calculations; thus obtaining the projected feature representation. and : ; (11); The projection results from each level are stitched together to form an intermediate projection feature vector representing the information of a single-modal level. and ; d2, bidirectional cross-modal attention interaction; such as Figure 4 As shown, after feature alignment is completed, at each sampling level Construct a bidirectional cross-modal attention interaction flow, utilizing a shared weight matrix. This facilitates deep information exchange between two heterogeneous features; in the attention mechanism, , , These represent the query projection matrix, key projection matrix, and value projection matrix, respectively. Their function is to linearly transform the input features to different representation spaces to compute attention weights and aggregate information. "Sharing" here refers to using the same set of projection matrices in the bidirectional attention interaction flow. Specifically, the same set is used in both the "structure-guided appearance enhancement flow" (with contour features as the query and skeleton features as the key and value) and the "appearance-guided structure enhancement flow" (with skeleton features as the query and contour features as the key and value). This parameter-sharing design forces the two modalities to interact within the same semantic space, promoting feature alignment and fusion. The query projection matrix, key projection matrix, and value projection matrix are learnable model parameters, obtained through the following methods:

[0049] Random initialization: At the start of training, these matrices are randomly initialized with a specific distribution such as a normal distribution or a uniform distribution.

[0050] End-to-end optimization: During model training, these parameters are continuously updated based on the loss function of the target task, such as gait recognition, through backpropagation and gradient descent, so that the model learns the optimal projection method.

[0051] The structure-guided appearance enhancement flow uses contour features as query terms and skeleton features as key and value terms to calculate appearance features incorporating structural constraints. : (12); in, This refers to the standard scaled dot product attention function, defined as: ;in, The dimension of the key vector is used for scaling to prevent the inner product from becoming too large and causing gradient saturation. It refers to the first The sampling level is determined by linear projection of silhouette features (through shared weights). The query matrix obtained represents "appearance information that needs to be enhanced", that is, appearance feature parts that are expected to be guided or constrained by the skeleton structure.

[0052] and : in the The hierarchy is formed by skeleton features sharing weights. and The projection yields a key matrix and a value matrix; together they provide "structured prior knowledge of motion": the keys are used to calculate similarity with the query, while the values ​​carry the structural information to be aggregated.

[0053] The appearance-guided structure enhancement flow, which is symmetrical to it, reverses the roles of the two, correcting the motion trajectory of the skeleton through appearance priors to obtain the structural features corrected by appearance priors. : (13); Appearance features Structural features These two enhancement features are then concatenated along the channel dimension and transformed into preliminary fused features for the current layer via a linear mapping layer. : (14); d3. Adaptive Feature Fusion and Multi-Level Representation Aggregation; To further improve the discriminative power of features, the system introduces an adaptive feature fusion mechanism to perform channel-level dynamic calibration on the initially fused features; such as... Figure 5 As shown, channel importance weights are predicted using a multilayer perceptron and activation function. : (15); in, The sigmoid function is represented by the mathematical form: Mapping the raw scores output by the Multilayer Perceptron (MLP) to... Intervals, generating channel importance weights Each channel's weight independently represents its contribution to the fused features at the current level. The closer the weight is to 1, the more important the channel is; the closer it is to 0, the more it should be suppressed. The output range of the Sigmoid is well-suited for use as soft attention weights, enabling channel-level dynamic calibration and differentiability, facilitating end-to-end training.

[0054] Weight Element-wise multiplication is performed with the original features to enhance key feature information, resulting in refined fused features. : (16); Ultimately, executing the hierarchical aggregation strategy will result in the aggregation of elements belonging to the previous... Refined features at each fine-grained stage are aggregated into mid-level fused features. : (17); in, It is a multilayer perceptron; specifically designed to process stitched multi-scale features and transform them into final mid-level fused features. It typically consists of several fully connected (Linear) layers, interspersed with non-linear activation functions (such as ReLU) and optional batch normalization layers. The input dimension is the total number of channels in the concatenated features, and the output dimension can be set according to the needs of subsequent tasks (e.g., set to a fixed-length feature vector). The concatenated high-dimensional features are reduced in dimensionality and abstracted, removing redundant information. Non-linear transformations enhance the discriminative power of the features, enabling features at different levels to interact fully. Multi-scale features are mapped to a unified feature space, facilitating their combined use with later-stage fusion features.

[0055] This indicates a concatenation operation; it typically refers to joining multiple tensors together along the feature dimension (channel dimension). In formula (17), it will... Refined features at a fine-grained stage (typically a shallow, high-resolution layer). By concatenating features along the channel dimension, a comprehensive feature containing multi-scale information is obtained. By fusing features from different receptive fields and resolutions, the mid-level features contain both detailed information (from the shallow layer) and semantic information (from the deep layer), thus enhancing the expressiveness of the features.

[0056] The first Features from each high-level semantic stage are independently transformed into late-stage fusion features, i.e., fusion output features. : (18); in, It is a late-stage fusion perceptron. It is used to independently process the highest level (the...). The refined characteristics of the layers Generate late-stage fusion features Similarly, a multilayer perceptron typically contains several fully connected layers and activation functions, with an input dimension of... The number of channels. Highest-level features typically possess the strongest semantic information but have lower spatial resolution. Through... Transforming it into late-stage fusion features can preserve high-level abstract information and supplement the global semantics that may be missing in mid-level features.

[0057] The above process fully realizes the interactive fusion of multimodal gait features in a multi-scale space.

[0058] In step E, the specific implementation process of overall feature aggregation is as follows: To integrate multi-source heterogeneous information and alleviate the gradient vanishing problem in deep networks, the system constructs an overall feature aggregation module to jointly aggregate the features output from each branch to form the final identity representation. The contour unimodal features obtained in step B are then used... The skeleton single-modal features obtained in step C and the intermediate projection features generated in step D. With fusion output features By splicing and aggregating the data, the final co-evolutionary feature representation is constructed. :

[0059] (19).

[0060] This feature represents a complete aggregation of appearance abstract information, structural motion information, and cross-modal interaction information enhanced by attention mechanisms, providing full-dimensional identity features for subsequent accurate identification.

[0061] In step F, the specific implementation process of training the multimodal gait recognition model includes: During the training phase, the final co-evolutionary feature representation obtained in step E is used as the core supervision object. The multimodal gait recognition model is trained end-to-end through the joint drive of triplet loss and cross-entropy loss. Specifically, triplet loss enhances the clustering of samples with the same identity and the discriminative power of samples with different identities through metric learning, thereby improving the feature's metric discriminative ability. Cross-entropy loss enhances the model's classification discriminative ability by establishing identity mappings. Furthermore, the loss function is directly applied to the feature set. For each member, calculate the total loss. : (20); in, and To balance the hyperparameters; It refers to each of the following: single-modal global features, mid-level fusion features, late-stage fusion features, and intermediate projection features; This refers to Cross-Entropy Loss, used in classification tasks. It enhances the class separability of features by mapping features to an identity class space and calculating the difference between the predicted probability and the true label. This refers to Triplet Loss. It is used in metric learning by constructing triplets (anchor, positive sample, negative sample) to bring similar samples closer together and widen the distance between dissimilar samples, thereby enhancing the discriminative and generalization abilities of features.

[0062] By directly constraining intermediate and fused features, the system can force each part of the network to generate representations with independent discriminative capabilities, thereby providing clear gradient signals for the entire encoder and fusion module.

[0063] In step G, multimodal cross-view gait recognition includes: The gait contour sequences and gait skeleton sequences from the registration set and query set are respectively input into a pre-trained multimodal gait recognition model. Through contour branch hierarchical spatiotemporal feature extraction, skeleton branch hierarchical spatiotemporal feature extraction, bidirectional hierarchical cross-attention collaborative fusion, and overall feature aggregation, the corresponding final identity feature representations are extracted. ; The similarity of features is measured by calculating the cosine or Euclidean distance between the features of the query sample and the features of all samples in the registration set. The samples are then ranked according to their similarity scores, and the identity of the registration sample with the highest score is used as the identification result. This completes the identity matching and confirmation of the query sample within the registration set, effectively solving the problem of multimodal gait recognition in complex scenarios such as cross-viewpoints and cross-clothing styles.

[0064] In this embodiment, to verify the effectiveness of the gait recognition method and system based on bidirectional cross-attention contour and skeleton fusion of the present invention, a publicly available gait dataset was used to train and test the method. During the data preprocessing stage, to ensure strict temporal consistency between the gait contour modality and the gait skeleton modality, a fixed-length sequence was constructed using the same sampling frame index for both modalities, maintaining a one-to-one correspondence at each time step. In terms of sampling strategy, a local continuous sampling method was adopted to retain short-term motion change information. For the CASIA-B dataset, the input sequence length was set to 60 frames; the contour image was normalized and adjusted to 64×44 pixels; the skeleton data adopted the standard COCO joint format. To further improve the model robustness, a triple sampler was used to construct batch samples during the training stage. Each batch contained 8 identities, and each identity sampled 8 gait sequences, with a total batch size of 64; the total number of iterations for overall model training was set to 60k. In terms of optimization strategy, the Adam optimizer is adopted, with an initial learning rate set to 1e-4. A multi-step learning rate decay adjustment strategy is adopted, and the learning rate is decayed by 0.1 times at a preset iteration milestone. The decay milestone is set to 40k iterations.

[0065] To verify the recognition performance of the method of this invention, a comparative experiment was conducted on the CASIA-B dataset, comparing the invention with existing gait recognition methods. A comprehensive performance comparison was performed on CASIA-B, and recognition results were reported under three walking conditions: normal walking (NM), carrying a package (BG), and wearing a coat (CL). Detailed Rank-1 accuracy for 11 viewpoints was also provided. Table 1 shows the comparison of Rank-1 accuracy between the method of this invention and existing methods on the CASIA-B dataset.

[0066] Table 1 Experimental results show that the method of this invention achieves an average Rank-1 accuracy of 97.1% on the CASIA-B dataset; specifically, the average recognition accuracy reaches 98.4% under normal walking conditions (NM), 97.5% under carrying a package conditions (BG), and 95.4% under wearing a coat conditions (CL). These results demonstrate that the bidirectional hierarchical cross-attention collaborative fusion mechanism proposed in this invention can effectively integrate the complementary advantages of contour information and skeleton information, thereby achieving higher recognition accuracy and stronger robustness under conditions of cross-viewpoint, appearance changes, and occlusion interference.

[0067] Example 3 A gait recognition method and system based on bidirectional cross-attention contour and skeleton fusion includes: The gait multimodal data acquisition and alignment processing module is configured to: acquire gait contour sequence data and gait skeleton sequence data, and perform preprocessing; the preprocessing includes contour sequence normalization, skeleton joint point sequence construction, and data alignment processing; wherein, the gait contour sequence and gait skeleton sequence are synchronously sampled and time-aligned to ensure the correspondence between the two modalities in the time dimension, providing alignment input for subsequent cross-modal hierarchical interactive fusion.

[0068] The contour branch hierarchical convolutional contour encoding module is configured to: extract multi-level spatiotemporal features of gait contour sequences; for the gait contour sequence processed by step A, hierarchical spatiotemporal features are extracted through the initial layer and multi-stage convolutional blocks to obtain intermediate contour features at different levels; among them, the final-level contour features are processed by horizontal pooling and temporal pooling to obtain the contour single-modal feature representation, i.e., contour features. The skeleton branch hierarchical spatiotemporal transformation coding module is configured to: model the spatial dependencies between human joints and the long-distance temporal dependencies between joints; for the processed gait skeleton sequence, perform skeleton spatiotemporal feature modeling to obtain intermediate skeleton features at different levels; and fuse the spatial branch output and temporal branch output through a dynamic weighted fusion mechanism; after the final-level skeleton features are processed by temporal pooling, the skeleton single-modal feature representation, i.e., skeleton features, is obtained. The bidirectional hierarchical cross-attention collaborative fusion module is configured to: perform multi-stage collaborative interaction and fusion on the obtained contour intermediate features and the obtained skeleton intermediate features; for contour intermediate features and skeleton intermediate features at different levels, compact representations are obtained through contour branch symmetric spatiotemporal pooling and skeleton branch symmetric spatiotemporal pooling, respectively, and then dimension mapping is performed through a parallel feature projector to obtain projected intermediate features, realizing a unified embedding representation of features at different levels and different modalities; then, bidirectional cross-modal attention interaction is performed on the contour features and skeleton features through cross-attention collaboration to obtain fused features at each level; finally, an adaptive feature fusion mechanism is used to adaptively aggregate the fused features at each level to obtain the fused output features; The overall feature aggregation module is configured to: jointly aggregate the features output by the contour branch, skeleton branch, and cross-modal collaborative fusion module to form the final identity representation; and splice or combine the contour single-modal features, skeleton single-modal features, projection intermediate features, and fusion output features to obtain the final collaborative co-evolutionary feature representation for subsequent training and recognition. The multimodal gait recognition model training module is configured to: use the final co-evolutionary feature representation as the pedestrian identity representation, and jointly train the multimodal gait recognition model using triplet loss and cross-entropy loss; wherein, triplet loss is used to enhance the feature measurement and discrimination ability, and cross-entropy loss is used to enhance the identity classification and discrimination ability, so as to jointly supervise the training process of the multimodal gait recognition model; The multimodal cross-view gait recognition module is configured to: input the gait contour sequences and gait skeleton sequences of the registration set and query set into the trained multimodal gait recognition model, extract the corresponding final identity feature representation; and complete the identity matching and confirmation of the query sample in the registration set by comparing the similarity between the query sample features and the registration sample features, thereby realizing multimodal cross-view gait recognition.

Claims

1. A gait recognition method based on contour and skeleton fusion using bidirectional cross-attention, characterized in that, Includes the following steps: A. Gait multimodal data acquisition and alignment processing; Gait contour sequence data and gait skeleton sequence data are acquired and preprocessed. The preprocessing includes contour sequence normalization, skeleton joint point sequence construction, and data alignment. Among them, the gait contour sequence and gait skeleton sequence are synchronously sampled and time-aligned. B. Contour branch hierarchical spatiotemporal feature extraction; Extract multi-level spatiotemporal features of gait contour sequences; for the gait contour sequences processed in step A, perform hierarchical spatiotemporal feature extraction through initial layers and multi-stage convolutional blocks to obtain intermediate contour features at different levels; among them, the final-level contour features are processed by horizontal pooling and temporal pooling to obtain the contour single-modal feature representation, i.e., contour features. C. Extraction of spatiotemporal features by skeleton branch hierarchical structure; Model the spatial dependencies between human joints and the long-distance temporal dependencies between joints; for the gait skeleton sequence processed in step A, perform spatiotemporal feature modeling of the skeleton to obtain intermediate skeleton features at different levels; and fuse the spatial branch output and temporal branch output through a dynamic weighted fusion mechanism; after the final-level skeleton features are processed by temporal pooling, the skeleton single-modal feature representation, i.e., skeleton features, is obtained. D. Two-way hierarchical cross-attention collaborative fusion; Multi-stage collaborative interaction and fusion are performed on the contour intermediate features obtained in step B and the skeleton intermediate features obtained in step C. For contour intermediate features and skeleton intermediate features at different levels, compact representations are obtained through contour branch symmetric spatiotemporal pooling and skeleton branch symmetric spatiotemporal pooling, respectively. Dimensional mapping is then performed through a parallel feature projector to obtain projected intermediate features, achieving a unified embedding representation of features at different levels and modalities. Subsequently, bidirectional cross-modal attention interaction is performed on contour features and skeleton features through cross-attention collaboration to obtain fused features at each level. Finally, an adaptive feature fusion mechanism is used to adaptively aggregate the fused features at each level to obtain the fused output features. E. Overall feature aggregation; The features output by the contour branch, skeleton branch, and cross-modal collaborative fusion module are jointly aggregated to form the final identity representation; the contour single-modal features obtained in step B, the skeleton single-modal features obtained in step C, the projection intermediate features in step D, and the fusion output features are spliced ​​or combined to obtain the final co-evolutionary feature representation, which is used for subsequent training and recognition. F. Training of multimodal gait recognition model; The final co-evolutionary feature representation obtained in step E is used as the pedestrian identity representation. The multimodal gait recognition model is jointly trained using triplet loss and cross-entropy loss. The triplet loss is used to enhance the feature measurement and discrimination ability, and the cross-entropy loss is used to enhance the identity classification and discrimination ability, so as to jointly supervise the training process of the multimodal gait recognition model. G. Multimodal cross-view gait recognition; The gait contour sequences and gait skeleton sequences of the registration set and query set are input into the trained multimodal gait recognition model to extract the corresponding final identity feature representations. By comparing the similarity between the features of the query sample and the features of the registration sample, the identity matching and confirmation of the query sample in the registration set is completed, thus realizing multimodal cross-view gait recognition.

2. The gait recognition method based on bidirectional cross-attention contour and skeleton fusion according to claim 1, characterized in that, In step A, the gait contour sequence is compared. With gait skeleton sequence The specific preprocessing process is as follows: Data alignment and sampling; acquiring the original gait walking sequence, and simultaneously retrieving and loading the corresponding contour feature sequence and skeleton pose sequence using the "triple" information of identity label, walking conditions, and viewpoint; adopting a fixed-length sequential sampling strategy: first, calculating the target sampling frame number. With frame skipping threshold The sampling span constituted, if the original sequence length If insufficient, then proceed according to The original index is expanded cyclically by a multiple of the original index; then, a starting offset is randomly selected within the expanded index field, and samples are extracted in a non-repeating sampling manner within this field window. Arrange the indexes in ascending order to generate a uniformly aligned index set. Ultimately, utilizing Perform synchronous slicing and normalization on the two branches to extract the contents. Frame contour sequence and skeleton sequence .

3. The gait recognition method based on bidirectional cross-attention contour and skeleton fusion according to claim 1, characterized in that, The multimodal gait recognition model includes a contour branch hierarchical convolutional contour encoding module, a skeleton branch hierarchical spatiotemporal transformation encoding module, a bidirectional hierarchical cross-attention collaborative fusion module, and an overall feature aggregation module; The contour branch hierarchical convolutional contour encoding module is used to extract multi-level spatiotemporal features of gait contour sequences; The skeleton branch hierarchical spatiotemporal transformation encoding module is used to model the spatial dependencies between human joints and the long-distance temporal dependencies of joints; the bidirectional hierarchical cross-attention collaborative fusion module is used to perform multi-stage collaborative interaction and fusion of the contour intermediate features obtained in step B and the skeleton intermediate features obtained in step C. The overall feature aggregation module is used to jointly aggregate the features output by the contour branch, skeleton branch and cross-modal collaborative fusion module to form the final identity representation.

4. The gait recognition method based on bidirectional cross-attention contour and skeleton fusion according to claim 1, characterized in that, In step B, the specific implementation process of contour branch hierarchical spatiotemporal feature extraction is as follows: b1. Multi-stage spatiotemporal convolutional coding of contours; the contour branch uses a 3D-CNN-based backbone network for hierarchical spatiotemporal feature extraction; for the input contour sequence First, basic spatiotemporal features are captured through an initial 3D convolutional layer, and then the code chain consists of four consecutive main stages. b2. Spatiotemporal co-sampling: Spatiotemporal downsampling operators are executed between stages to reduce spatial resolution and compress the temporal dimension. Construct a containing The feature pyramid has several levels, and the output is denoted as the set of intermediate layer feature maps. ; b3. Dual-path output mechanism; The contour branch hierarchical convolutional contour encoding module adopts a dual-path output strategy: Interaction path: Previous The set of intermediate layer feature maps generated at each stage It is directly transmitted to the bidirectional hierarchical cross-attention collaborative fusion module, providing unpooled multi-scale detail information; Single-modal aggregation path: The high-level semantic feature map output from the last stage Transform into a single-modal identity representation The process includes: First, horizontal pooling preserves the proportions of the human body structure; then, time pooling aggregates the motion features of the entire gait cycle. The calculation formula is as follows: （1）； in, This indicates a time pooling operation. This indicates a horizontal pooling operation.

5. The gait recognition method based on bidirectional cross-attention contour and skeleton fusion according to claim 1, characterized in that, In step C, the specific implementation process of skeleton branch hierarchical spatiotemporal feature extraction is as follows: c1. Spatiotemporal embedding and local feature initialization; aligning the gait skeleton sequence... The input linear embedding layer is projected into a high-dimensional space, and a learnable spatial location embedding is superimposed. With time series embedding Subsequently, an initial temporal convolutional network is used to capture the local instantaneous dynamics of limb swinging, generating the initial input feature map for the two-stream spatiotemporal transformer. : （2）； in, This represents a linear embedding layer mapping operation. Indicates temporal convolutional network operations; c2. Parallel timing and topological coding based on dual-stream space-time transformers; constructing a coding chain consisting of D layers of dual-stream space-time transformers, with each layer modeled using a dual-stream parallel structure: in each coding layer... In the middle, input features The data is fed into the spatial and temporal branches in parallel for processing. Spatial branching is achieved through joint dimensions Multi-head self-attention combined with layer normalization and multilayer perceptron is used to capture long-range topological relationships between key points in the human body, and the output spatial features are obtained. Represented as: （3）； （4）； This refers to the intermediate layer features of spatial branches; At the same time, time branches, through the time dimension A multi-head self-attention mechanism is implemented to capture the global motion dynamics features of the gait sequence during cross-frame evolution and output the temporal features. ; This refers to the time encoding function; The intermediate features of the time branch are calculated as follows: （5）； The final output features of the time branch are calculated as follows: （6）； Introducing a dynamic fusion gating mechanism to incorporate the aforementioned spatial features With time characteristics Channel concatenation is performed, and a set of adaptive weights is obtained through a linear mapping layer and a Softmax function prediction. : （7）； Finally, adaptive weights are used. Element-wise weighted summation is performed on the spatial and temporal features to obtain the final output features of the current coding layer. : （8）； c3. Multi-level feature pyramid and global representation generation: Intermediate features are extracted from different specified coding levels of the two-stream spatiotemporal transformer to construct a skeleton feature set with multi-scale semantic depth. This refers to the intermediate features of the skeleton at different levels; simultaneously, it refers to the output of the last layer of the encoding chain. Perform global pooling to eliminate spatiotemporal redundancy and extract feature vectors that reflect the global identity information of the skeleton unimodality. That is, skeleton single-modal feature representation.

6. The gait recognition method based on contour and skeleton fusion according to claim 1, characterized in that, In step D, the specific implementation process of bidirectional hierarchical cross-attention collaborative fusion is as follows: d1. Symmetric Spatiotemporal Pooling and Feature Projection: First, symmetric spatiotemporal pooling and feature projection operations are used to unify the dimensions between modalities; specifically, this includes: for the intermediate tensors of each level of the skeleton branch. First, temporal max pooling is performed to aggregate motion cues. Then, global joint-level average pooling is performed to generate compact skeleton feature vectors. : ； （9）； Meanwhile, the feature map for the contour branch Based on max pooling in the execution time dimension, a generalized mean-level pyramid pooling operator is combined to capture spatial layout features and generate corresponding contour feature vectors. : （10）； A set of stage-specific linear projection operators are used to map the two modal features to a shared high-dimensional embedding space, resulting in projected feature representations. and : ； （11） ； The projection results from each level are stitched together to form an intermediate projection feature vector representing the information of a single-modal level. and ; d2. Bidirectional cross-modal attention interaction; after feature alignment is completed, at each sampling level... Construct a bidirectional cross-modal attention interaction flow, utilizing a shared weight matrix. This facilitates deep information exchange between two heterogeneous features; in the attention mechanism, , , These represent the query projection matrix, key projection matrix, and value projection matrix, respectively. The structure-guided appearance enhancement flow uses contour features as the query term and skeleton features as the key and value terms to calculate the appearance features incorporating structural constraints. : （12）； in, This refers to the standard scaled dot product attention function, defined as: ;in, The dimension of the key vector is used for scaling to prevent the inner product from becoming too large and causing gradient saturation. It refers to the first Sampling level, the query matrix obtained by linear projection of contour features; and : in the The hierarchy is determined by the skeleton features sharing weights. and The key matrix and value matrix obtained by projection; The appearance-guided structure enhancement flow, which is symmetrical to it, reverses the roles of the two, correcting the motion trajectory of the skeleton through appearance priors to obtain the structural features corrected by appearance priors. : （13）； Appearance features Structural features These two enhancement features are then concatenated along the channel dimension and transformed into preliminary fused features for the current layer via a linear mapping layer. : （14）； d3. Adaptive Feature Fusion and Multi-Level Representation Aggregation: An adaptive feature fusion mechanism is introduced to perform channel-level dynamic calibration of the initially fused features; channel importance weights are predicted using a multilayer perceptron and activation function. : （15）； in, The sigmoid function is represented by the mathematical form: ; Weight Element-wise multiplication is performed with the original features to enhance key feature information, resulting in refined fused features. : （16）； Ultimately, executing the hierarchical aggregation strategy will result in the aggregation of elements belonging to the previous... Refined features at each fine-grained stage are aggregated into mid-level fused features. : （17）； in, It is a multilayer perceptron; Indicates a splicing operation; The first Features from each high-level semantic stage are independently transformed into late-stage fusion features, i.e., fusion output features. : （18）； in, It is a late-stage fusion sensor.

7. The gait recognition method based on bidirectional cross-attention contour and skeleton fusion according to claim 1, characterized in that, In step E, the specific implementation process of overall feature aggregation is as follows: The contour single-modal features obtained in step B The skeleton single-modal features obtained in step C and the intermediate projection features generated in step D. With fusion output features By splicing and aggregating the data, the final co-evolutionary feature representation is constructed. : （19）。 8. The gait recognition method based on bidirectional cross-attention contour and skeleton fusion according to claim 1, characterized in that, In step F, the specific implementation process of training the multimodal gait recognition model includes: During the training phase, the final co-evolutionary feature representation obtained in step E is used as the core supervision object. The multimodal gait recognition model is trained end-to-end through the joint drive of triplet loss and cross-entropy loss. Specifically, triplet loss enhances the clustering of samples with the same identity and the discriminative power of samples with different identities through metric learning, thereby improving the feature's metric discriminative ability. Cross-entropy loss enhances the model's classification discriminative ability by establishing identity mappings. Furthermore, the loss function is directly applied to the feature set. For each member, calculate the total loss. : （20）； in, and To balance the hyperparameters; It refers to each of the following: single-modal global features, mid-level fusion features, late-stage fusion features, and intermediate projection features; This refers to cross-entropy loss; This refers to the loss of the triplet.

9. The gait recognition method based on bidirectional cross-attention contour and skeleton fusion according to any one of claims 1-8, characterized in that, In step G, multimodal cross-view gait recognition includes: The gait contour sequences and gait skeleton sequences from the registration set and query set are respectively input into a pre-trained multimodal gait recognition model. Through contour branch hierarchical spatiotemporal feature extraction, skeleton branch hierarchical spatiotemporal feature extraction, bidirectional hierarchical cross-attention collaborative fusion, and overall feature aggregation, the corresponding final identity feature representations are extracted. ; Feature similarity is measured by calculating the cosine or Euclidean distance between the features of the query sample and the features of all samples in the registration set; the samples are then sorted according to their similarity scores, and the identity of the registration sample with the highest score is taken as the identification result.

10. A gait recognition method and system based on bidirectional cross-attention contour and skeleton fusion, characterized in that, include: The gait multimodal data acquisition and alignment processing module is configured to: acquire gait contour sequence data and gait skeleton sequence data, and perform preprocessing; the preprocessing includes contour sequence normalization, skeleton joint point sequence construction, and data alignment processing; wherein, the gait contour sequence and gait skeleton sequence are synchronously sampled and time-aligned. The contour branch hierarchical convolutional contour encoding module is configured to: extract multi-level spatiotemporal features of gait contour sequences; for the gait contour sequence processed by step A, hierarchical spatiotemporal features are extracted through the initial layer and multi-stage convolutional blocks to obtain intermediate contour features at different levels; among them, the final-level contour features are processed by horizontal pooling and temporal pooling to obtain the contour single-modal feature representation, i.e., contour features. The skeleton branch hierarchical spatiotemporal transformation coding module is configured to: model the spatial dependencies between human joints and the long-distance temporal dependencies between joints; for the processed gait skeleton sequence, perform skeleton spatiotemporal feature modeling to obtain intermediate skeleton features at different levels; and fuse the spatial branch output and temporal branch output through a dynamic weighted fusion mechanism; after the final-level skeleton features are processed by temporal pooling, the skeleton single-modal feature representation, i.e., skeleton features, is obtained. The bidirectional hierarchical cross-attention collaborative fusion module is configured to: perform multi-stage collaborative interaction and fusion on the obtained contour intermediate features and the obtained skeleton intermediate features; for contour intermediate features and skeleton intermediate features at different levels, compact representations are obtained through contour branch symmetric spatiotemporal pooling and skeleton branch symmetric spatiotemporal pooling, respectively, and then dimension mapping is performed through a parallel feature projector to obtain projected intermediate features, realizing a unified embedding representation of features at different levels and different modalities; then, bidirectional cross-modal attention interaction is performed on the contour features and skeleton features through cross-attention collaboration to obtain fused features at each level; finally, an adaptive feature fusion mechanism is used to adaptively aggregate the fused features at each level to obtain the fused output features; The overall feature aggregation module is configured to: jointly aggregate the features output by the contour branch, skeleton branch, and cross-modal collaborative fusion module to form the final identity representation; and splice or combine the contour single-modal features, skeleton single-modal features, projection intermediate features, and fusion output features to obtain the final collaborative co-evolutionary feature representation for subsequent training and recognition. The multimodal gait recognition model training module is configured to: use the final co-evolutionary feature representation as the pedestrian identity representation, and jointly train the multimodal gait recognition model using triplet loss and cross-entropy loss; wherein, triplet loss is used to enhance the feature measurement and discrimination ability, and cross-entropy loss is used to enhance the identity classification and discrimination ability, so as to jointly supervise the training process of the multimodal gait recognition model; The multimodal cross-view gait recognition module is configured to: input the gait contour sequences and gait skeleton sequences of the registration set and query set into the trained multimodal gait recognition model, extract the corresponding final identity feature representation; and complete the identity matching and confirmation of the query sample in the registration set by comparing the similarity between the query sample features and the registration sample features, thereby realizing multimodal cross-view gait recognition.

Images