A gait recognition method based on unified space-time feature extraction

Abstract

The present application relates to the technical field of computer vision, pattern recognition and biometric identification, and particularly relates to a gait recognition method based on unified spatiotemporal feature extraction. The method comprises: obtaining gait input data of an object to be identified and performing preprocessing; constructing a gait recognition model comprising an information extraction and low-dimensional feature fusion module, a unified spatiotemporal feature extraction module, a pooling module and an identification output module; obtaining contour graph feature representation or contour graph and skeleton graph fusion feature representation through the information extraction and low-dimensional feature fusion module; sequentially extracting spatial attention, temporal channel attention and common gait features between different samples through the unified spatiotemporal feature extraction module to obtain enhanced spatiotemporal features; obtaining sequence-level gait representation through the pooling module; training the model through the identification output module in combination with a triplet loss function and a cross-entropy loss function to determine a trained gait recognition model; inputting the object to be identified into the trained gait recognition model and performing sequence-level matching on the output sequence-level gait representation. The method can improve the discrimination ability, robustness and generalization performance of gait recognition in complex scenarios.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, pattern recognition, and biometric recognition, specifically to a gait recognition method based on unified spatiotemporal feature extraction. Background Technology

[0002] Gait recognition, which is the process of identifying individuals based on walking patterns, is a non-contact, long-distance, and difficult-to-disguise biometric identification method with significant application value in various practical scenarios such as intelligent security, identity authentication, public safety, and behavior analysis. Compared with methods that only focus on spatial feature extraction, gait recognition methods that consider both spatial patterns and temporal dynamic information are more conducive to improving feature representation capabilities. However, existing related methods still have shortcomings in terms of unified spatiotemporal modeling.

[0003] Existing gait recognition methods mainly include appearance-based gait recognition methods, model-based gait recognition methods, and multimodal gait recognition methods. Appearance-based methods typically extract gait features using human contour images or RGB images, possessing strong morphological representation capabilities. Model-based methods typically use human skeletons, key points, or simplified human structures for modeling, exhibiting good structural robustness. Multimodal methods attempt to combine different modal information such as contours and skeletons to improve gait recognition performance and generalization ability.

[0004] However, existing gait recognition technologies still have certain limitations: First, most existing methods focus on spatial feature extraction during feature modeling, neglecting the mining of temporal dynamic information contained in gait sequences, making it difficult to fully reflect the dynamic changes of the human body during continuous walking; Second, existing methods typically employ traditional deep network structures or relatively simple spatiotemporal feature extraction modules, making it difficult to effectively capture fine-grained spatial and temporal patterns in complex gait sequences, resulting in limited adaptability of the model to complex scenarios; Third, in real-world environments, gait data is often affected by factors such as changes in viewpoint, occlusion, background interference, clothing changes, and changes in carried items. If the model lacks a unified spatiotemporal feature extraction capability, it is prone to over-reliance on static spatial appearance information, thereby affecting recognition accuracy and generalization performance; Fourth, although some existing methods introduce multimodal inputs, the extraction and fusion of spatial and temporal features are still relatively separate, and a unified, stable, and efficient spatiotemporal feature modeling framework has not yet been formed. Summary of the Invention

[0005] To address the technical problems of existing gait recognition methods, such as overemphasis on spatial feature extraction, insufficient utilization of temporal dynamic information, difficulty in effectively modeling complex spatiotemporal patterns, and limited robustness and generalization ability in complex scenarios, this invention aims to provide a gait recognition method based on unified spatiotemporal feature extraction. The specific technical solution adopted is as follows:

[0006] Acquire the gait input data of the object to be identified and perform preprocessing;

[0007] A gait recognition model is constructed, which includes an information extraction and low-dimensional feature fusion module, a unified spatiotemporal feature extraction module, a pooling module, and a recognition output module.

[0008] Based on the information extraction and low-dimensional feature fusion module, feature representation is constructed on the preprocessed gait input data to obtain contour map feature representation or contour map and skeleton map fusion feature representation;

[0009] Based on the unified spatiotemporal feature extraction module, feature representation is processed, spatial attention and temporal channel attention are extracted sequentially, and common gait features between different samples are extracted to obtain enhanced spatiotemporal features;

[0010] Based on the pooling module, the enhanced spatiotemporal features are sequentially subjected to ensemble pooling in the temporal dimension and horizontal pyramid pooling in the spatial dimension to obtain a sequence-level gait representation.

[0011] The sequence-level gait representation is input into the recognition output module. Based on the embedded features output by the feature mapping unit and the identity category prediction results output by the classifier, the gait recognition model is trained using a loss function. The parameters of the gait recognition model are adjusted to determine the trained gait recognition model.

[0012] Obtain the contour map and original skeletal coordinates to be identified, and input them into the trained gait recognition model. Perform sequence-level matching on the output sequence-level gait representation.

[0013] Preferably, the gait input data of the object to be identified is acquired and preprocessed, including:

[0014] Gait input data of the object to be identified is obtained. The gait input data includes a binary contour map sequence and an original skeletal coordinate sequence constructed from key points of the human skeleton. The original skeletal coordinate sequence is preprocessed.

[0015] Coordinate normalization is performed based on the original skeletal coordinate sequence. The coordinate normalization includes center normalization and scale normalization. Center normalization is used to move the center of gravity of the human joint coordinates to a fixed reference point, and scale normalization is used to standardize the human height to a fixed range.

[0016] Joint and limb heatmaps are created based on the coordinate normalization results. The joint heatmap is used to characterize the positional distribution of each joint point and is obtained by superimposing the Gaussian responses corresponding to each joint. The corresponding calculation formula is as follows:

[0017]

[0018] in, Represents the pixel coordinates in the heatmap; This indicates the joint heatmap at the pixel level. The response value at the location; Indicates a joint index; Represents the normalized i-th The coordinates of each joint position; Indicates the first Confidence scores for each joint; Parameters representing the Gaussian distribution;

[0019] Limb heatmaps are used to represent the connectivity between adjacent joints. They are generated by encoding the distance from each pixel in the image to the nearest limb. The corresponding calculation formula is as follows:

[0020]

[0021] in, This indicates the limb heatmap at the pixel level. The response value at the location; Indicates a limb index; Indicates based on the first The line segments corresponding to each limb; Represents pixels To limbs The distance; Indicates the formation of the first The two endpoint indices of the bar; Indicates the first The confidence score for each limb is determined by taking the smaller confidence score of the two endpoint joints. Parameters representing the Gaussian distribution;

[0022] The joint and limb diagrams are stacked into a two-channel skeleton diagram. The blank area is centrally cropped using a subject-centered alignment. After cropping, the skeleton diagram size is adjusted according to the input size to obtain the final skeleton diagram.

[0023] The preprocessed gait input data is processed to unify the length and size. One frame is sampled from each original sequence every preset number of frames in chronological order, and a fixed number of gait frames are extracted as a complete sample.

[0024] Preferably, the preprocessed gait input data is used to construct a feature representation based on the information extraction and low-dimensional feature fusion module to obtain a contour map feature representation or a fusion feature representation of the contour map and skeleton map, including:

[0025] The preprocessed gait input data is input into the information extraction module, which extracts the contour features corresponding to the binary contour map modality and the skeletal features corresponding to the skeletal map modality, respectively. When a single modality input is used, the contour map feature representation is output. When a multimodal input is used, the contour features and skeletal features are mapped to the same low-dimensional feature space, and dynamic weighted fusion is performed based on the low-dimensional feature fusion module through an attention mechanism to obtain the fused feature representation of the contour map and skeletal map.

[0026] Preferably, the feature representation is processed based on the unified spatiotemporal feature extraction module, spatial attention and temporal channel attention are extracted sequentially, and common gait features between different samples are extracted to obtain enhanced spatiotemporal features, including:

[0027] The unified spatiotemporal feature extraction module includes a first two-dimensional convolutional unit, a spatial attention unit, a dimension restoration unit, a three-dimensional convolutional unit, a temporal channel attention unit, a second two-dimensional convolutional unit, and a residual connection unit;

[0028] The dimensionality is reshaped based on the input feature map, and then sequentially input into the first two-dimensional convolutional unit and the spatial attention unit to obtain the spatially enhanced features.

[0029] After restoration based on spatial augmented feature maps, the data is input into a 3D convolutional unit and a temporal channel attention unit, and common gait features between different samples are extracted to obtain preliminary spatiotemporal augmented features.

[0030] The initial spatiotemporal enhancement features are refined based on the second two-dimensional convolutional unit, and the output features are residually connected with the input feature map to obtain the final enhanced spatiotemporal features.

[0031] Preferably, the input feature map is reshaped in dimension and then sequentially input into a first two-dimensional convolutional unit and a spatial attention unit to obtain spatially enhanced features, including:

[0032] Let the input feature map be denoted as Its dimensions are ,in, Indicates batch dimension, Indicates the number of channels. Indicates the number of time frames. and These represent the height and width of the feature map, respectively; the input feature map is rearranged along the batch and time dimensions to reshape it into... Its dimensions are ;

[0033] The reshaped feature map is input into the first two-dimensional convolutional unit for spatial convolution operation to extract local spatial response features and provide basic spatial representation for subsequent spatial attention units.

[0034] Global average pooling and global max pooling are performed on the reshaped feature maps respectively. Statistical information is aggregated in the channel dimension to obtain average pooling feature maps and max pooling feature maps. The average pooling feature maps and max pooling feature maps are then concatenated in the channel dimension.

[0035] The concatenated feature map is input into a convolutional layer with a kernel size of 7×7 to obtain a spatial attention weight map. The spatial attention weight map is then normalized using the Sigmoid function, and the corresponding calculation formula is as follows:

[0036]

[0037] in, This represents the feature map obtained by performing global average pooling on the reconstructed feature map; This represents the feature map obtained by performing global max pooling on the reshaped feature map; Indicates a splicing operation; This represents the Sigmoid normalization function; Indicates the kernel size as Convolution operation; Represents the spatial attention weight map;

[0038] The reshaped feature map is then element-wise weighted based on the normalized spatial attention weight map to obtain the spatially augmented feature map. The corresponding calculation formula is as follows:

[0039]

[0040] in, This represents the reshaped input feature map; Indicates position Spatial attention weights at locations; The feature map is represented by the augmented representation space.

[0041] Preferably, after restoration based on the spatial augmentation feature map, the data is input into a 3D convolutional unit and a temporal channel attention unit, and common gait features between different samples are extracted to obtain preliminary spatiotemporal augmentation features, including:

[0042] The spatially augmented feature maps are restored to their original temporal arrangement according to the batch and time dimensions to obtain the restored spatiotemporal feature representation.

[0043] The recovered spatiotemporal feature representation is input into a three-dimensional convolutional unit, and the convolutional kernel slides synchronously along the time and space dimensions to extract dynamic change features across time frames. Based on the feature map output by the three-dimensional convolutional unit, a spatiotemporal feature representation containing the initial spatiotemporal dependency is obtained.

[0044] The spatiotemporal features output by the 3D convolutional unit are subjected to global average pooling in the spatial dimension to obtain a temporal feature vector that retains the global context information of each time step. The corresponding calculation formula is as follows:

[0045]

[0046] in, Represents the spatiotemporal feature map output by a 3D convolutional unit; This indicates global average pooling along the spatial dimension; Indicates the height and width of the feature map; Indicates a spatial dimension index; This represents the time-series feature vector after pooling;

[0047] The temporal feature vector is input into the first convolutional layer for channel compression to obtain intermediate temporal features. The corresponding calculation formula is as follows:

[0048]

[0049] in, and These represent the weight parameters and bias parameters of the first convolutional layer, respectively. This represents the intermediate timing characteristics after compression;

[0050] The intermediate temporal features are input into the common gait feature extraction unit to model the common gait patterns of different samples in the temporal dimension, thereby obtaining common gait enhancement features. These common gait enhancement features are then input into the second convolutional layer for dimensionality restoration, and temporal channel attention weights are generated using the Sigmoid function. These temporal channel attention weights are then used to recalibrate the initial spatiotemporal features element-wise, resulting in a temporally enhanced feature map. The corresponding calculation formula is as follows:

[0051]

[0052] in, This represents a common gait feature extraction unit, preferably a state space sequence modeling unit, and more preferably a Mamba2 module; This represents the initial spatiotemporal feature map output by the 3D convolutional unit; and These represent the weight parameters and bias parameters of the second convolutional layer or linear mapping layer, respectively. This represents the Sigmoid normalization function; This represents the feature map after temporal enhancement.

[0053] Preferably, the preliminary spatiotemporal enhancement features are refined based on the second two-dimensional convolutional unit, and the output features are residually concatenated with the input feature map to obtain the final enhanced spatiotemporal features, including:

[0054] The temporally enhanced feature map is input into the second two-dimensional convolutional unit to perform further nonlinear mapping and local structure refinement on the spatiotemporally fused feature map, enhance the feature recombination ability between different channels and the local expression ability between adjacent spatial regions, and output the refined spatiotemporal feature map after processing by the second two-dimensional convolutional unit.

[0055] The refined spatiotemporal feature map is added element-wise to the original feature map before the input spatiotemporal feature extraction module to output the final enhanced spatiotemporal feature.

[0056] Preferably, the enhanced spatiotemporal features are sequentially subjected to ensemble pooling in the temporal dimension and horizontal pyramid pooling in the spatial dimension based on the pooling module to obtain a sequence-level gait representation, including:

[0057] The input enhanced spatiotemporal feature sequence is regarded as a set of gait features composed of features from multiple time steps, denoted as:

[0058]

[0059] in, Indicates the first The feature vector corresponding to the frame, This indicates the length of the gait sequence; different sequence lengths are allowed for different gait samples. Represents the set of gait features;

[0060] The gait feature set is aggregated along the time dimension using a set pooling module. A max-pooling strategy is then applied to aggregate the temporal features, extracting the most significant features from the entire gait sequence. This transforms the variable-length gait sequence into a fixed-dimensional sequence-level gait representation. The corresponding calculation formula is as follows:

[0061]

[0062] in, Indicates the first The feature vector corresponding to the frame; Indicates the length of the gait sequence; This represents the fixed-dimensional feature representation after pooling.

[0063] The feature map after ensemble pooling is divided into layers along the height dimension to obtain multiple horizontal stripe regions. Each horizontal stripe corresponds to a local region of the human body at different height positions, used to represent local gait features in the head and shoulder region, trunk region, leg region, or foot region, respectively. The corresponding calculation formula is as follows:

[0064]

[0065] in, This represents the feature map after pooling; and Indicates the first The vertical boundary of a horizontal stripe; This represents the total number of local feature maps corresponding to the horizontal stripes; Indicates the first Local feature map corresponding to each horizontal stripe;

[0066] Pooling is applied to each horizontal stripe region to integrate the spatial features within that local region, resulting in a corresponding local feature vector. This partitioning and pooling operation is repeated at different granularities. All local feature vectors are then concatenated and flattened to obtain the final sequence-level gait representation, calculated using the following formula:

[0067]

[0068] in, Indicates a splicing operation; Indicates the flattening operation; This represents the pooling function, preferably max pooling or average pooling. This represents the final sequence-level gait representation.

[0069] Preferably, the sequence-level gait representation is input into the recognition output module. Based on the embedded features output by the feature mapping unit and the identity category prediction results output by the classifier, the gait recognition model is trained using a loss function. The parameters of the gait recognition model are adjusted to determine the trained gait recognition model, including:

[0070] The sequence-level gait representation is input into the feature mapping unit to obtain the embedded features used for inter-sample distance calculation. The embedded features are then input into the batch normalization neck network to normalize and reshape the feature distribution. Finally, the processing result is input into the classifier.

[0071] Triple sample pairs are constructed based on the embedded features output by the feature mapping unit, and anchor samples are selected from a batch of training samples. Positive samples belonging to the same identity category as the anchor sample. And negative samples that belong to a different identity category than the anchor sample. Calculate the distance between the anchor point sample and the positive sample, and the distance between the anchor point sample and the negative sample;

[0072] Based on the preset margin parameter constraining the distance relationship between positive and negative samples, the distance between the anchor sample and the positive sample is made smaller than the distance between the anchor sample and the negative sample. The loss values ​​of all triplet sample pairs are summed or averaged to obtain the triplet loss value. The corresponding calculation formula is as follows:

[0073]

[0074] in, Indicates the number of triplet sample pairs; Indicates sample With sample The distance metric function between them, wherein the distance metric function is Euclidean distance; Indicates the first Anchor sample embedding features in each triplet; Indicates the first Positive sample embedding features in triplet; Indicates the first Negative sample embedding features in triplet; The margin parameter is described. The value is 0.2; Represents the triplet loss function;

[0075] The classification error of a single sample is calculated based on the predicted probability of the identity category output by the classifier and the real identity label. The average classification error of all samples in a batch is then calculated to obtain the cross-entropy loss value. The corresponding calculation formula is as follows:

[0076]

[0077] in, Indicates the number of samples in the batch; Indicates the first The predicted probability that a sample belongs to its true category; Represents the cross-entropy loss function;

[0078] The total loss function for model training is calculated by summing the triplet loss function and the cross-entropy loss function together. The corresponding calculation formula is as follows:

[0079]

[0080] in, Represents the total loss function. Represents the triplet loss function. This represents the cross-entropy loss function.

[0081] The present invention has the following beneficial effects:

[0082] 1. This invention constructs a unified spatiotemporal feature extraction structure that includes spatial attention, temporal channel attention, and common gait pattern extraction modules. This structure can simultaneously enhance the spatial representation and temporal dynamic modeling capabilities of gait. Compared with existing methods that only focus on spatial features or use simple temporal modeling, this invention can more fully characterize the discriminative information in gait sequences.

[0083] 2. This invention introduces a sequence modeling module into the time channel attention module to extract common gait patterns from multiple time steps or even multiple samples. This effectively suppresses the influence of clothing changes, changes in carried items, and complex background interference, thereby improving the robustness of the model. Attached Figure Description

[0084] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0085] Figure 1 A schematic diagram of the overall framework of a gait recognition method based on unified spatiotemporal feature extraction provided in an embodiment of the present invention;

[0086] Figure 2 This is a schematic diagram illustrating the process of generating a human skeleton heatmap using a gait recognition method based on unified spatiotemporal feature extraction, as provided in an embodiment of the present invention.

[0087] Figure 3 This is a schematic diagram of the unified spatiotemporal feature extraction module structure of a gait recognition method based on unified spatiotemporal feature extraction provided in an embodiment of the present invention;

[0088] Figure 4 A schematic diagram of the temporal channel attention unit structure of a gait recognition method based on unified spatiotemporal feature extraction provided in an embodiment of the present invention;

[0089] Figure 5 A t-SNE visualization of a contour-based gait recognition method based on unified spatiotemporal feature extraction provided in an embodiment of the present invention on the CASIA-B dataset;

[0090] Figure 6 A t-SNE visualization of a contour-based gait recognition method based on unified spatiotemporal feature extraction provided in an embodiment of the present invention on the Gait3D dataset;

[0091] Figure 7 This is a t-SNE visualization of a multimodal model of a gait recognition method based on unified spatiotemporal feature extraction provided in an embodiment of the present invention on the Gait3D dataset. Detailed Implementation

[0092] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a gait recognition method based on unified spatiotemporal feature extraction proposed according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0093] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0094] The following description, in conjunction with the accompanying drawings, details a specific scheme for a gait recognition method based on unified spatiotemporal feature extraction provided by the present invention.

[0095] Please see Figure 1 The first embodiment of the present invention provides a gait recognition method based on unified spatiotemporal feature extraction, the method comprising:

[0096] Step S1: Obtain the gait input data of the object to be identified and perform preprocessing;

[0097] Step S2: Construct a gait recognition model, which includes an information extraction and low-dimensional feature fusion module, a unified spatiotemporal feature extraction module (STBlock), a pooling module (TP, HPP), and a recognition output module;

[0098] Step S3: Based on the information extraction and low-dimensional feature fusion module, construct feature representations for the preprocessed gait input data to obtain contour map feature representations or contour map and skeleton map fusion feature representations;

[0099] Step S4: Based on the unified spatiotemporal feature extraction module, the feature representation is processed, spatial attention and temporal channel attention are extracted in sequence, and common gait features between different samples are extracted to obtain the enhanced spatiotemporal features;

[0100] Step S5: Based on the pooling module, perform sequential ensemble pooling in the temporal dimension and horizontal pyramid pooling in the spatial dimension on the enhanced spatiotemporal features to obtain sequence-level gait representation;

[0101] Step S6: Input the sequence-level gait representation into the recognition output module. Based on the embedded features output by the feature mapping unit and the identity category prediction results output by the classifier, train the gait recognition model using the loss function, adjust the gait recognition model parameters, and determine the trained gait recognition model.

[0102] Step S7: Obtain the contour map and original skeletal coordinates to be identified, and input them into the trained gait recognition model. Perform sequence-level matching on the output sequence-level gait representation.

[0103] To better illustrate the gait recognition method based on unified spatiotemporal feature extraction provided by this invention, the overall process, module structure, and working mechanism of the method are further explained below with reference to the accompanying drawings. Since gait recognition relies not only on spatial information such as human contours and skeletal structure, but also on the periodic dynamic changes exhibited by the human body during continuous walking, this invention, from the perspective of unified spatiotemporal modeling, first preprocesses the gait input data of the object to be recognized; then, it constructs a gait recognition model including an information extraction and low-dimensional feature fusion module, a unified spatiotemporal feature extraction module, a pooling module, and a recognition output module; further, it utilizes the information extraction and low-dimensional feature fusion module to obtain contour map feature representations or contour map and skeletal map fusion feature representations, and sequentially completes spatial attention enhancement, temporal channel attention enhancement, and common gait feature extraction through the unified spatiotemporal feature extraction module to obtain enhanced spatiotemporal features; then, the pooling module converts the enhanced spatiotemporal features into sequence-level gait representations, and combines them with the recognition output module to complete model training and identity recognition. Through the above methods, the present invention can enhance the collaborative modeling capability of gait spatial structure information and temporal dynamic information within a unified framework, thereby improving the accuracy, robustness and generalization performance of gait recognition in complex scenarios.

[0104] Further, step S1 includes:

[0105] Step S11: Obtain gait input data of the object to be identified. The gait input data includes a binary contour map sequence and an original skeletal coordinate sequence constructed from key points of the human skeleton. The original skeletal coordinate sequence is preprocessed.

[0106] It can be noted that, in order to improve the stability of the subsequent model training process and the consistency of the processing flow between different modalities, this embodiment first performs unified preprocessing on the input gait data, especially converting the original skeletal coordinates into a skeletal heatmap representation suitable for convolutional network processing, so as to enhance the expressive ability of human structural relationships.

[0107] Optionally, the CASIA-B dataset is a multi-view gait dataset collected under controlled conditions, containing gait sequences under different viewpoints, clothing conditions, and changes in the objects carried, which is suitable for verifying the recognition ability of the gait recognition method in standard controlled scenarios; the Gait3D dataset is a gait dataset for real complex scenarios, containing complex backgrounds, occlusions, lighting changes, and multiple viewpoint conditions, which is suitable for verifying the robustness and generalization ability of the gait recognition method in real environments.

[0108] Furthermore, when the original gait data lacks human posture modal information, human key point coordinates can be generated based on the original gait image using a human posture estimation algorithm to complete the original skeletal coordinate sequence. Preferably, in this embodiment, the HRNet high-precision posture estimation algorithm is used to generate human key point coordinates. The generated key point coordinates can be 17 key point coordinates conforming to the COCO17 format, and can be further used for subsequent skeletal heatmap generation.

[0109] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the process of generating a human skeleton heatmap according to an embodiment of the present invention; wherein, Figure 2 (a) is the visualization result of the original human posture coordinates. The red numbers represent the key point numbers. The key point numbers preferably conform to the COCO17 format definition. The color intensity of different key points indicates the confidence level of the corresponding key points. Figure 2 (b) is a limb heatmap generated based on the normalized key point connections; Figure 2 (c) is a joint heatmap generated based on the normalized key point locations; Figure 2 (d) A dual-channel skeletal map formed by stacking limb heat maps and joint heat maps; the process includes:

[0110] Step S12: Perform coordinate normalization based on the original skeletal coordinate sequence. The coordinate normalization includes center normalization and scale normalization. Center normalization is used to move the center of gravity of the human joint coordinates to a fixed reference point, and scale normalization is used to standardize the human height to a fixed range.

[0111] Specifically, given the human joint coordinates as ,in, Indicates the first The coordinates of each joint. Indicates the first The confidence level corresponding to each joint. , This represents the total number of joints. The midpoint between the key points of the left and right hips is taken as the center of gravity of the human body, and is denoted as... And set the normalization parameters. ,in, This represents the target height of the area where the human body is located after normalization. If we represent the overall range of the heatmap, then center normalization can be expressed as:

[0112]

[0113]

[0114] in, Indicates the first The normalized coordinates of each joint center.

[0115] Furthermore, let the maximum height of a human joint be... Minimum height is Then, scale normalization can be expressed as:

[0116]

[0117]

[0118] in, Indicates the first The coordinates after joint scale normalization.

[0119] It is explained that through the above normalization process, the original human body position can be uniformly mapped into a standardized space, so that the skeletal coordinates of different subjects, different shooting distances and different viewing angles have higher consistency, thereby providing a stable input for subsequent heatmap generation and feature extraction.

[0120] Step S13: Create joint heatmaps and limb heatmaps based on the coordinate normalization results. The joint heatmap is used to characterize the positional distribution of each joint point. It is obtained by superimposing the Gaussian responses corresponding to each joint. The corresponding calculation formula is:

[0121]

[0122] in, Represents the pixel coordinates in the heatmap; This indicates the joint heatmap at the pixel level. The response value at the location; Indicates a joint index; Represents the normalized i-th The coordinates of each joint position; Indicates the first Confidence scores for each joint; Parameters representing the Gaussian distribution;

[0123] Limb heatmaps are used to represent the connectivity between adjacent joints. They are generated by encoding the distance from each pixel in the image to the nearest limb. The corresponding calculation formula is as follows:

[0124]

[0125] in, This indicates the limb heatmap at the pixel level. The response value at the location; Indicates a limb index; Indicates based on the first The line segments corresponding to each limb; Represents pixels To limbs The distance; Indicates the formation of the first The two endpoint indices of the bar; Indicates the first The confidence score for each limb is determined by taking the smaller confidence score of the two endpoint joints. Parameters representing the Gaussian distribution;

[0126] It should be noted that the joint thermogram and limb thermogram use the same Gaussian kernel parameters. This ensures the consistency of the two heatmaps in terms of spatial scale, facilitating subsequent feature fusion and unified learning.

[0127] Step S14: Stack the joint diagram and limb diagram into a two-channel skeleton diagram, use subject-centered alignment to centrally crop the blank area, and adjust the size of the skeleton diagram according to the input size after cropping to obtain the final skeleton diagram;

[0128] It can be explained that by combining joint heatmaps and limb heatmaps into a unified dual-channel representation, on the one hand, the local positional information of joints and the structural information of limb connections can be preserved simultaneously, and on the other hand, the skeletal modality data can be made more suitable for subsequent convolutional networks to extract spatial features, thereby improving the model's ability to perceive human structural relationships.

[0129] Step S15: Perform length and size unification processing on the preprocessed gait input data. Sample one frame from each original sequence in chronological order every preset number of frames to extract a fixed number of gait frames as a complete sample. Preferably, in this embodiment, the sequence sample is constructed by sampling one frame every four frames and extracting a total of 30 frames.

[0130] Furthermore, to enhance the model's adaptability to complex scene changes and improve data diversity and generalization ability, data augmentation operations can be performed on the binary contour map sequence and skeleton map sequence. These data augmentation operations include random horizontal flipping, random rotation, and random perspective transformation. Preferably, in this embodiment, the probability of applying each augmentation strategy is 0.2. After augmentation, the image is cropped and standardized to a fixed resolution; preferably, in this embodiment, it is standardized to an input size of 64×44.

[0131] As explained, in step S2, a gait recognition model is constructed. The gait recognition model includes an information extraction and low-dimensional feature fusion module, a unified spatiotemporal feature extraction module, a pooling module, and a recognition output module. That is, while maintaining a simple and efficient overall structure, the gait recognition model completes the initial feature extraction of gait input data, low-dimensional fusion of different modal data, modeling of complex spatiotemporal dependencies, generation of sequence-level gait representations, and final identity matching, thereby constructing a gait recognition framework with unified spatiotemporal modeling capabilities to improve the recognition accuracy, robustness, and generalization ability of the model in controlled scenarios and real complex scenarios.

[0132] Furthermore, step S3 includes:

[0133] Step S31: Input the preprocessed gait input data into the information extraction module to extract the contour features corresponding to the binary contour map mode and the skeletal features corresponding to the skeletal map mode, respectively;

[0134] Preferably, the binary contour map and the skeleton map are respectively extracted through their respective convolutional networks. More preferably, they are respectively extracted through their respective residual networks to capture modality-specific information. The binary contour map branch is used to extract the shape features and global appearance information of the gait, and the skeleton map branch is used to extract the motion patterns and spatial structure information of the human joints.

[0135] Furthermore, the contour map branch outputs a contour feature map, denoted as... The skeletal graph branch outputs a skeletal feature map, denoted as... In the preferred embodiment, and The dimensions are all represented as ,in, Indicates batch dimension, Indicates the number of channels. Indicates the number of time frames. and These represent the height and width of the feature map, respectively. By unifying features from different modalities into the same dimensional space, a consistent input representation can be provided for subsequent low-dimensional feature fusion.

[0136] Step S32: When using a single-modal input, output the contour map feature representation; when using a multimodal input, map the contour features and skeletal features to the same low-dimensional feature space, and perform dynamic weighted fusion based on the low-dimensional feature fusion module through an attention mechanism to obtain the fused feature representation of the contour map and skeletal map.

[0137] Specifically, when a multimodal implementation is adopted, after completing the contour feature map... and skeletal feature map After extraction, the contour feature map and the skeletal feature map are concatenated along the channel dimension to obtain the concatenated fused input feature map. .

[0138] The fused input feature map An input attention fusion unit is used to generate channel attention weights corresponding to the contour feature map and the skeletal feature map; preferably, the attention fusion unit employs a compact neural network with compression and activation structures; more preferably, the attention fusion unit includes a Convolutional layers and two Convolutional layer, where the first Convolutional layers are used for feature compression. The convolutional layer is used for attention weight generation, the second one. Convolutional layers are used to linearly map features back to the original dimensions, and the Softmax function is used to normalize the weights in the channel dimension to obtain the weights corresponding to the contour modality and the weights corresponding to the skeletal modality, respectively.

[0139] Based on the attention weights of the contour feature map and the skeletal feature map, the contour feature map is... and the skeletal feature map Each channel is dynamically weighted, and the weighted results are then fused to obtain a fused feature representation of the contour map and the skeleton map. Preferably, in a single-modal implementation, the information extraction and low-dimensional feature fusion module directly outputs a contour map feature representation, and uses this contour map feature representation as input to the subsequent unified spatiotemporal feature extraction module; in a multimodal implementation, the information extraction and low-dimensional feature fusion module outputs the fused feature representation. The fused feature representation is then used as the input to the subsequent unified spatiotemporal feature extraction module.

[0140] Understandably, through the above processing method, this embodiment achieves effective integration of complementary information of contour modality and skeletal modality in the information extraction and low-dimensional feature fusion module, so that the output features not only retain the overall discrimination information of human appearance contour, but also enhance the semantic expression ability of human joints and limb structure, thereby providing better input for the subsequent unified spatiotemporal feature extraction module to further model complex gait spatiotemporal dependencies.

[0141] Further, step S4 includes:

[0142] Step S41: The unified spatiotemporal feature extraction module includes a first two-dimensional convolutional unit, a spatial attention unit, a dimension restoration unit, a three-dimensional convolutional unit, a temporal channel attention unit, a second two-dimensional convolutional unit, and a residual connection unit;

[0143] Understandably, existing gait recognition methods typically process spatial and temporal features separately during spatiotemporal modeling, making it difficult to fully describe the significant differences in local human body regions and the dynamic changes across time frames in gait sequences. In contrast, this embodiment uses a hierarchical processing approach to uniformly model the spatiotemporal dependencies of gait, thereby improving the accuracy, robustness, and generalization ability of gait recognition in complex scenarios.

[0144] Please see Figure 3 , Figure 3 This is a schematic diagram of a unified spatiotemporal feature extraction module provided in an embodiment of the present invention. The unified spatiotemporal feature extraction module specifically includes:

[0145] Step S42: Reshape the dimensions based on the input feature map, and then input the first two-dimensional convolutional unit and the spatial attention unit in sequence to obtain the spatially enhanced features;

[0146] Further, step S42 includes:

[0147] Step S421: Denote the input feature map as Its dimensions are ,in, Indicates batch dimension, Indicates the number of channels. Indicates the number of time frames. and These represent the height and width of the feature map, respectively; the input feature map is rearranged along the batch and time dimensions to reshape it into... Its dimensions are ;

[0148] Understandably, through the aforementioned dimensional reshaping, the spatial features of each time frame can be regarded as an independent two-dimensional feature map, so that two-dimensional convolutional units and spatial attention units can be used to spatially enhance the key regions of gait.

[0149] Step S422: Input the reshaped feature map into the first two-dimensional convolutional unit for spatial convolution operation to extract local spatial response features and provide basic spatial representation for subsequent spatial attention units (SA);

[0150] Step S423: Perform global average pooling and global max pooling on the reshaped feature maps respectively, aggregate statistical information in the channel dimension to obtain average pooling feature maps and max pooling feature maps, and then concatenate the average pooling feature maps and max pooling feature maps in the channel dimension.

[0151] The concatenated feature map is input into a convolutional layer with a kernel size of 7×7 to obtain a spatial attention weight map. The spatial attention weight map is then normalized using the Sigmoid function, and the corresponding calculation formula is as follows:

[0152]

[0153] in, This represents the feature map obtained by performing global average pooling on the reconstructed feature map; This represents the feature map obtained by performing global max pooling on the reshaped feature map; Indicates a splicing operation; This represents the Sigmoid normalization function; Indicates the kernel size as Convolution operation; Represents the spatial attention weight map;

[0154] Step S424: Based on the normalized spatial attention weight map, perform element-wise weighting on the reshaped feature map to obtain the spatially augmented feature map. The corresponding calculation formula is:

[0155]

[0156] in, This represents the reshaped input feature map; Indicates position Spatial attention weights at locations; The feature map is represented by the augmented representation space.

[0157] It is explained that through the above spatial attention processing, key areas in human body contours and skeletal features that contribute more to identity recognition can be highlighted, while background interference, redundant areas and irrelevant responses can be suppressed, thereby improving the model's ability to express gait local structural information.

[0158] Step S43: After restoration based on the spatial augmentation feature map, the data is input into a 3D convolutional unit and a temporal channel attention unit (TCA), and common gait features between different samples are extracted to obtain preliminary spatiotemporal augmentation features;

[0159] Further, step S43 includes:

[0160] Step S431: Restore the spatially enhanced feature maps to their original temporal arrangement according to the batch dimension and time dimension to obtain the restored spatiotemporal feature representation;

[0161] Step S432: Input the recovered spatiotemporal feature representation into the three-dimensional convolutional unit, and make the convolutional kernel slide synchronously along the time and space dimensions to extract dynamic change features across time frames. Based on the feature map output by the three-dimensional convolutional unit, obtain the spatiotemporal feature representation containing the initial spatiotemporal dependency.

[0162] Please see Figure 4 , Figure 4 This is a schematic diagram of a timing channel attention unit structure provided in an embodiment of the present invention. The timing channel attention unit specifically includes:

[0163] Step S433: Perform global average pooling on the spatiotemporal features output by the 3D convolutional unit in the spatial dimension to obtain a temporal feature vector that retains the global context information of each time step. The corresponding calculation formula is as follows:

[0164]

[0165] in, Represents the spatiotemporal feature map output by a 3D convolutional unit; This indicates global average pooling along the spatial dimension; Indicates the height and width of the feature map; Indicates a spatial dimension index; This represents the time-series feature vector after pooling;

[0166] The temporal feature vector is input into the first convolutional layer for channel compression to obtain intermediate temporal features. The corresponding calculation formula is as follows:

[0167]

[0168] in, and These represent the weight parameters and bias parameters of the first convolutional layer, respectively. This represents the intermediate timing characteristics after compression;

[0169] Step S434: Input the intermediate temporal features into the common gait feature extraction unit to model the common gait patterns of different samples in the temporal dimension, obtain common gait enhancement features, input the common gait enhancement features into the second convolutional layer for dimensionality recovery, and generate temporal channel attention weights through the Sigmoid function. Use the temporal channel attention weights to recalibrate the initial spatiotemporal features element-wise to obtain the temporally enhanced feature map. The corresponding calculation formula is:

[0170]

[0171] in, This represents a common gait feature extraction unit, preferably a state space sequence modeling unit, and more preferably a Mamba2 module; This represents the initial spatiotemporal feature map output by the 3D convolutional unit; and These represent the weight parameters and bias parameters of the second convolutional layer or linear mapping layer, respectively. This represents the Sigmoid normalization function; This represents the feature map after temporal enhancement.

[0172] Step S44: Refine the preliminary spatiotemporal enhancement features based on the second two-dimensional convolutional unit, and perform residual connection between the output features and the input feature map to obtain the final enhanced spatiotemporal features.

[0173] The temporally enhanced feature map is input into the second two-dimensional convolutional unit to perform further nonlinear mapping and local structure refinement on the spatiotemporally fused feature map, enhance the feature recombination ability between different channels and the local expression ability between adjacent spatial regions, and output the refined spatiotemporal feature map after processing by the second two-dimensional convolutional unit.

[0174] The refined spatiotemporal feature map is added element-wise to the original feature map before the input spatiotemporal feature extraction module to output the final enhanced spatiotemporal feature.

[0175] Furthermore, step S5 includes:

[0176] Step S51: Treat the input enhanced spatiotemporal feature sequence as a set of gait features composed of features from multiple time steps, denoted as:

[0177]

[0178] in, Indicates the first The feature vector corresponding to the frame, This indicates the length of the gait sequence; different sequence lengths are allowed for different gait samples. Represents the set of gait features;

[0179] The gait feature set is aggregated along the time dimension using a set pooling module. A max-pooling strategy is then applied to aggregate the temporal features, extracting the most significant features from the entire gait sequence. This transforms the variable-length gait sequence into a fixed-dimensional sequence-level gait representation. The corresponding calculation formula is as follows:

[0180]

[0181] in, Indicates the first The feature vector corresponding to the frame; Indicates the length of the gait sequence; This represents the fixed-dimensional feature representation after pooling.

[0182] To clarify, the core function of ensemble pooling is twofold: firstly, it maps gait sequences of different lengths into a fixed-dimensional representation, facilitating unified processing by subsequent classifiers and metric learning modules; secondly, by compressing information across the entire time dimension, it can reduce redundant temporal interference while preserving key dynamic features, thereby improving the model's robustness to occlusion, noise, gait speed changes, and sequence length differences.

[0183] Optionally, the pooling module may also employ an average pooling strategy or an attention-based pooling strategy; wherein, average pooling is used to balance the feature contributions of each time frame, while attention-based pooling highlights more important temporal information by assigning different weights to different time frames. However, in a preferred embodiment, to enhance the ability to preserve significant dynamic patterns in the gait sequence, this embodiment employs a maximum pooling strategy.

[0184] Step S52: Divide the feature map after ensemble pooling into layers along the height dimension to obtain multiple horizontal stripe regions. Each horizontal stripe corresponds to a local region of the human body at different height positions, used to represent local gait features in the head and shoulder region, trunk region, leg region, or foot region, respectively. The corresponding calculation formula is as follows:

[0185]

[0186] in, This represents the feature map after pooling; and Indicates the first The vertical boundary of a horizontal stripe; This represents the total number of local feature maps corresponding to the horizontal stripes; Indicates the first Local feature map corresponding to each horizontal stripe;

[0187] Pooling is applied to each horizontal stripe region to integrate the spatial features within that local region, resulting in a corresponding local feature vector. This partitioning and pooling operation is repeated at different granularities. All local feature vectors are then concatenated and flattened to obtain the final sequence-level gait representation, calculated using the following formula:

[0188]

[0189] in, Indicates a splicing operation; Indicates the flattening operation; This represents the pooling function, preferably max pooling or average pooling. This represents the final sequence-level gait representation.

[0190] It can be explained that horizontal pyramid pooling, by stratifying different height regions of the human body, preserves the local structural differences in the vertical direction. This allows the model to not only learn the global features of the overall gait contour but also extract fine-grained local features corresponding to different body parts, thereby improving the model's ability to perceive local posture changes and regional differences. Especially under complex conditions such as changes in viewpoint, clothing, and partial occlusion, this approach can enhance the discriminativeness and robustness of gait features.

[0191] The sequence-level gait representation has the following advantages: First, it can adapt to variable-length gait sequence inputs, improving the model's flexibility in processing sequences at different time scales; second, it can retain the most significant dynamic response in the time dimension and preserve the local structural differences of different body regions in the spatial dimension; third, it can provide stable and highly discriminative feature inputs for subsequent recognition output modules, thereby further improving the recognition performance of the gait recognition model in complex scenarios.

[0192] Further, step S6 includes:

[0193] Step S61: Input the sequence-level gait representation into the feature mapping unit to obtain the embedded features for calculating the distance between samples. Input the embedded features into the batch normalization neck network to normalize and reshape the feature distribution, and input the final processing result into the classifier.

[0194] It is explained that the embedded features mainly serve the metric learning objective. Their core function is to bring samples with the same identity closer to each other in the feature space and to keep samples with different identities further apart, thereby providing a more discriminative feature basis for calculating the distance between samples and subsequent sequence-level matching. The role of the batch normalization neck network is to appropriately decouple the embedded features used for metric learning from the discriminative features used for classification learning, improve the consistency and stability of the feature distribution, and enable the classifier to learn identity boundaries in a more regular feature space, thereby improving the convergence effect and recognition performance of the classification training process.

[0195] Step S62: Construct triplet sample pairs based on the embedded features output by the feature mapping unit, and select anchor samples from a batch of training samples. Positive samples belonging to the same identity category as the anchor sample. And negative samples that belong to a different identity category than the anchor sample. Calculate the distance between the anchor point sample and the positive sample, and the distance between the anchor point sample and the negative sample;

[0196] Based on the preset margin parameter constraining the distance relationship between positive and negative samples, the distance between the anchor sample and the positive sample is made smaller than the distance between the anchor sample and the negative sample. The loss values ​​of all triplet sample pairs are summed or averaged to obtain the triplet loss value. The corresponding calculation formula is as follows:

[0197]

[0198] in, Indicates the number of triplet sample pairs; Indicates sample With sample The distance metric function between them, wherein the distance metric function is Euclidean distance; Indicates the first Anchor sample embedding features in each triplet; Indicates the first Positive sample embedding features in triplet; Indicates the first Negative sample embedding features in triplet; The margin parameter is described. The value is 0.2; Represents the triplet loss function;

[0199] It is explained that by minimizing the triplet loss function, the model can cluster similar samples into neighboring regions in the embedding space while pushing dissimilar samples away, thereby enhancing the intra-class compactness and inter-class separability of features.

[0200] Step S63: Calculate the single-sample classification error based on the predicted probability of the identity category output by the classifier and the real identity label. Calculate the average classification error of all samples within a batch to obtain the cross-entropy loss value. The corresponding calculation formula is:

[0201]

[0202] in, Indicates the number of samples in the batch; Indicates the first The predicted probability that a sample belongs to its true category; Represents the cross-entropy loss function;

[0203] Understandably, the cross-entropy loss function enables the model to learn feature boundaries that have global discriminative power for identity categories by supervising and optimizing the classifier; compared with the triplet loss function, the cross-entropy loss focuses more on the accuracy of category prediction, while the triplet loss focuses more on optimizing the relative distance relationship between samples.

[0204] Step S64: Jointly sum the triplet loss function and the cross-entropy loss function to form the total loss function for model training. The corresponding calculation formula is as follows:

[0205]

[0206] in, Represents the total loss function. Represents the triplet loss function. This represents the cross-entropy loss function.

[0207] The explanation is that combining the triplet loss function with the cross-entropy loss function has at least the following advantages: First, it can simultaneously improve the discriminative and classification capabilities of features; second, it can enhance the robustness of the model to cross-viewpoint, cross-pose changes, and complex gait change scenarios; third, in scenarios with uneven sample distribution or complex scenarios, it can improve the overall recognition performance of the model through complementary optimization of classification and distance objectives.

[0208] Specifically, during training, a sampling method based on triplet loss is used to construct training batches, and a sampler is used to perform balanced sampling of samples from different identity categories to mitigate the adverse effects of class imbalance and sample redundancy on model convergence. The training data is randomly shuffled before loading to reduce the potential bias of sample order on the model learning process. The training batch size is set to... Where 8 represents the number of identities sampled in each batch, 16 represents the number of gait sequences corresponding to each identity, and 30 represents the frame length of each gait sequence; the initial learning rate is set to 0.05, the total training cycle is set to 60,000 rounds, and a phased learning rate decay strategy is adopted, reducing the learning rate by 0.1 at the 20,000th, 40,000th, and 50,000th rounds respectively to improve the convergence stability in the later stages of training. The gait recognition model uses a stochastic gradient descent optimizer to update the model parameters, and the margin parameter in the triplet loss function is set to 0.2.

[0209] Furthermore, to further balance the training contributions of different modal branches, a classifier can be set before the main network output, and cross-entropy loss can be introduced to enhance the independent discrimination ability of each modal branch and improve the training stability of the overall model. The weight hyperparameter of the auxiliary classification loss can be set to 0.5.

[0210] Please see Figures 5 to 7 To better illustrate this, this embodiment provides a visual analysis of the feature distribution of the gait recognition method based on unified spatiotemporal feature extraction provided by the present invention under different datasets and different input modes. Figure 5A t-SNE visualization of the contour-based model on the CASIA-B dataset. Figure 6 A t-SNE visualization of the contour-based model on the Gait3D dataset. Figure 7 This is a t-SNE visualization of the multimodal model on the Gait3D dataset. By visualizing the high-dimensional gait features output by the model through dimensionality reduction, the distribution of different identity categories in the feature space can be reflected more intuitively, thereby verifying the ability of the method of this invention to extract gait discrimination features.

[0211] Please see Figure 5 On the CASIA-B dataset, the model output features exhibit a relatively clear clustered distribution in two-dimensional space, with distinct boundaries between different categories, and most categories forming relatively independent cluster regions. This indicates that the method of this invention can effectively extract gait features of different identities in a controlled scene and maintain good class discriminability after dimensionality reduction. Although the boundaries between categories are generally clear, a small amount of sample overlap still exists at some cluster boundaries. This phenomenon may be related to the large fluctuations in gait features under varying clothing conditions.

[0212] Please see Figure 6 On the Gait3D dataset, compared to Figure 5 As shown in the CASIA-B dataset, the category distribution density is significantly increased, and the separation between clusters is weakened. This result reflects the challenge of gait recognition tasks in real-world complex scenes. Influenced by factors such as complex backgrounds, occlusion, lighting variations, and viewpoint differences, the distribution of samples from different categories is denser, and there is boundary overlap between some categories. However, despite the high complexity of the Gait3D dataset, the model can still achieve a certain degree of clustering effect in most categories, indicating that the method of this invention still has good generalization ability on real-world environmental data.

[0213] Please see Figure 7 On the Gait3D dataset, t-SNE visualization results for the multimodal model show that the class organization trend in the feature space is further improved after incorporating skeletal modality information. This result indicates that fusing human contour modality with skeletal modality can enhance the separability of gait features across different identities to a certain extent, thereby improving the model's discriminative ability in complex real-world scenes. Combined with the aforementioned... Figure 5 and Figure 6 It can be seen that the method of the present invention can not only maintain good class distinguishability in controlled scenarios, but also further enhance the feature expression effect by introducing multimodal information in real complex scenarios.

[0214] To better illustrate this, this embodiment compares and analyzes the recognition results of contour-based models on the CASIA-B dataset and the Gait3D dataset, as well as the recognition results of multimodal models on the Gait3D dataset, resulting in Tables 1 and 2.

[0215] Table 1: Comparison of recognition results of contour-based models on the CASIA-B and Gait3D datasets.

[0216]

[0217] Table 2: Comparison of recognition results of multimodal models on the Gait3D dataset

[0218]

[0219] As shown in Table 1, the contour-based model described in this invention maintains good recognition performance on the controlled scene dataset CASIA-B. On the real-world complex scene dataset Gait3D, the contour-based model achieves a recognition accuracy of 73.4%, which is higher than the corresponding results of methods such as GaitSet, GaitPart, GaitGL, and DeepGaitV2. This indicates that the unified spatiotemporal feature extraction framework constructed in this invention has good recognition performance and generalization ability on real-world scene data.

[0220] As shown in Table 2, in the multimodal recognition task on the Gait3D dataset, the multimodal model described in this invention exhibits superior results in Rank-1, Rank-5, mAP, and mINP metrics, with Rank-1 reaching 78.0%, Rank-5 reaching 89.6%, mAP reaching 71.1%, and mINP reaching 43.2%. This indicates that by fusing human contour modality and skeletal modality information and combining them with a unified spatiotemporal feature extraction structure, the feature discrimination ability and retrieval performance in gait recognition tasks under complex real-world scenarios can be effectively improved.

[0221] It can be noted that by constructing a unified spatiotemporal feature extraction module and combining complementary information from contour modalities and skeletal modalities, this invention can not only form a better category organization trend in the feature space, but also achieve better recognition results in real and complex scenarios. This demonstrates that this invention has a positive effect on improving the discrimination ability, robustness and generalization performance of gait recognition models.

[0222] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0223] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

Claims

1. A gait recognition method based on unified spatiotemporal feature extraction, characterized in that, The method includes the following steps: Acquire gait input data and preprocess the gait input data; Based on the information extraction and low-dimensional feature fusion module, feature representation is constructed on the preprocessed gait input data to obtain contour map feature representation or contour map and skeleton map fusion feature representation; The feature representation is processed by the unified spatiotemporal feature extraction module to obtain enhanced spatiotemporal features. The unified spatiotemporal feature extraction module is used to perform spatial enhancement, temporal dynamic modeling and common gait feature extraction on the input features. The enhanced spatiotemporal features are aggregated using a pooling module to obtain a sequence-level gait representation. The gait recognition model is trained using sequence-level gait representation based on the recognition output module to obtain the trained gait recognition model. The gait input data corresponding to the object to be identified is input into the trained gait recognition model, and the output sequence-level gait representation is matched at the sequence level to obtain the identity recognition result.

2. The gait recognition method according to claim 1, characterized in that, The gait input data includes a binary contour map sequence and a raw skeletal coordinate sequence; Preprocessing of gait input data includes: Center normalization and scale normalization are performed based on the original skeleton coordinate sequence; Joint heatmaps and limb heatmaps are generated based on normalized skeletal coordinates; The joint heatmap and the limb heatmap are stacked to form a dual-channel skeleton map, and the dual-channel skeleton map is then cropped at the center and its size is standardized. The sequence length and input size of the gait input data are standardized. The joint heatmap is obtained by superimposing the Gaussian responses corresponding to each joint, and the limb heatmap is generated by encoding the distance from each pixel in the image to the nearest limb. Specifically, joint heatmaps and limb heatmaps are created based on the coordinate normalization results. The joint heatmap is used to characterize the positional distribution of each joint point and is obtained by superimposing the Gaussian responses corresponding to each joint. The corresponding calculation formula is as follows: in, Represents the pixel coordinates in the heatmap; This indicates the joint heatmap at the pixel level. The response value at the location; Indicates a joint index; Represents the normalized i-th The coordinates of each joint position; Indicates the first Confidence scores for each joint; Parameters representing the Gaussian distribution; Limb heatmaps are used to represent the connectivity between adjacent joints. They are generated by encoding the distance from each pixel in the image to the nearest limb. The corresponding calculation formula is as follows: in, This indicates the limb heatmap at the pixel level. The response value at the location; Indicates a limb index; Indicates based on the first The line segments corresponding to each limb; Represents pixels To limbs The distance; Indicates the formation of the first The two endpoint indices of the bar; Indicates the first The confidence score for each limb is determined by taking the smaller confidence score of the two endpoint joints. Parameters representing the Gaussian distribution; The joint and limb diagrams are stacked into a two-channel skeleton diagram. The blank areas are then centrally cropped using a subject-centered alignment. After cropping, the skeleton diagram size is adjusted according to the input dimensions to obtain the final skeleton diagram.

3. The gait recognition method according to claim 1, characterized in that, The information extraction and low-dimensional feature fusion module includes a contour graph branch and a skeleton graph branch; When a single-modal input is used, the output contour map feature representation is used. When using multimodal input, contour features and skeletal features are mapped to the same low-dimensional feature space, and dynamic weighted fusion is performed through an attention mechanism to obtain a fused feature representation of contour map and skeletal map.

4. The gait recognition method according to claim 1, characterized in that, The unified spatiotemporal feature extraction module includes a first two-dimensional convolutional unit, a spatial attention unit, a dimension restoration unit, a three-dimensional convolutional unit, a temporal channel attention unit, a second two-dimensional convolutional unit, and a residual connection unit; The unified spatiotemporal feature extraction module processes the feature representation, including: The input feature map is dimensionally reshaped and then sequentially input into the first two-dimensional convolutional unit and the spatial attention unit to obtain spatially enhanced features. The spatially enhanced features are restored to their original temporal arrangement and input into the 3D convolutional unit and the temporal channel attention unit to obtain preliminary spatiotemporal enhanced features; The initial spatiotemporal enhancement features are refined based on the second two-dimensional convolutional unit, and the output features are residually connected with the input features to obtain the final enhanced spatiotemporal features.

5. The gait recognition method according to claim 4, characterized in that, The spatial attention unit performs global average pooling and global max pooling on the reshaped feature map, and generates spatial attention weights based on the pooling results to weight the reshaped feature map to obtain the spatially enhanced features. The time-series channel attention unit includes: Global average pooling is performed on the spatiotemporal features output by the 3D convolutional unit in the spatial dimension to obtain the temporal feature vector; The temporal feature vector is input into the first convolutional layer for channel compression to obtain intermediate temporal features; The intermediate temporal features are input into the common gait feature extraction unit to model the common gait patterns of different samples in the temporal dimension, thereby obtaining common gait enhancement features; The common gait enhancement features are input into the second convolutional layer for dimensionality recovery and temporal channel attention weights are generated. The initial spatiotemporal features are then recalibrated using the temporal channel attention weights to obtain the temporally enhanced feature map. Specifically, the spatiotemporal features output by the 3D convolutional unit are subjected to global average pooling in the spatial dimension to obtain a temporal feature vector that retains the global context information of each time step. The corresponding calculation formula is as follows: in, Represents the spatiotemporal feature map output by a 3D convolutional unit; This indicates global average pooling along the spatial dimension; Indicates the height and width of the feature map; Indicates a spatial dimension index; This represents the time-series feature vector after pooling; The temporal feature vector is input into the first convolutional layer for channel compression to obtain intermediate temporal features. The corresponding calculation formula is as follows: in, and These represent the weight parameters and bias parameters of the first convolutional layer, respectively. This represents the intermediate timing characteristics after compression.

6. The gait recognition method according to claim 5, characterized in that, The common gait feature extraction unit is a state-space sequence modeling unit. The state-space sequence modeling unit is the Mamba2 module.

7. The gait recognition method according to claim 1, characterized in that, The pooling module includes a collection pooling module and a horizontal pyramid pooling module, wherein: The pooling module is used to aggregate enhanced spatiotemporal feature sequences along the time dimension to convert variable-length gait sequences into fixed-dimensional sequence-level gait representations. The horizontal pyramid pooling module is used to perform hierarchical partitioning and local pooling of the sequence-level gait representation along the height dimension to obtain the sequence-level gait representation.

8. The gait recognition method according to claim 1, characterized in that, The recognition output module includes a feature mapping unit, a batch normalization network, and a classifier, wherein: The feature mapping unit is used to map the sequence-level gait representation to the metric learning space; The batch normalization neck network is used to optimize feature distribution; The classifier is used to output identity category prediction results to assist training.

9. The gait recognition method according to claim 8, characterized in that, The loss functions used to train the gait recognition model include the triplet loss function and the cross-entropy loss function; The triplet loss function is calculated based on triplet sample pairs constructed from anchor samples, positive samples, and negative samples; The cross-entropy loss function is calculated based on the predicted probability of the identity category output by the classifier and the real identity label; The distance metric in the triplet loss function is Euclidean distance. The margin parameter in the triplet loss function is a preset value. The classification error for a single sample is calculated based on the predicted probability of the identity category output by the classifier and the real identity label. The average classification error of all samples in a batch is then used to obtain the cross-entropy loss value, and the corresponding calculation formula is as follows: in, Indicates the number of samples in the batch; Indicates the first The predicted probability that a sample belongs to its true category; Represents the cross-entropy loss function; The total loss function for model training is calculated by jointly summing the triplet loss function and the cross-entropy loss function. The corresponding calculation formula is as follows: in, Represents the total loss function. Represents the triplet loss function. This represents the cross-entropy loss function.

10. The gait recognition method according to claim 1, characterized in that, The sequence-level matching uses cosine similarity for similarity calculation, and the registered identity with the smallest distance or the highest similarity to the sequence-level gait representation of the object to be identified is output as the identity recognition result.

Images