A gait recognition method and system based on multi-modal fusion and space-time modeling

Abstract

The application belongs to the technical field of computer vision, and specifically discloses a gait recognition method and system based on multi-modal fusion and space-time modeling. The method builds a multi-modal gait recognition model based on the MultiGait++ architecture. A progressive difference complementary fusion module is designed in the model, which divides the modal fusion into two stages of common alignment and difference complementation, and realizes the fusion of cross-modal features through a difference residual guiding mechanism. The application also designs a modal adaptive gating module, which dynamically adjusts the weights of each modal feature through a global average pooling and confidence prediction mechanism to improve the robustness of the system. The application also designs a space-time modeling module, which captures the space-time dependence of gait actions through a window shifting mechanism and fuses with the convolution layer output to realize the collaborative representation of global dynamics and local details. The application can fully exploit the multi-modal gait features in complex scenes, and further realize high-precision identity recognition.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a gait recognition method and system based on multimodal fusion and spatiotemporal modeling. Background Technology

[0002] Gait recognition is an important research area in computer vision and behavior analysis, enabling contactless identity verification by analyzing human posture, shape, and motion characteristics during walking. Compared to traditional biometric identification methods based on facial features and fingerprints, gait recognition can still identify individuals at long distances and under occlusion, and has broad application prospects in scenarios such as public safety, intelligent monitoring, and human-computer interaction.

[0003] However, existing gait recognition methods still face several challenges. On the one hand, single-modal methods (such as those using only silhouette images) are easily affected by factors such as clothing, lighting, and posture changes in complex scenes, resulting in insufficient feature representation capabilities. Especially under cross-scene and cross-viewpoint conditions, silhouette features often suffer from blurred boundaries and missing details, making it difficult for the model to extract stable identity representations. On the other hand, although multimodal gait recognition is gradually becoming the mainstream trend, existing fusion strategies are mostly static splicing or simple weighted fusion, which cannot fully explore the complementary information between different modalities. For example, analytical maps can reflect the semantics of human body structure, and optical flow maps can describe the direction and speed of movement. If these are directly spliced ​​and fused with silhouette features, redundancy and conflicts will occur due to differences in modal scale and noise interference, reducing the overall recognition performance.

[0004] Furthermore, gait sequences are inherently spatiotemporally dynamic processes, and traditional two-dimensional convolutional networks struggle to effectively model inter-frame dependencies, resulting in insufficient utilization of temporal information. While three-dimensional convolutional or recurrent networks can capture dynamic features to some extent, their high computational complexity and large parameter count make them unsuitable for large-scale deployments.

[0005] In summary, existing gait recognition technologies still have the following shortcomings in terms of multimodal fusion and spatiotemporal modeling: the multimodal information fusion method lacks a dynamic adaptive mechanism, making it difficult to balance the feature contributions of different modalities; the commonalities and differences between modalities are not modeled hierarchically, resulting in insufficient cross-modal correlation; and the temporal feature modeling capability is limited, failing to capture cross-frame spatiotemporal residual information. Summary of the Invention

[0006] The purpose of this invention is to propose a gait recognition method and system based on multimodal fusion and spatiotemporal modeling. This method realizes dynamic fusion and hierarchical optimization of multimodal features, which can significantly improve recognition accuracy.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A gait recognition method based on multimodal fusion and spatiotemporal modeling includes the following steps:

[0009] Step 1. Acquire gait recognition images, preprocess the gait recognition images, and construct a training dataset;

[0010] Step 2. Build a multimodal gait recognition model based on the MultiGait++ architecture. This model is designed with a progressive difference complementarity fusion module. Modal Adaptive Gating Module (AMG Module) and Spatiotemporal Modeling Module;

[0011] in The module in the common-difference feature extraction module ( Based on the module, common features, appearance features and motion features are fused to obtain complementary enhanced features, and then linear fused features are obtained through linear projection;

[0012] The linear fusion features output by the module are processed by pseudo-3D convolution and then fed into the AMG module. Through global average pooling and confidence prediction, the weights of common features, appearance features, motion features and complementary enhancement features are dynamically adjusted to obtain weighted fusion features.

[0013] The weighted fusion features are divided into two branches in the spatiotemporal modeling module. One branch is processed through a window shifting mechanism, and the other branch is processed by convolution. Then the outputs of the two branches are fused to obtain the spatiotemporal fusion features.

[0014] Step 3. Train the multimodal gait recognition model built based on the training dataset in Step 1, and use the trained multimodal gait recognition model to perform gait recognition.

[0015] Furthermore, based on the aforementioned gait recognition method based on multimodal fusion and spatiotemporal modeling, this invention also proposes a corresponding gait recognition system based on multimodal fusion and spatiotemporal modeling, which adopts the following technical solution:

[0016] A gait recognition system based on multimodal fusion and spatiotemporal modeling includes the following modules:

[0017] The preprocessing module is used to acquire gait recognition images, preprocess the gait recognition images, and build a training dataset;

[0018] And a multimodal gait recognition module, used to build a multimodal gait recognition model based on the MultiGait++ architecture, which is designed with Modules, AMG module, and spatiotemporal modeling module;

[0019] in Module in Based on the modules, common features, appearance features and motion features are fused to obtain complementary enhanced features, and then linearly fused features are obtained through linear projection.

[0020] The linear fusion features output by the module are processed by pseudo-3D convolution and then fed into the AMG module. Through global average pooling and confidence prediction, the weights of common features, appearance features, motion features and complementary enhancement features are dynamically adjusted to obtain weighted fusion features.

[0021] The weighted fusion features are divided into two branches in the spatiotemporal modeling module. One branch is processed through a window shifting mechanism, and the other branch is processed by convolution. Then the outputs of the two branches are fused to obtain the spatiotemporal fusion features.

[0022] The multimodal gait recognition model is trained based on the training dataset, and then used to perform gait recognition.

[0023] Furthermore, based on the aforementioned gait recognition method based on multimodal fusion and spatiotemporal modeling, this invention also proposes a computer device comprising a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it implements the steps of the aforementioned gait recognition method based on multimodal fusion and spatiotemporal modeling.

[0024] Furthermore, based on the gait recognition method based on multimodal fusion and spatiotemporal modeling described above, this invention also proposes a computer-readable storage medium storing a program that, when executed by a processor, implements the steps of the gait recognition method based on multimodal fusion and spatiotemporal modeling described above.

[0025] The present invention has the following advantages:

[0026] As described above, this invention relates to a gait recognition method based on multimodal fusion and spatiotemporal modeling, which constructs a multimodal gait recognition model based on the MultiGait++ architecture. Specifically, regarding the model architecture, this invention designs... Module, in Based on modules, form – Union structure The module separates common and differential features at the shallow stage. In the mid-level stage, the module achieves complementary fusion through cross-modal convolution and adaptive weights, effectively solving the problems of insufficient feature coupling and low utilization of complementary information in traditional multimodal fusion, and significantly enhancing the collaborative expression ability between silhouette, parsing, and optical flow modalities. Furthermore, this invention also designs an AMG module, which dynamically allocates weights based on input data quality and confidence, automatically reducing the weights of modalities affected by noise, occlusion, or illumination interference, thereby improving the system's recognition stability in complex environments and cross-scene conditions. This mechanism achieves "dynamic confidence perception" of multimodal inputs, effectively enhancing the model's adaptability and anti-interference ability. Simultaneously, to strengthen spatiotemporal semantic modeling capabilities, this invention introduces a spatiotemporal modeling module. Through multi-head self-attention and sliding window attention mechanisms, it establishes cross-frame temporal dependencies and global contextual associations. Compared to traditional 3D convolutional structures, this module can capture long-range temporal dependencies while maintaining local spatial features, forming a more stable and temporally continuous gait representation. Overall, this invention can significantly improve recognition accuracy and cross-modal consistency on multimodal gait datasets, while maintaining lightweight design and high accuracy and robustness, and has broad application value and promotion potential. Attached Figure Description

[0027] Figure 1 This is an overall flowchart of the gait recognition method based on multimodal fusion and spatiotemporal modeling in an embodiment of the present invention.

[0028] Figure 2 This is a structural diagram of a multimodal gait recognition model based on the MultiGait++ architecture in an embodiment of the present invention.

[0029] Figure 3 As described in the embodiments of the present invention Module processing flowchart.

[0030] Figure 4 This is a flowchart of the AMG module processing in an embodiment of the present invention.

[0031] Figure 5 This is a structural diagram of the spatiotemporal modeling module in an embodiment of the present invention. Detailed Implementation

[0032] To overcome the shortcomings of existing gait recognition methods, such as insufficient multimodal fusion, feature redundancy, and limited temporal modeling capabilities, this invention proposes a gait recognition method based on multimodal fusion and spatiotemporal modeling. Addressing the deficiencies in multimodal gait recognition, such as large differences in information distribution among silhouette, analytical, and optical flow modes, static and rigid fusion methods, and insufficient utilization of dynamic features, this method proposes a multimodal gait recognition model based on the MultiGait++ architecture, introducing… The module, AMG module, and spatiotemporal modeling module process feature images, realizing dynamic fusion and hierarchical optimization of multimodal features. This invention can achieve adaptive dynamic fusion while maintaining complementary multimodal features, and combined with a gait recognition method based on spatiotemporal attention modeling, it improves the robustness and discriminative ability of the model in complex environments.

[0033] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0034] Example 1

[0035] This embodiment 1 describes a gait recognition method based on multimodal fusion and spatiotemporal modeling. This method, based on the original MultiGait++ framework, introduces a multi-module collaborative design to improve the multimodal information fusion capability, robustness, and temporal modeling accuracy in complex scenarios. Figure 1 As shown, the multimodal gait recognition method includes the following steps:

[0036] Step 1. Obtain gait recognition images, preprocess the gait recognition images, and construct a training dataset.

[0037] Gait recognition images undergo preprocessing. The gait recognition image data includes three modalities: silhouette images, analytical images, and optical flow maps. Silhouette images are generated by extracting the foreground from video frames and removing the background; background subtraction algorithms (such as Gaussian mixture models) are commonly used to separate the human body contour, followed by binarization to obtain a binary silhouette of the human body. Analytical images utilize human body segmentation techniques to divide the human body into different parts (such as head, torso, arms, legs, etc.); pre-trained deep learning models are typically used to extract the segmentation results from the dataset, obtaining label information for each part. Optical flow maps capture motion information by analyzing pixel changes between consecutive video frames; optical flow algorithms (such as VideoFlow) calculate the motion of each pixel between consecutive frames, generating an optical flow map for each frame, reflecting dynamic changes.

[0038] Each sample corresponds to a continuous walking video sequence. Each frame of the image is processed by unifying the size, cropping the frame order, normalizing the pixels, and converting the format. The images are then stored as serialized pkl files according to their modalities.

[0039] During the model loading phase, based on the sample index recorded in the JSON file, the corresponding multimodal pkl file with the corresponding ID is read synchronously and concatenated along the channel dimension to obtain the multimodal fusion input tensor X, the expression of which is as follows:

[0040] ;

[0041] Where S represents the silhouette mode, P represents the analytical mode, F represents the optical flow mode, and B represents the batch. The channel number is represented by T, the frame number by H, the height by W, and the width by W. Silhouettes and resolution maps occupy the first two channels, while optical flow maps occupy three channels. ( ) indicates a fusion operation.

[0042] Step 2. Build a multimodal gait recognition model based on the MultiGait++ architecture, such as... Figure 2 As shown, this model adds [something] in the middle-level semantic stage. The module performs complementary learning on multimodal features to achieve progressive feature extraction from commonality alignment to difference fusion; after the fusion stage, the AMG module is embedded to dynamically adjust the fusion weights according to the confidence of each modality; in the high-level modeling stage, a spatiotemporal modeling module is introduced to capture the spatiotemporal dependencies of cross-frame gait and fuse them with convolutional branch features.

[0043] in Module in Based on the modules, common features, appearance features and motion features are fused to obtain complementary enhanced features, and then linearly fused features are obtained through linear projection.

[0044] The linear fusion features output by the module are processed by pseudo-3D convolution and then fed into the AMG module. Through global average pooling and confidence prediction, the weights of common features, appearance features, motion features and complementary enhancement features are dynamically adjusted to obtain weighted fusion features.

[0045] The weighted fusion features are divided into two branches in the spatiotemporal modeling module. One branch is processed through a window shifting mechanism, and the other branch is processed by convolution. Then the outputs of the two branches are fused to obtain the spatiotemporal fusion features.

[0046] Among the three modalities of images, the optical flow map represents motion features and can effectively capture the motion information of the target; the silhouette map reflects the shape outline of the target; and the analytical map provides the structural information of the target. By stitching the silhouette map and the analytical map together, the advantages of both can be combined. The image after stitching the silhouette map and the analytical map represents the appearance features.

[0047] In this embodiment, the modal gait recognition model includes two convolutional layers, two Stage1 modules, three Stage2 modules, four Stage3 modules, and one... Module, one The system consists of a module, an AMG module, a spatiotemporal modeling module, and a gait recognition head; the specific process is as follows:

[0048] First, the motion feature map and appearance feature map are input into the convolutional layer and Stage1 module for feature extraction. Then, the fused features from the two branches are fed into... Module;

[0049] The module's output features are divided into three branches, each processed by the Stage2 module, and then the fused features from the three branches are fed into... Module;

[0050] The output features of the module are divided into four branches, which are processed by the Stage3 module respectively, and then the fused features of the four branches are sent to the AMG module.

[0051] The output features of the AMG module are processed by the spatiotemporal modeling module and then sent to the gait recognition head.

[0052] In the shallow stages of the backbone network, i.e., from the convolutional layers to Stage 1, the system first performs convolutional feature extraction on the input images of different modalities to obtain appearance features. With motion characteristics .

[0053] The Stage 1 module employs a two-dimensional convolutional structure for feature extraction. Specifically, it performs shallow feature encoding on the appearance features composed of silhouette and analytical modes, and spatial structure modeling on the motion features composed of optical flow modes. Specifically, the Stage 1 module first extracts local edge and contour information through a 3×3 two-dimensional convolution, then performs batch normalization and ReLU activation function for nonlinear mapping and feature distribution normalization, thereby enhancing the model's ability to represent human body shape, posture, and local texture.

[0054] To enhance structural alignment and differential representation between modalities, this invention builds upon existing... A progressive complementary integration mechanism is designed based on the modules. The module can generate modal common attention maps and modal difference attention maps to guide the feature attention regions for each modality. The module's appearance features are output by the Stage1 module. Motion characteristics For input, The output characteristics of the module include common features. Appearance features and motion characteristics The common features obtained after processing by the Stage2 module are obtained respectively. Appearance features and motion characteristics .

[0055] The Stage2 module employs a pseudo-3D convolutional structure for feature extraction, decomposing 3D convolution into a concatenated operation of spatial and temporal convolutions. This effectively extracts spatiotemporal features while reducing computational complexity. Specifically, the Stage2 module first extracts local spatial information through 2D convolution, and then uses 3D convolution in the temporal direction to capture cross-frame dynamic features, thereby achieving mid-level spatiotemporal modeling of multimodal gait information.

[0056] This invention proposes a progressive difference-complementary fusion module. After the Stage2 module is deployed in the backbone network, such as Figure 3 As shown, The processing flow is as follows:

[0057] Common features after processing by the Stage2 module Appearance features and motion characteristics To guide semantic understanding, progressive fusion is performed, and the output features of the Stage2 module are fused features. The expression is as follows:

[0058] ={ ;

[0059] Then the appearance features processed by the Stage2 module and motion characteristics The process involves two layers of 3D convolution, batch normalization, and nonlinear activation. Specifically, the appearance features of the Stage2 module are first processed. With motion characteristics Three-dimensional convolution is performed separately to achieve channel compression and local dynamic feature extraction. The calculation formula is as follows:

[0060] ;

[0061] ;

[0062] in express Features after 3D convolution processing; express Features after 3D convolution processing; Represents a three-dimensional convolution operation; This represents a learnable 3D convolutional kernel that focuses on low-level feature extraction. The kernel is assigned random values ​​during model initialization, and its weights are continuously optimized and updated during model training through backpropagation, thereby gradually learning the parameter distribution that best represents the input modality features.

[0063] The features after convolution are normalized and nonlinearly mapped to ensure distribution consistency and gradient stability among multiple modalities. The calculation formula is as follows:

[0064] ;

[0065] ;

[0066] in express Features after normalization and nonlinear mapping processing express Features after normalization and nonlinear mapping processing ( ) indicates that each channel is normalized to zero mean and unit variance. ( ) represents the activation function.

[0067] Then on and Three-dimensional convolution is performed separately to achieve feature recovery and semantic fusion. The calculation formula is as follows:

[0068] ;

[0069] ;

[0070] in express Features after 3D convolution processing; express Features after 3D convolution processing; It is a learnable 3D convolutional kernel responsible for high-level semantic aggregation.

[0071] After that and Batch normalization is performed separately, and the calculation formula is as follows:

[0072] ;

[0073] ;

[0074] in express Features after batch normalization express Features after batch normalization ( ) indicates batch normalization.

[0075] Then, common features after processing in the Stage2 module Under the guidance of , Weighted fusion is performed to achieve complementary and dynamic balance of cross-modal features, resulting in complementary enhanced features. Its expression is as follows:

[0076] ;

[0077] in , The modality adaptive weights are normalized using the Sigmoid function and their values ​​are limited to (0, 1) to represent the relative importance of each modality in the fusion process. During training, this parameter is automatically optimized through backpropagation to achieve adaptive control of the modality weights.

[0078] Subsequently, complementary enhancement features will be used. Features fused with the output features of the three Stage2 modules Linear projection yields linearly fused features. The calculation formula is as follows:

[0079] ;

[0080] in ( ) indicates a fusion operation; This represents the learnable channel transformation matrix, which generates initial weights through random distribution during model initialization. In each iteration, the system dynamically adjusts these weights based on loss feedback during the training phase. The weight distribution.

[0081] The module implements progressive feature modeling from shallow commonality extraction to mid-level difference complementarity, outputting linearly fused features. After processing, it becomes the input for the subsequent AMG module.

[0082] This invention introduces [the technology] into the shallow to mid-layers of the backbone network. – Union structure, firstly through The module extracts common features and separates differential features from three modalities: silhouette, parsing, and optical flow, generating a set of attention maps reflecting the consistency and differences between modalities. Subsequently... The module introduces cross-modal convolutional mapping and adaptive weighting mechanisms to progressively fuse common and dissimilar features layer by layer, achieving a gradual fusion process from "feature decomposition" to "complementary enhancement." This design strengthens information interaction and semantic collaboration between modalities, effectively solving the problems of insufficient feature coupling and low utilization of complementary information in traditional multimodal fusion. While maintaining the consistency of modal structure, it significantly enhances the collaborative expressive ability between silhouette, parsing, and optical flow modalities.

[0083] like Figure 4 As shown, the processing flow of the AMG module in this embodiment is as follows:

[0084] First receive The fused features of the module's output after processing by the Stage3 module Its expression is as follows:

[0085] ;

[0086] in express Common characteristics output by the Stage3 module express The appearance differences are output by the Stage3 module. express Motion difference features output by the Stage3 module express Complementary features output by the Stage3 module.

[0087] The Stage3 module is also built on a pseudo-3D convolutional structure and is used to perform deeper spatiotemporal modeling and semantic enhancement of mid-level features. Compared to the Stage2 module, the Stage3 module, while maintaining the pseudo-3D convolutional structure, achieves the capture of complex temporal dependencies in gait sequences by stacking more convolutional units and higher channel dimensions.

[0088] Then, the fusion features are analyzed. Global average pooling is applied to the four branches, and their gating weights are calculated using two fully connected layers and activation functions. The calculation formula is as follows:

[0089] ;

[0090] ;

[0091] in , express Features after global average pooling express Modal gating strength, express Features that undergo average pooling, where || represents feature concatenation; , This represents the parameter matrix of the two fully connected layers of the weight generation network. , C represents the number of channels. Nonlinear mapping and dimensionality compression for splicing features Used to generate initial weight values ​​for each modality, the parameters are automatically updated during training through backpropagation, thereby achieving adaptive learning of multimodal feature weights; This indicates a global average pooling operation. This represents the Sigmoid function. This represents the ReLU activation function.

[0092] Finally, the outputs of the four branches are weighted and fused based on the confidence level. The fusion formula is as follows:

[0093] ;

[0094] in This represents the weighted fusion feature after dynamic weighting of the four branches. This indicates element-wise multiplication.

[0095] This invention introduces an AMG module in the mid-to-high-level stages to replace the fixed-weight modal feature fusion method. This module performs global average pooling and channel compression on each modal feature and the fused features from the previous stage to generate a modal confidence vector, and learns the dynamic weight distribution through a Sigmoid gating function. The AMG module automatically adjusts the contribution ratio of each modality in the fusion process based on the quality, noise level, and modal correlation of the input data, thereby ensuring stable feature representation even under conditions of illumination changes, occlusion, and partial modal loss. This module achieves dynamic confidence perception and adaptive control of multimodal inputs, enhancing the system's robustness and generalization ability under cross-scene conditions.

[0096] , , These are all common features, used to distinguish common features obtained after processing at different stages;

[0097] , , These are all appearance features, used to distinguish the appearance features obtained after different stages of processing;

[0098] , , These are all motion features, used to distinguish motion features obtained after processing at different stages.

[0099] like Figure 5 As shown, the processing flow of the spatiotemporal modeling module in this embodiment is as follows:

[0100] First, receive the fusion features output from the AMG module. Then, it is fed into two parallel branches for spatiotemporal modeling: one branch is processed by the stage4 module; the other branch is processed by the Swing Transformer module; the spatiotemporal modeling expression is as follows:

[0101] ;

[0102] ;

[0103] in This represents the output features after processing by the Stage4 module, used to extract local spatial textures; This represents the output features processed by the Swin Transformer module, responsible for global spatiotemporal dependency modeling. This indicates that the Stage4 module performs convolutional processing. This indicates that the Swing Transformer module is handling the process.

[0104] Specifically, the Stage 4 module is in the high-level modeling stage of the model and is mainly responsible for the unified projection and identity feature generation of the deep features after multimodal fusion. While maintaining the pseudo-3D convolutional structure, this module introduces multi-layer convolutional units and global pooling operations to achieve comprehensive compression and abstract expression of spatiotemporal semantics.

[0105] The Swin Transformer module employs alternating stacking of local window multi-head self-attention (W-MSA) and sliding window multi-head attention (SW-MSA) in each stage. W-MSA performs self-attention computation within a fixed window to model local features; SW-MSA, on the other hand, achieves cross-window information interaction by translating the window position along the time and spatial axes, thereby capturing global spatiotemporal dependencies while maintaining computational efficiency. This alternating stacking method forms a "window shifting mechanism," aligning and fusing different feature sub-regions at each layer by shifting the window position, thereby improving feature continuity and global consistency.

[0106] Will and Spatiotemporal fusion features are obtained by fusing learnable weighted parameters. The fusion formula is as follows:

[0107] + ;

[0108] in , This represents a learnable weighted parameter used to control the relative contributions of the Stage4 module and the Swing Transformer module in the feature fusion process. During the training phase, the system dynamically optimizes this parameter based on loss feedback. The value of is chosen to achieve an optimal balance between maintaining local details and global semantic consistency in the fused features. To maintain stability... , After applying the Softmax normalization constraint, it satisfies + =1.

[0109] This invention introduces a spatiotemporal modeling module at a high-level stage, including a parallel Swing Transformer branch and a Stage4 branch. The Swing Transformer branch uses multi-head window self-attention and sliding window attention mechanisms to perform global dependency modeling on the temporal frames of the gait sequence, capturing cross-frame spatial-temporal correlation information. This branch provides temporal correlation and global attention features, while window attention aligns the change trends of each modality in the temporal dimension, improving cross-modal matching. The Stage4 branch preserves spatial boundaries and shape details. The two branches share input and gradient propagation paths, ensuring stable training. Furthermore, residual connections and hierarchical normalization strategies are used to achieve multi-scale fusion of local and global features.

[0110] Step 3. Train the multimodal gait recognition model built based on the training dataset in Step 1, and use the trained multimodal gait recognition model to perform gait recognition.

[0111] During training, feature alignment and fusion optimization are performed on the model to improve its discriminativeness and stability.

[0112] After training, the performance of the trained multimodal gait recognition model was validated using a test set. The test set was used to evaluate the model's recognition performance and robustness under different modal combinations and viewpoints, verifying the model's generalization ability and spatiotemporal feature representation ability in multimodal scenarios.

[0113] The test set gait input sequence is input into the trained multimodal gait recognition model to extract gait feature vectors, calculate the similarity between samples, realize identity recognition and feature matching output, and obtain the identity category, feature representation and matching score of the identified object.

[0114] Example 2

[0115] This embodiment 2 describes a gait recognition system based on multimodal fusion and spatiotemporal modeling. This gait recognition system based on multimodal fusion and spatiotemporal modeling is based on the same inventive concept as the gait recognition method based on multimodal fusion and spatiotemporal modeling in the above embodiment 1.

[0116] The gait recognition system based on multimodal fusion and spatiotemporal modeling in this embodiment includes the following modules:

[0117] The preprocessing module is used to acquire gait recognition images, preprocess the gait recognition images, and build a training dataset;

[0118] And a multimodal gait recognition module, used to build a multimodal gait recognition model based on the MultiGait++ architecture, which is designed with Modules, AMG module, and spatiotemporal modeling module;

[0119] in Module in Based on the modules, common features, appearance features and motion features are fused to obtain complementary enhanced features, and then linearly fused features are obtained through linear projection.

[0120] The linear fusion features output by the module are processed by pseudo-3D convolution and then fed into the AMG module. Through global average pooling and confidence prediction, the weights of common features, appearance features, motion features and complementary enhancement features are dynamically adjusted to obtain weighted fusion features.

[0121] The weighted fusion features are divided into two branches in the spatiotemporal modeling module. One branch is processed through a window shifting mechanism, and the other branch is processed by convolution. Then the outputs of the two branches are fused to obtain the spatiotemporal fusion features.

[0122] The multimodal gait recognition model is trained based on the training dataset, and then used to perform gait recognition.

[0123] It should be noted that any content not mentioned in the above-described functional modules of the system described in Embodiment 2 can be referred to the step description of the corresponding method in Embodiment 1 above, and will not be repeated in detail here.

[0124] Example 3

[0125] This embodiment 3 describes a computer device including a memory and one or more processors. Executable code is stored in the memory. When the processor executes the executable code, it implements the steps of the gait recognition method based on multimodal fusion and spatiotemporal modeling described in embodiment 1 above.

[0126] Example 4

[0127] This embodiment 4 describes a computer-readable storage medium storing a program that, when executed by a processor, implements the steps of the gait recognition method based on multimodal fusion and spatiotemporal modeling described in embodiment 1 above.

[0128] The computer-readable storage medium can be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc.

[0129] Of course, the above description is only a preferred embodiment of the present invention. The present invention is not limited to the above-described embodiments. It should be noted that any equivalent substitutions or obvious modifications made by those skilled in the art under the guidance of this specification fall within the scope of this specification and should be protected by the present invention.

Claims

1. A gait recognition method based on multimodal fusion and spatiotemporal modeling, characterized in that, Includes the following steps: Step 1. Acquire gait recognition images, preprocess the gait recognition images, and construct a training dataset; Step 2. Build a multimodal gait recognition model based on the MultiGait++ architecture. This model incorporates a progressive differential complementary fusion module. The modules, namely the modal adaptive gating module, are also known as the AMG module and the spatiotemporal modeling module; in The module in the common-difference feature extraction module, namely Based on the modules, common features, appearance features and motion features are fused to obtain complementary enhanced features, and then linearly fused features are obtained through linear projection. The linear fusion features output by the module are processed by pseudo-3D convolution and then fed into the AMG module. Through global average pooling and confidence prediction, the weights of common features, appearance features, motion features and complementary enhancement features are dynamically adjusted to obtain weighted fusion features. The weighted fusion features are divided into two branches in the spatiotemporal modeling module. One branch is processed through a window shifting mechanism, and the other branch is processed by convolution. Then the outputs of the two branches are fused to obtain the spatiotemporal fusion features. Step 3. Train the multimodal gait recognition model built based on the training dataset in Step 1, and use the trained multimodal gait recognition model to perform gait recognition.

2. The gait recognition method based on multimodal fusion and spatiotemporal modeling according to claim 1, characterized in that, In step 2, the multimodal gait recognition model includes two convolutional layers, two Stage 1 modules, three Stage 2 modules, four Stage 3 modules, and one... Module, one The system consists of a module, an AMG module, a spatiotemporal modeling module, and a gait recognition head; the specific process is as follows: First, the motion feature map and appearance feature map are input into the convolutional layer and Stage1 module for feature extraction. Then, the fused features from the two branches are fed into... Module; The module's output features are divided into three branches, each processed by the Stage2 module, and then the fused features from the three branches are fed into... Module; The output features of the module are divided into four branches, which are processed by the Stage3 module respectively, and then the fused features of the four branches are sent to the AMG module. The output features of the AMG module are processed by the spatiotemporal modeling module and then sent to the gait recognition head.

3. The gait recognition method based on multimodal fusion and spatiotemporal modeling according to claim 2, characterized in that, In step 2, The module's processing flow is as follows: The output characteristics of the module include common features. Appearance features and motion characteristics The common features obtained after processing by the Stage2 module are obtained respectively. Appearance features and motion characteristics ; First, we analyze the common characteristics of the three Stage2 modules after processing. Appearance features and motion characteristics If progressive fusion is performed, the fused features of the output features from the three Stage2 modules will be... The expression is as follows: ； Appearance features after processing the Stage2 module With motion characteristics Three-dimensional convolution is performed separately to achieve channel compression and local dynamic feature extraction. The calculation formula is as follows: ； ； in express Features after 3D convolution processing express Features after 3D convolution processing Represents a three-dimensional convolution operation. Represents a learnable 3D convolutional kernel; The features after convolution are normalized and non-linearly mapped, and the calculation formula is as follows: ； ； in express Features after normalization and nonlinear mapping processing express Features after normalization and nonlinear mapping processing ( This indicates that each channel has been normalized to zero mean and unit variance. ( () represents the activation function; Then on and The 3D convolution process is performed separately, and the calculation formula is as follows: ； ； in express Features after 3D convolution processing express Features after 3D convolution processing Represents a learnable 3D convolutional kernel; After that and Batch normalization is performed separately, and the calculation formula is as follows: ； ； in express Features after batch normalization express Features after batch normalization ( () indicates batch normalization processing; Then, common features after processing in the Stage2 module Under the guidance of , Weighted fusion is performed to obtain complementary and enhanced features. Its expression is as follows: ； in , These are modal adaptive weights, and their values ​​are limited to (0, 1). Subsequently, complementary enhancement features will be used. Features fused with the output features of the three Stage2 modules Linear projection yields linearly fused features. The calculation formula is as follows: ； in This represents the learnable channel transformation matrix. This indicates a fusion operation.

4. The gait recognition method based on multimodal fusion and spatiotemporal modeling according to claim 3, characterized in that, In step 2, the processing flow of the AMG module is as follows: First receive The fused features of the module's output after processing by the Stage3 module Its expression is as follows: ； in express Common characteristics output by the Stage3 module express The appearance features output by the Stage3 module. express Motion features output by the Stage3 module express Complementary features output by the Stage3 module; Then, the fusion features are analyzed. Global average pooling is applied to the four branches, and their gating weights are calculated using two fully connected layers and activation functions. The calculation formula is as follows: ； ； in , express Features after global average pooling express Modal gating strength, express Features after average pooling, where || represents feature concatenation. , This represents the parameter matrix of the two fully connected layers of the weight generation network. , C represents the number of channels. This indicates a global average pooling operation. This represents the Sigmoid function. Represents the ReLU activation function; Finally, the outputs of the four branches are weighted and fused based on the confidence level. The fusion formula is as follows: ； in This represents the weighted fusion feature after dynamic weighting of the four branches. This indicates element-wise multiplication.

5. The gait recognition method based on multimodal fusion and spatiotemporal modeling according to claim 4, characterized in that, In step 2, the processing flow of the spatiotemporal modeling module is as follows: First, the weighted fusion features output by the AMG module are received. Then, it is fed into two parallel branches for spatiotemporal feature modeling: one branch is processed by the Stage4 module; the other branch is processed by the Swing Transformer module; the spatiotemporal modeling expression is as follows: ； ； in This indicates the output characteristics after processing by the Stage4 module. This represents the output characteristics after processing by the Swing Transformer module. This indicates that the Stage4 module performs convolutional processing. This indicates that the SwinTransformer module is handling the process; The Swin Transformer module employs alternating stacking of local window multi-head self-attention and sliding window multi-head attention to form a window shifting mechanism. Specifically, local window multi-head self-attention performs self-attention calculation within a fixed window, while sliding window multi-head attention translates the window position along the time axis and spatial axis. Will and Spatiotemporal fusion features are obtained by fusing learnable weighted parameters. The fusion formula is as follows: + ； in , This indicates learnable weighted parameters, and + =1.

6. The gait recognition method based on multimodal fusion and spatiotemporal modeling according to claim 1, characterized in that, In step 1, the preprocessing process specifically includes: Gait recognition images generate data in three modalities, including silhouette images, analytical images, and optical flow images; each frame of the image is processed by size unification, frame order cropping, pixel normalization, and format conversion, and then stored as serialized pkl files according to modality; Then, based on the sample index recorded in the JSON file, the corresponding ID's pkl file is read synchronously and concatenated along the channel dimension to obtain the multimodal fusion input tensor X, whose expression is as follows: ； Where S represents the silhouette mode, P represents the analytical mode, F represents the optical flow mode, and B represents the batch. T represents the number of channels, H represents the number of frames, H represents the height, and W represents the width.

7. The gait recognition method based on multimodal fusion and spatiotemporal modeling according to claim 6, characterized in that, Optical flow diagrams represent motion characteristics; the combined image of silhouette and analytical diagrams represents appearance characteristics.

8. A gait recognition system based on multimodal fusion and spatiotemporal modeling, characterized in that, Includes the following modules: The preprocessing module is used to acquire gait recognition images, preprocess the gait recognition images, and build a training dataset; And a multimodal gait recognition module, used to build a multimodal gait recognition model based on the MultiGait++ architecture, which is designed with Modules, AMG module, and spatiotemporal modeling module; in Module in Based on the modules, common features, appearance features and motion features are fused to obtain complementary enhanced features, and then linearly fused features are obtained through linear projection. The linear fusion features output by the module are processed by pseudo-3D convolution and then fed into the AMG module. Through global average pooling and confidence prediction, the weights of common features, appearance features, motion features and complementary enhancement features are dynamically adjusted to obtain weighted fusion features. The weighted fusion features are divided into two branches in the spatiotemporal modeling module. One branch is processed through a window shifting mechanism, and the other branch is processed by convolution. Then the outputs of the two branches are fused to obtain the spatiotemporal fusion features. The multimodal gait recognition model is trained based on the training dataset, and then used to perform gait recognition.

9. A computer device, comprising a memory and one or more processors; characterized in that, The memory stores executable code, which, when executed by the processor, implements the steps of the gait recognition method based on multimodal fusion and spatiotemporal modeling as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon; characterized in that, When executed by a processor, the program is used to implement the steps of the gait recognition method based on multimodal fusion and spatiotemporal modeling as described in any one of claims 1 to 7.

Images