Robust gait recognition method and system based on 3D-CNN, inner convolution operator and transformer multi-fusion

Abstract

This invention discloses a robust gait recognition method and system based on 3D convolutional neural networks, involution operators, and Transformer multi-fusion. The method includes: performing data augmentation and spatiotemporal alignment preprocessing on binary silhouette sequences extracted from walking videos; extracting local spatiotemporal features through an improved 3D convolutional network fusing involution operators, frequency domain spectrum enhancement layers, and pseudo-3D residual modules; extracting global temporal features through an improved Vision Transformer incorporating viewpoint and walking state conditional embeddings; batch normalizing the two feature streams separately, concatenating them, and mapping them with a fully connected layer to obtain fused features; performing end-to-end training using a joint loss function including cross-entropy loss, triplet loss, and strongly constrained triplet loss; and during the inference phase, binarizing the fused features into a 512-byte irreversible template using a learnable step function and performing authentication using normalized Hamming distance.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and biometric recognition technology, specifically involving a robust gait recognition method and system based on 3D convolutional neural network, involution operator and Transformer multi-fusion. Background Technology

[0002] Gait recognition, due to its long-distance, non-cooperative acquisition characteristics, has significant application value in fields such as security monitoring, intelligent buildings, and criminal investigation. Current technologies primarily rely on binary silhouette sequences from walking videos, using deep neural networks to extract spatiotemporal features for identity recognition.

[0003] However, in open scenarios such as changes in clothing, carried items, severe occlusion, and cross-viewpoints, the silhouette appearance changes drastically, leading to a significant decrease in the recognition accuracy of existing methods. Especially under strict testing conditions excluding the same viewpoint, the false rejection rate (FRR) when the false alarm rate (FAR) is 0 is generally higher than 15%. Furthermore, due to differences in frame rates and gait cycles among different cameras, traditional methods struggle to achieve stable pixel-level spatiotemporal alignment, further exacerbating performance degradation under cross-viewpoints and low-quality sequences. In addition, existing methods often use floating-point feature vectors as the final template, which are not only typically several KB to tens of KB in size and time-consuming to match, but also pose a security risk of being reverse-engineered into the original silhouette, limiting real-time deployment on edge devices.

[0004] Therefore, there is an urgent need for a robust gait recognition method that can simultaneously achieve high-precision recognition, pixel-level spatiotemporal alignment, and secure, compact, and irreversible template generation in complex real-world scenarios.

[0005] A search revealed that Chinese invention patent CN103268500A discloses a gait recognition method robust to changes in walking state, belonging to the field of pattern recognition and machine learning technology. The method includes: S1: establishing a distance metric expression between gait feature matrices of different walking states; S2: during the training phase, establishing a similarity matrix and an objective optimization function in the training sample set, obtaining the respective transformation matrices for different walking states' gaits by decoupling the objective function, and performing two projections and coupled metric learning on the samples in matrix space and vector space to obtain the final feature set of the registration set samples; S3: during the recognition phase, when the walking state of the test sample is inconsistent with that of the registration set samples, performing a projection transformation using the trained transformation matrix, and finally using a nearest neighbor classifier to determine the category to which the gait sample belongs.

[0006] The technologies of the aforementioned patents and this application are compared as follows:

[0007] 1. Differences in spatiotemporal alignment and preprocessing mechanisms

[0008] Patent CN103268500A employs traditional spatial projection transformation and matrix metric methods. This method directly searches for similarity and performs spatial mapping at the level of the original gait feature matrix, lacking attention to the underlying pixel misalignment issues caused by differences in frame rates from different cameras or inconsistent gait cycles in the input video. This leads to feature distortion in low-quality sequences. In contrast, this application employs a spatiotemporal alignment preprocessing mechanism based on implicit functions. This patent utilizes continuous implicit functions composed of multilayer perceptrons (MLPs), taking normalized spatiotemporal coordinates as input, and achieving pixel-level alignment and arbitrary frame rate interpolation by minimizing self-reconstruction loss and perceptual loss. This mechanism breaks through the resolution and frame rate limitations of traditional discrete video frames from the ground up, generating a preprocessed sequence with frame number normalization and pixel-level alignment. This effectively solves the technical problem of performance degradation across viewpoints and in low-quality sequences caused by original spatiotemporal misalignment in patent CN103268500A, significantly improving the model's adaptability to open real-world scenes.

[0009] 2. The essential differences between feature extraction architecture and deep semantic modeling

[0010] The feature extraction in patent CN103268500A relies solely on subspace learning based on linear mapping (such as using spectral decomposition to obtain the transformation matrix), coupling samples from different walking states into the same image space, which is a shallow feature transformation. It struggles to fully explore and decouple the deep nonlinear spatiotemporal features of gait in complex scenes (such as severe occlusion or multiple objects carried). This patent innovatively introduces a dual-branch deep multi-fusion network of "3D-CNN + Transformer" into its feature extraction architecture. This patent extracts fine-grained local spatiotemporal features based on an improved 3D convolutional network that integrates involution, a frequency domain spectral enhancement layer, and a pseudo-3D residual module. Simultaneously, it utilizes an improved VisionTransformer with added viewpoint and walking state conditional embedding to extract global temporal features with a long receptive field. Specifically, this patent performs multi-scale stitching mapping between deep local spatiotemporal features and global temporal features, enabling it to keenly capture and overcome drastic silhouette appearance changes caused by clothing variations, carried objects, and cross-viewpoint variations. This enables the patent to maintain extremely high recognition accuracy even in complex, variable, and severely occluded real-world scenarios, a technical effect that the shallow matrix projection method of patent CN103268500A completely lacks.

[0011] 3. Differences in feature template generation morphology and recognition security

[0012] Patent CN103268500A ultimately extracts and uses traditional floating-point matrices or vectors for matching. Its distance metric is performed directly in a continuous real-number space; for example, the core feature distance evaluation of its objective function highly depends on the matrix trace metric formula in continuous space.

[0013]

[0014] This computational method not only results in feature templates requiring significant storage space and time-consuming high-dimensional matrix multiplication or Euclidean distance comparisons on massive datasets, but more critically, the floating-point feature space poses a privacy and security risk, as it can be reverse-engineered to reconstruct the original human gait silhouette. To balance comparison efficiency and data privacy, this patent designs an irreversible binarization template generation mechanism based on a learnable step function. During the inference phase, this patent applies a learnable step function to the fused high-dimensional continuous features for dimension-by-dimensional binarization, combined with a learnable threshold. (Initially set to 0) Perform discretization mapping to force the generation of an irreversible binary template with a fixed length of 4096 bits (i.e., 512 bytes). During authentication, complex floating-point distance calculations were completely abandoned, and instead, arbitrary binary templates were calculated. and Normalized Hamming distance between This design enables efficient verification. It completely transforms the feature comparison process mathematically into extremely fast bitwise operations (XOR operations), achieving millisecond-level retrieval of massive databases. Simultaneously, by introducing histograms during training... Figure 1 The consistency loss is used to ensure the balance of bit distribution, which in principle guarantees the complete irreversibility of the template, completely eliminates the risk of reverse leakage of original biometric features, and ensures absolute security and efficiency in long-distance non-cooperative biometric engineering applications. Summary of the Invention

[0015] The purpose of this invention is to provide a robust gait recognition method and system based on 3D-CNN, involution operator and Transformer multi-fusion, which significantly improves recognition accuracy in complex scenarios such as clothing changes, carried objects, occlusion and cross-viewpoints. At the same time, it generates a 512-byte irreversible binary template and reduces the number of model parameters and template storage volume, enabling efficient end-to-end deployment of edge devices.

[0016] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0017] A robust gait recognition method based on 3D convolutional neural networks, involution operators, and Transformer multi-fusion includes the following steps:

[0018] S1. Obtain the binary silhouette sequence extracted from the RGB video stream, and perform data augmentation processing on the binary silhouette sequence to generate a new silhouette sequence. ;

[0019] S2, the augmented silhouette sequence Perform spatiotemporal preprocessing of implicit functions, utilizing continuous implicit functions based on neural networks. Achieve pixel-level spatiotemporal alignment and interpolation to generate a preprocessed gait silhouette sequence. ;

[0020] S3. A feature extraction module based on 3D-CNN and involution operator for preprocessed gait silhouette sequences. Local spatiotemporal feature extraction is performed, and multi-layer semantic fusion is achieved by combining pyramid pooling and feature pyramid attention to generate local spatiotemporal features. ;

[0021] S4. Process the preprocessed gait silhouette sequence An improved Vision Transformer with added viewpoint and walking state conditional embeddings is input, and global temporal features are output. ;

[0022] S5. Local spatiotemporal features output in step S3 and the global temporal features output in step S4 Batch normalization was performed separately to obtain the normalized local spatiotemporal features. and normalized global temporal features ;

[0023] S6. Local spatiotemporal features With global temporal features By splicing along the channel dimension, the first splicing feature is obtained. Simultaneously, the normalized local spatiotemporal features With normalized global temporal features By splicing along the channel dimension, the second splicing feature is obtained. ;

[0024] S7, Regarding the first splicing feature Global average pooling is performed, followed by mapping through at least one fully connected layer to obtain robust spatiotemporal feature vectors. ;

[0025] S8. Using the joint loss function composed of the triplet loss function and the cross-entropy loss function, the result obtained in step S6 is... and the result obtained in step S7 To conduct training, reasoning, and verification.

[0026] As a preferred technical solution of the present invention, step S1 is specifically as follows:

[0027] S11. Given a binary silhouette sequence S as input, it is represented as:

[0028] ;

[0029] Where N is the batch size, T is the number of frames in the silhouette sequence, H×W is the resolution of each silhouette frame, and the number of channels is 1;

[0030] S12. After performing data augmentation on the binary silhouette sequence S, the silhouette sequence is obtained. The data augmentation process includes at least one of random horizontal flipping, random rotation transformation, and random perspective transformation.

[0031] As a preferred technical solution of the present invention, step S2 is specifically as follows:

[0032] S21. Augmented binary silhouette sequence Morphological filtering is applied, and dilation and erosion operations are performed sequentially to obtain the preprocessed silhouette I1;

[0033] S22. Calculate the relative displacement features of each pixel relative to the image centroid coordinates:

[0034] ;

[0035] in, The coordinates of the image center are...

[0036] The coordinates are then normalized to the [-1, 1] interval to generate a relative displacement feature map;

[0037] S23. Calculate the first-order frame difference. and second-order frame difference And a weighted combination method is used to generate velocity features. , where α1 and α2 are learnable weights or preset weights;

[0038] S24. Based on the 8-connected domain, select the coordinates of the neighboring pixels of each pixel, and calculate the neighborhood difference to obtain the spatial structure features. ;

[0039] S25. By splicing relative displacement features, velocity features, and spatial structure features along the channel dimension, the gait features of the extended channel are obtained. ;

[0040] S26. Hidden function neural networks constructed using multilayer perceptrons. With normalized spatiotemporal coordinates As input, predict the continuous pixel value p∈[0,1] at the corresponding position; optimize the latent function by minimizing the self-reconstruction loss and the perceptual loss, so that it can continuously represent and interpolate the extended feature I2 at any spatial and temporal resolution, thereby generating a preprocessed gait sequence with frame number normalization and pixel-level alignment. .

[0041] As a preferred technical solution of the present invention, step S3 is as follows:

[0042] S31. Construct a local spatiotemporal feature extraction branch and use an improved 3D-ResNet as the backbone network;

[0043] S32. The preprocessed gait sequence The input is an improved 3D-ResNet, which sequentially passes through an initial residual convolution to extract low-level structural information, a frequency domain spectrum enhancement layer to improve spatiotemporal resolution, a pseudo-3D residual block to extract high-level spatiotemporal features, and an involution operator to achieve parameter compression.

[0044] S33. At the end of the network, a pyramid pooling module and a feature pyramid attention module are cascaded. The feature pyramid attention module includes channel attention branches and spatial attention branches, which perform channel-dimensional weighting and spatial-dimensional weighting on the feature maps, respectively. After multi-scale pooling and attention fusion, local spatiotemporal features are output.

[0045] ;

[0046] Where T', H', and W' are the resolutions after downsampling, and C is the number of feature channels.

[0047] As a preferred embodiment of the present invention: in step S31, the improved 3D-ResNet includes:

[0048] The standard 3D convolutional layer is replaced with an involution operator, whose convolution kernel is dynamically generated by the channel information and spatial position of the input feature.

[0049] A frequency domain spectral enhancement layer is inserted into the residual block, and a learnable frequency domain weight is applied after performing a fast Fourier transform on the feature map, and then returned to the time domain by an inverse transform.

[0050] The remaining 3D convolutions are decomposed into pseudo-3D structures consisting of spatial 2D convolutions and temporal 1D convolutions.

[0051] As a preferred technical solution of the present invention, step S4 is specifically as follows:

[0052] S41. Construct a global temporal feature extraction branch, using Vision Transformer as the backbone network, and connecting the lightweight adaptation module AdaptFormer in parallel in each Transformer block.

[0053] S42. Extend the position encoding to a summation of standard sinusoidal position encoding, view conditional embedding, and walking state conditional embedding, wherein the view conditional embedding and walking state conditional embedding are generated by the learnable embedding layer based on the input view angle and walking state category, respectively.

[0054] S43. The preprocessed gait sequence After being flattened into a patch sequence, it is input into the VisionTransformer mentioned above. After processing by a multi-head self-attention mechanism and a lightweight adaptation module, the global temporal features are output:

[0055] ;

[0056] Where T' is the temporal length after serialization, and C is the feature dimension.

[0057] As a preferred technical solution of the present invention, step S8 is specifically as follows:

[0058] S81. During the training phase: A joint loss function consisting of cross-entropy loss, standard triplet loss, and optimized strongly constrained triplet loss is used to robustly train spatiotemporal feature vectors. and Perform end-to-end monitoring and optimization;

[0059] S82. During the inference phase, a learnable step function is applied to it for dimension-by-dimensional binarization to generate an irreversible binary template B with a length of 4096 bits.

[0060] S83. During identity verification: Calculate the normalized Hamming distance between any two binary templates B1 and B2. When the distance is less than a preset threshold, they are determined to be the same identity, thereby achieving a false rejection rate (FRR) of no more than 5% under the condition that the false alarm rate (FAR) is 0.

[0061] As a preferred technical solution of the present invention: in step S82, the learnable step function can be jointly optimized with network parameters during the training phase.

[0062] A robust gait recognition system based on 3D convolutional neural networks, involution operators, and Transformer multi-fusion includes a data augmentation module, a latent function spatiotemporal preprocessing module, a local spatiotemporal feature extraction module, a global temporal feature extraction module, a feature normalization and concatenation module, a fully connected mapping module, a joint loss optimization module, a learnable binarized template generation module, and a Hamming distance-based authentication module. The data augmentation module is connected to the latent function spatiotemporal preprocessing module, which is connected to the inputs of both the local spatiotemporal feature extraction module and the global temporal feature extraction module. The outputs of both modules are connected to the feature normalization and concatenation module, which is connected to the fully connected mapping module. The fully connected mapping module is also connected to the joint loss optimization module, the learnable binarized template generation module, and the Hamming distance-based authentication module.

[0063] As a preferred technical solution of the present invention, it further includes a binary template storage module, which is used to store an irreversible binary template with a length of 4096 bits.

[0064] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0065] This invention achieves pixel-level spatiotemporal alignment and arbitrary frame rate interpolation through a hidden function neural network, effectively solving the problems of gait cycle misalignment and viewpoint projection distortion. By integrating an improved 3D convolutional network with an involution operator, a frequency domain spectrum enhancement layer, and a pseudo-3D residual module, it significantly reduces the number of model parameters and computational overhead while fully preserving local spatiotemporal details. Introducing viewpoint conditional embedding and walking state conditional embedding into the Vision Transformer achieves effective long-range temporal modeling and significantly improves cross-viewpoint robustness. Through joint optimization of dual-path batch normalized feature concatenation, strongly constrained triplet loss, and a learnable step function, it generates an irreversible binary template of only 512 bytes in length without sacrificing recognition accuracy, greatly reducing template storage volume and comparison time, and fundamentally preventing the security risk of the template being reverse-constructed into the original silhouette. Extensive experiments on public datasets demonstrate that this invention significantly outperforms existing methods in recognition performance and security under complex open scenarios such as clothing changes, carried objects, occlusion, and cross-viewpoint conditions, exhibiting higher robustness and practical deployment value. Attached Figure Description

[0066] Figure 1 This is a flowchart of a robust gait recognition method based on 3D convolutional neural networks, involution operators, and Transformer multi-fusion.

[0067] Figure 2 This is a flowchart of the algorithm of the present invention;

[0068] Figure 3 This is a diagram showing the successful operation of a robust gait recognition system based on 3D convolutional neural networks, involution operators, and Transformer multi-fusion.

[0069] Figure 4 This is a human feature map extracted by the present invention. Detailed Implementation

[0070] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0071] In this embodiment, the resolution of the input binary silhouette image sequence is uniformly adjusted to 64×44, the sequence length is normalized to 60 frames through implicit temporal resampling, the batch size is N=32, and training is performed on 4 NVIDIA RTX 4090 GPUs.

[0072] like Figure 1-2 As shown, this embodiment provides a robust gait recognition method based on 3D convolutional neural networks, involution operators, and Transformer multi-fusion, specifically including the following steps:

[0073] S1. Obtain the binary silhouette sequence extracted from the RGB video stream, and perform data augmentation on the binary silhouette sequence to enhance the model's generalization ability to environmental noise and covariates, generating a silhouette sequence. ;

[0074] Specifically as follows:

[0075] S11. Given a binary silhouette sequence S as input, it is represented as:

[0076] ;

[0077] Where N is the batch size, T is the number of frames in the silhouette sequence, H×W is the resolution of each silhouette frame, and the number of channels is 1 (0 represents the background, and 1 represents the foreground human silhouette).

[0078] S12. After performing data augmentation on the binary silhouette sequence S, the silhouette sequence is obtained. The data augmentation process includes at least one of random horizontal flipping, random rotation transformation, and random perspective transformation.

[0079] The data augmentation process is as follows:

[0080] When random number When the value exceeds the preset flip threshold, a horizontal flip operation is performed so that the pixel value at coordinate (h, w) of the flipped frame t is equal to the pixel value at coordinate (H-1-h, w) of the original sequence frame t.

[0081] The rotation transformation is performed with a preset probability, the rotation angle is randomly sampled within a preset range, and the boundary pixels are filled using bilinear interpolation.

[0082] Perform a perspective transformation, whereby the perspective transformation matrix M is randomly generated based on the affine transformation parameters and applied to the rotated sequence;

[0083] The above processing yields the data-enhanced silhouette sequence. This data augmentation operation is used to improve the model's ability to generalize to changes in walking direction, viewpoint, and lens distortion.

[0084] Among them, the flip threshold, rotation angle range, and perspective transformation matrix are randomly sampled in real time during training to enhance the model's generalization ability to real-world scene factors such as changes in walking direction, differences in camera perspective, and lens distortion.

[0085] S2, the augmented silhouette sequence Perform spatiotemporal preprocessing of implicit functions, utilizing continuous implicit functions based on neural networks. Achieve pixel-level spatiotemporal alignment and interpolation to generate a preprocessed gait silhouette sequence. ;

[0086] Specifically as follows:

[0087] S21. Augmented binary silhouette sequence Apply morphological filtering (structural element is 3×3), and perform dilation and erosion operations in sequence to obtain the preprocessed silhouette I1;

[0088] S22. Calculate the relative displacement features of each pixel relative to the image centroid coordinates:

[0089] ;

[0090] in, The coordinates of the image center are...

[0091] The coordinates are then normalized to the [-1, 1] interval to generate a relative displacement feature map;

[0092] S23. Calculate the first-order frame difference. and second-order frame difference And a weighted combination method is used to generate velocity features. , where α1 and α2 are learnable weights or preset weights;

[0093] S24. Based on the 8-connected domain, select the coordinates of the neighboring pixels of each pixel, and calculate the neighborhood difference to obtain the spatial structure features. ;

[0094] S25. By splicing relative displacement features, velocity features, and spatial structure features along the channel dimension, the gait features of the extended channel are obtained. ;

[0095] S26. Hidden function neural network constructed using multilayer perceptron (MLP) With normalized spatiotemporal coordinates As input, predict the continuous pixel value p∈[0,1] at the corresponding position; optimize the latent function by minimizing the self-reconstruction loss and the perceptual loss, so that it can continuously represent and interpolate the extended feature I2 at any spatial and temporal resolution, thereby generating a preprocessed gait sequence with frame number normalization and pixel-level alignment. .

[0096] S3. A feature extraction module based on 3D-CNN and involution operator for preprocessed gait silhouette sequences. Local spatiotemporal features are extracted, and multi-layer semantic fusion is achieved by combining pyramid pooling (PPM) and feature pyramid attention (FPA) to generate local spatiotemporal features. ;

[0097] Specifically as follows:

[0098] S31. Construct a local spatiotemporal feature extraction branch and use an improved 3D-ResNet as the backbone network;

[0099] S32. The preprocessed gait sequence The input is an improved 3D-ResNet, which sequentially passes through an initial residual convolution to extract low-level structural information, a frequency domain spectrum enhancement layer to improve spatiotemporal resolution, a pseudo-3D residual block to extract high-level spatiotemporal features, and an involution operator to achieve parameter compression.

[0100] S33. At the end of the network, a pyramid pooling module (PPM) and a feature pyramid attention module (FPA) are cascaded. The feature pyramid attention module (FPA) includes channel attention branches and spatial attention branches, which perform channel dimension weighting and spatial dimension weighting on the feature maps, respectively. After multi-scale pooling and attention fusion, the local spatiotemporal features are output.

[0101] ;

[0102] Where T', H', and W' are the resolutions after downsampling, and C is the number of feature channels.

[0103] The improvements of the improved 3D-ResNet are as follows:

[0104] Some standard 3D convolutional layers are replaced with involution operators, whose convolution kernels are dynamically generated by the channel information and spatial location of the input features.

[0105] A frequency domain spectral enhancement layer is inserted into the residual block, and a learnable frequency domain weight is applied after performing a fast Fourier transform on the feature map, and then returned to the time domain by an inverse transform.

[0106] The remaining 3D convolutions are decomposed into pseudo-3D structures consisting of spatial 2D convolutions and temporal 1D convolutions.

[0107] S4. Process the preprocessed gait silhouette sequence An improved Vision Transformer with added viewpoint and walking state conditional embeddings is input, and global temporal features are output. ;

[0108] Specifically as follows:

[0109] S41. Construct a global temporal feature extraction branch, using Vision Transformer as the backbone network, and connecting the lightweight adaptation module AdaptFormer in parallel in each Transformer block.

[0110] S42. Extend the position encoding to a summation of standard sinusoidal position encoding, view conditional embedding, and walking state conditional embedding, wherein the view conditional embedding and walking state conditional embedding are generated by the learnable embedding layer based on the input view angle and walking state category, respectively.

[0111] S43. The preprocessed gait sequence After being flattened into a patch sequence, it is input into the VisionTransformer mentioned above. After processing by a multi-head self-attention mechanism and a lightweight adaptation module, the global temporal features are output:

[0112] ;

[0113] Where T' is the temporal length after serialization, and C is the feature dimension.

[0114] S5. Local spatiotemporal features output in step S3 and the global temporal features output in step S4 Batch normalization was performed separately to obtain the normalized local spatiotemporal features. and normalized global temporal features ;

[0115] S6. Local spatiotemporal features With global temporal features By splicing along the channel dimension, the first splicing feature is obtained. Simultaneously, the normalized local spatiotemporal features With normalized global temporal features By splicing along the channel dimension, the second splicing feature is obtained. ;

[0116] S7, Regarding the first splicing feature Global average pooling is performed, followed by mapping through at least one fully connected layer to obtain robust spatiotemporal feature vectors. ;

[0117] S8. Using the joint loss function composed of the triplet loss function and the cross-entropy loss function, the result obtained in step S6 is... and the result obtained in step S7 To conduct training, reasoning, and verification;

[0118] Specifically as follows:

[0119] S81. During the training phase: A joint loss function consisting of cross-entropy loss, standard triplet loss, and optimized strongly constrained triplet loss is used to robustly train spatiotemporal feature vectors. and Perform end-to-end monitoring and optimization;

[0120] S82. During the inference phase, a learnable step function is applied to it for dimension-wise binarization to generate an irreversible binary template B with a length of 4096 bits (i.e. 512 bytes). The learnable step function is jointly optimized with the network parameters during the training phase.

[0121] S83. During identity verification: Calculate the normalized Hamming distance between any two binary templates B1 and B2. When the distance is less than a preset threshold, they are determined to be the same identity, thereby achieving a false rejection rate (FRR) of no more than 5% under the condition that the false alarm rate (FAR) is 0.

[0122] The present invention will now be described in detail with reference to specific examples.

[0123] In specific implementation, step S1 includes:

[0124] S11. Given a binary silhouette sequence as input. , where 0 represents the background and 1 represents the foreground human figure outline.

[0125] S12. Apply data augmentation in real time for each batch:

[0126] When random number When the value exceeds a preset flip threshold (preferably 0.5), a horizontal flip is performed.

[0127]

[0128] A random rotation transformation is performed with a preset probability (preferably 0.5), and the rotation angle is... Uniform sampling is performed within the range of [-15°, 15°], and bilinear interpolation is used to fill the boundaries.

[0129] Then, a perspective transformation is applied, with the transformation matrix M generated by random affine parameters and the distortion coefficient preferably within 0.1, to simulate projection deformation in a real scene.

[0130] The above processing yields the data-enhanced silhouette sequence. .

[0131] In specific implementation, step S2 includes:

[0132] S21. A 5-layer MLP (with hidden layer dimensions of 128, 256, 256, 256, and 128 respectively, and the activation function being SiLU) is used to construct the hidden function. Input normalized spatiotemporal coordinates Output continuous pixel values .

[0133] S22. Optimize the following self-reconstruction loss:

[0134]

[0135] in For L1 loss, For the L2 distance of pre-trained VGG-16 features, L1 loss due to expansion-corrosion difference (3×3 structural elements). , Weights can be learned or preset.

[0136] S23. Using the AdamW optimizer (initial learning rate preferred to be 1e-3, weight decay 5e-4), the implicit function is obtained after training for 1000 iterations.

[0137] S24. During the backbone training phase, the original binary image and the normalized coordinate image are... First- and second-order velocity fields and morphological structural features (expansion-corrosion difference) are spliced ​​along the channels to form a 6-channel extended feature. The implicit function performs continuous reconstruction and arbitrary frame rate interpolation to generate a preprocessed sequence with pixel-level alignment and frame number normalization. .

[0138] In specific implementation, step S3 includes:

[0139] S31, the initial 3D residual convolutional layer has a kernel size of 3×3×3, a stride of (1,2,2), a padding of 1, and an output channel of 64;

[0140] S32. Insert a frequency domain spectral enhancement layer after each residual block:

[0141]

[0142] in For learnable frequency domain weights, The high-frequency enhancement factor is preferably 0.5.

[0143] S33. Decompose the standard 3D convolution into a pseudo-3D structure consisting of temporal 1D convolution (3×1×1) and spatial 2D convolution (1×3×3), stack multiple residual blocks, and gradually expand the channels.

[0144] S34. Apply the convolution operator to the features of each frame. The convolution kernel K is determined by a lightweight MLP (input channel number, output k). ² The parameters are dynamically predicted (×g) and L1 sparse regularization is applied (the coefficient is preferably 0.01) to achieve a significant reduction in the number of parameters.

[0145] S35. At the end of the network, cascade a pyramid pooling module PPM (pooling scale {1,2,3,6}) and a feature pyramid attention module FPA (including channel attention and spatial attention) to output local spatiotemporal features. .

[0146] In specific implementation, step S4 includes:

[0147] S41, will Flatten using an 8×8 patch and linearly project to dimension 64;

[0148] S42, Position coding extended to:

[0149]

[0150] in and These are learnable embedding layers with dimensions of 16 and 8, respectively.

[0151] S43. Integrate the AdaptFormer lightweight adaptation module (structure: downsampling-multi-head attention-upsampling) in parallel into each Transformer block to output global temporal features. .

[0152] In specific implementation, step S8 includes:

[0153] S81, The joint loss function is:

[0154]

[0155] in The loss is a strongly constrained triplet loss with dynamic margins, where α and β are learnable or preset weights.

[0156] S82. A learnable step function is used during the inference phase. Dimension-by-dimensional binarization:

[0157]

[0158] in To set a learnable threshold (initially 0), a histogram is added during training. Figure 1 Consistency loss ensures bit balance, ultimately generating a 4096-bit (512-byte) binary template B.

[0159] S83. During identity verification, calculate the normalized Hamming distance d(B1,B2). If d is less than the preset threshold (preferably 0.8), return the same identity.

[0160] like Figure 3-4 As shown, the present invention also provides a robust gait recognition system based on 3D convolutional neural networks, involution operators, and Transformer multi-fusion, for performing the methods described in the above embodiments, as follows:

[0161] The system includes a data augmentation module, a latent function spatiotemporal preprocessing module, a local spatiotemporal feature extraction module, a global temporal feature extraction module, a feature normalization and concatenation module, a fully connected mapping module, a joint loss optimization module, a learnable binarization template generation module, and a Hamming distance-based authentication module. The data augmentation module is connected to the latent function spatiotemporal preprocessing module. The latent function spatiotemporal preprocessing module is connected to the inputs of the local spatiotemporal feature extraction module and the global temporal feature extraction module, respectively. The outputs of the local spatiotemporal feature extraction module and the global temporal feature extraction module are connected to the feature normalization and concatenation module. The feature normalization and concatenation module is connected to the fully connected mapping module. The fully connected mapping module is connected to the joint loss optimization module, the learnable binarization template generation module, and the Hamming distance-based authentication module, respectively.

[0162] It also includes a binary template storage module, which is used to store an irreversible binary template with a length of 4096 bits.

[0163] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A robust gait recognition method based on 3D convolutional neural networks, involution operators, and Transformer multi-fusion, characterized in that, Includes the following steps: S1. Obtain the binary silhouette sequence extracted from the RGB video stream, and perform data augmentation processing on the binary silhouette sequence to generate a new silhouette sequence. ; S2, the augmented silhouette sequence Perform spatiotemporal preprocessing of implicit functions, utilizing continuous implicit functions based on neural networks. Achieve pixel-level spatiotemporal alignment and interpolation to generate a preprocessed gait silhouette sequence. ; S3. A feature extraction module based on 3D-CNN and involution operator for preprocessed gait silhouette sequences. Local spatiotemporal feature extraction is performed, and multi-layer semantic fusion is achieved by combining pyramid pooling and feature pyramid attention to generate local spatiotemporal features. ; S4. Process the preprocessed gait silhouette sequence An improved Vision Transformer with added viewpoint and walking state conditional embeddings is input, and global temporal features are output. ; S5. Local spatiotemporal features output in step S3 and the global temporal features output in step S4 Batch normalization was performed separately to obtain the normalized local spatiotemporal features. and normalized global temporal features ; S6. Local spatiotemporal features With global temporal features By splicing along the channel dimension, the first splicing feature is obtained. Simultaneously, the normalized local spatiotemporal features With normalized global temporal features By splicing along the channel dimension, the second splicing feature is obtained. ; S7, Regarding the first splicing feature Global average pooling is performed, followed by mapping through at least one fully connected layer to obtain robust spatiotemporal feature vectors. ; S8. Using the joint loss function composed of the triplet loss function and the cross-entropy loss function, the result obtained in step S6 is... and the result obtained in step S7 To conduct training, reasoning, and verification.

2. The robust gait recognition method based on 3D convolutional neural network, involution operator and Transformer multi-fusion as described in claim 1, characterized in that, Step S1 is as follows: S11. Given a binary silhouette sequence S as input, it is represented as: ； Where N is the batch size, T is the number of frames in the silhouette sequence, H×W is the resolution of each silhouette frame, and the number of channels is 1; S12. After performing data augmentation on the binary silhouette sequence S, the silhouette sequence is obtained. The data augmentation process includes at least one of random horizontal flipping, random rotation transformation, and random perspective transformation.

3. The robust gait recognition method based on 3D convolutional neural network, involution operator and Transformer multi-fusion as described in claim 1, characterized in that, Step S2 is as follows: S21. Augmented binary silhouette sequence Morphological filtering is applied, and dilation and erosion operations are performed sequentially to obtain the preprocessed silhouette I1; S22. Calculate the relative displacement features of each pixel relative to the image centroid coordinates: ； in, The coordinates of the image center are... The coordinates are then normalized to the [-1, 1] interval to generate a relative displacement feature map; S23. Calculate the first-order frame difference. and second-order frame difference And a weighted combination method is used to generate velocity features. , where α1 and α2 are learnable weights or preset weights; S24. Based on the 8-connected domain, select the coordinates of the neighboring pixels of each pixel, and calculate the neighborhood difference to obtain the spatial structure features. ; S25. By splicing relative displacement features, velocity features, and spatial structure features along the channel dimension, the gait features of the extended channel are obtained. ; S26. Hidden function neural networks constructed using multilayer perceptrons. With normalized spatiotemporal coordinates As input, predict the continuous pixel value p∈[0,1] at the corresponding position; optimize the latent function by minimizing the self-reconstruction loss and the perceptual loss, so that it can continuously represent and interpolate the extended feature I2 at any spatial and temporal resolution, thereby generating a preprocessed gait sequence with frame number normalization and pixel-level alignment. .

4. The robust gait recognition method based on 3D convolutional neural network, involution operator and Transformer multi-fusion as described in claim 1, characterized in that, Step S3 is as follows: S31. Construct a local spatiotemporal feature extraction branch and use an improved 3D-ResNet as the backbone network; S32. The preprocessed gait sequence The input is an improved 3D-ResNet, which sequentially passes through an initial residual convolution to extract low-level structural information, a frequency domain spectrum enhancement layer to improve spatiotemporal resolution, a pseudo-3D residual block to extract high-level spatiotemporal features, and an involution operator to achieve parameter compression. S33. At the end of the network, a pyramid pooling module and a feature pyramid attention module are cascaded. The feature pyramid attention module includes channel attention branches and spatial attention branches, which perform channel-dimensional weighting and spatial-dimensional weighting on the feature maps, respectively. After multi-scale pooling and attention fusion, local spatiotemporal features are output. ； Where T', H', and W' are the resolutions after downsampling, and C is the number of feature channels.

5. The robust gait recognition method based on 3D convolutional neural network, involution operator and Transformer multi-fusion as described in claim 4, characterized in that, In step S31, the improved 3D-ResNet includes: The standard 3D convolutional layer is replaced with an involution operator, whose convolution kernel is dynamically generated by the channel information and spatial position of the input feature. A frequency domain spectral enhancement layer is inserted into the residual block, and a learnable frequency domain weight is applied after performing a fast Fourier transform on the feature map, and then returned to the time domain by an inverse transform. The remaining 3D convolutions are decomposed into pseudo-3D structures consisting of spatial 2D convolutions and temporal 1D convolutions.

6. The robust gait recognition method based on 3D convolutional neural network, involution operator and Transformer multi-fusion as described in claim 1, characterized in that, Step S5 is as follows: Step S4 is as follows: S41. Construct a global temporal feature extraction branch, using Vision Transformer as the backbone network, and connecting the lightweight adaptation module AdaptFormer in parallel in each Transformer block. S42. Extend the position encoding to a summation of standard sinusoidal position encoding, view conditional embedding, and walking state conditional embedding, wherein the view conditional embedding and walking state conditional embedding are generated by the learnable embedding layer based on the input view angle and walking state category, respectively. S43. The preprocessed gait sequence After being flattened into a patch sequence, it is input into the Vision Transformer mentioned above. After processing by a multi-head self-attention mechanism and a lightweight adaptation module, the global temporal features are output: ； Where T' is the temporal length after serialization, and C is the feature dimension.

7. The robust gait recognition method based on 3D convolutional neural network, involution operator and Transformer multi-fusion as described in claim 1, characterized in that, Step S8 is as follows: S81. During the training phase: A joint loss function consisting of cross-entropy loss, standard triplet loss, and optimized strongly constrained triplet loss is used to robustly train spatiotemporal feature vectors. and Perform end-to-end monitoring and optimization; S82. During the inference phase, a learnable step function is applied to it for dimension-by-dimensional binarization to generate an irreversible binary template B with a length of 4096 bits. S83. During identity verification: Calculate the normalized Hamming distance between any two binary templates B1 and B2. When the distance is less than a preset threshold, they are determined to be the same identity, thereby achieving a false rejection rate (FRR) of no more than 5% under the condition that the false alarm rate (FAR) is 0.

8. The robust gait recognition method based on 3D convolutional neural network, involution operator and Transformer multi-fusion as described in claim 7, characterized in that, In step S82, the learnable step function is jointly optimized with network parameters during the training phase.

9. A robust gait recognition system based on 3D convolutional neural networks, involution operators, and Transformer multi-fusion, characterized in that, The system includes a data augmentation module, a latent function spatiotemporal preprocessing module, a local spatiotemporal feature extraction module, a global temporal feature extraction module, a feature normalization and concatenation module, a fully connected mapping module, a joint loss optimization module, a learnable binarization template generation module, and a Hamming distance-based authentication module. The data augmentation module is connected to the latent function spatiotemporal preprocessing module. The latent function spatiotemporal preprocessing module is connected to the inputs of the local spatiotemporal feature extraction module and the global temporal feature extraction module, respectively. The outputs of the local spatiotemporal feature extraction module and the global temporal feature extraction module are connected to the feature normalization and concatenation module. The feature normalization and concatenation module is connected to the fully connected mapping module. The fully connected mapping module is connected to the joint loss optimization module, the learnable binarization template generation module, and the Hamming distance-based authentication module, respectively.

10. The robust gait recognition system based on 3D convolutional neural network, involution operator and Transformer multi-fusion as described in claim 9, characterized in that, It also includes a binary template storage module, which is used to store an irreversible binary template with a length of 4096 bits.

Images