Dynamically densely connected spatiotemporal feature decoupling network for identifying cross-view gait

Abstract

This invention discloses a dynamically densely connected spatiotemporal feature decoupling network for recognizing cross-view gait, relating to the field of computer vision. It includes: an initial feature processing module, a dynamically dense spatiotemporal decoupling feature extraction module, and a feature enhancement processing module. The module uses dense spatiotemporal feature decoupling blocks and concatenation operations to achieve the sharing of shallow and deep network features, thereby addressing the problem of insufficient representation ability. Simultaneously, it employs an enhanced convolutional block attention mechanism to allow the network to focus on more important gait features. Finally, the feature enhancement processing module processes the five-dimensional feature mapping into multiple lateral features and performs batch standardization of the features to enhance representation ability and model generalization ability. This allows for the mining of correlations between shallow and deep features, alleviating the problem of insufficient information representation ability.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a dynamic densely connected spatiotemporal feature decoupling network for recognizing cross-view gait. Background Technology

[0002] Biometric technology refers to the technology of identifying individuals by closely combining computer science with principles of optics, acoustics, biosensors, and biostatistics, utilizing inherent physiological and behavioral characteristics of the human body. Gait recognition, on the other hand, is a technology that identifies individuals based on biometric features such as the silhouette, stride, and arm swing of a person while walking. It has been successfully applied to user identification, sports science, and influence analysis. Due to the advantages of gait features, such as ease of collection, long-distance operation, non-contact nature, and difficulty in spoofing, existing gait recognition methods utilize gait features to replace traditional biometrics, achieving exciting performance and attracting widespread attention.

[0003] Over the past two decades, numerous gait recognition methods have been developed to address the aforementioned problems. Prior to 2015, research on gait recognition primarily focused on manually extracting gait features, then generating templates through modeling, and finally achieving gait recognition through template matching. However, traditional methods did not yield significant results. Since the introduction of deep learning-based gait recognition methods in 2015, traditional machine learning methods have gradually been abandoned, and increasing attention has been paid to deep learning-based gait recognition approaches.

[0004] Traditional machine learning-based handcrafted feature extraction methods have limitations due to insufficient acquisition of global and temporal information. In contrast, most deep learning-based methods use raw gait sequence data as input, then extract spatiotemporal features of gait through convolutional neural networks, and finally predict the category using feature similarity metrics. Therefore, they can better focus on the spatiotemporal information of gait. Gait recognition methods based on ordered sequences are mainly divided into decoupled spatiotemporal modeling and model construction using 3D convolution. Decoupled spatiotemporal modeling generally refers to separating the temporal and spatial dimensions for feature extraction. The common approach is to first extract the spatial features of each gait frame using a two-dimensional convolutional network, and then use temporal modeling to extract the temporal gait information. For example, Zhang et al. used a Long Short-Term Memory network as a temporal attention model to obtain the temporal information of gait. Li et al. also proposed a gait recognition method called GaitSlice, which modeled the spatiotemporal information of gait in different ways. Spatially, GaitSlice utilizes a spatial feature extractor to obtain spatial features of gait, temporally it introduces a residual frame attention mechanism to obtain key gait frames, and finally uses maximum aggregation to obtain temporal features of gait.

[0005] Furthermore, to better utilize spatiotemporal correlation information, 3D convolutional networks are increasingly being applied to gait recognition. For example, Lin et al. proposed a novel multi-temporal-scale gait recognition method based on 3D convolution. MT3D has two feature extraction streams, one for small temporal scales and one for large temporal scales. It uses 3DCNN to extract spatiotemporal information from gait information at two different temporal scales, then uses frame pooling to fuse the information from the two scales, and finally uses Euclidean distance to calculate the distance between features to predict the category. Huang et al. proposed a new perspective, suggesting that different body parts exhibit different movement patterns during walking activities, and proposed a gait recognition method called 3D Local Convolutional Neural Network. This method segments gait frames into different locally ordered sequences based on body parts (such as legs, arms, etc.), then uses 3DCNN for spatiotemporal feature extraction, and finally concatenates all features to calculate feature similarity to predict the category.

[0006] However, existing cross-view gait recognition methods based on convolutional neural networks still have the following problems. First, most previous work on gait recognition has adopted a single spatiotemporal modeling approach and stacked a series of modules to obtain spatiotemporal features. Directly performing spatiotemporal modeling followed by feature extraction ignores the spatiotemporal decoupling features of gait, making it difficult to comprehensively learn the spatiotemporal decoupling feature representation of gait sequence data, thus having certain limitations. Another approach, which models time and space separately, can obtain the spatiotemporal decoupling features of gait, but often ignores the spatiotemporal correlation information of gait, leading to insufficient representation of gait feature information. In addition, using traditional 3D convolutional modules for modeling also leads to some problems. On the one hand, it may result in too many network parameters, leading to huge consumption of computational resources; on the other hand, it may introduce unnecessary redundant information or cause insufficient feature representation capabilities. Finally, previous techniques often overlooked the issue of sharing shallow and deep features in a network. Shallow feature maps are relatively large and contain more detailed features (which is more advantageous for the network to obtain detailed features), while deep feature maps are relatively small and contain more abstract semantic features (which is more advantageous for classification). However, most previous methods only considered deep features, thus often ignoring some more detailed features, which are important information for distinguishing the identity of the subjects.

[0007] Based on this, the present invention aims to provide a dynamic densely connected spatiotemporal feature decoupling network for identifying cross-view gait, which can mine the correlation between shallow and deep features and alleviate the problem of insufficient information representation ability. Summary of the Invention

[0008] To overcome the shortcomings of existing technologies, this invention aims to provide a dynamically densely connected spatiotemporal feature decoupling network for recognizing cross-view gait. It utilizes dense spatiotemporal feature decoupling blocks and concatenation operations to achieve the sharing of features between shallow and deep networks, thereby addressing the problem of insufficient representational ability. Simultaneously, an enhanced convolutional block attention mechanism is employed to allow the network to focus on more important gait features. Finally, in the feature enhancement processing module, the five-dimensional feature map is processed into multiple lateral features, and batch standardization of these features is performed to enhance representational ability and model generalization ability.

[0009] To achieve the above objectives, the present invention provides the following technical solution: A dynamically densely connected spatiotemporal feature decoupling network for identifying cross-view gait, including: The initial feature processing module extracts preliminary features from the data and reduces the time dimension to minimize computational resource consumption. The dynamic dense spatiotemporal decoupling feature extraction module includes a dense spatiotemporal feature decoupling block and an enhanced convolutional block attention mechanism. After the initial feature processing module obtains shallow spatiotemporal features as input, the dense spatiotemporal feature decoupling block realizes the sharing of shallow network features and deep network features, enhancing the representation power of the features; at the same time, the enhanced convolutional block attention mechanism is used to improve the model's attention to gait walking targets. The feature enhancement processing module includes temporal pooling, global average pooling, global max pooling, fully connected layers, and the BNNeck component. It transforms the features extracted by the network into multiple lateral gait features to enhance the representation ability and model generalization ability, and remove redundant information.

[0010] In this invention, the dynamic dense spatiotemporal decoupling feature extraction module includes three dense spatiotemporal feature decoupling blocks, three enhanced convolutional block attention mechanisms, and a max pooling module; the specific data processing procedure of the dynamic dense spatiotemporal decoupling feature extraction module is as follows: The output of the initial feature processing module is used as the input of the first dense spatiotemporal feature decoupling block to obtain the output of the first dense spatiotemporal feature decoupling block; The input and output of the first dense spatiotemporal feature decoupling block are concatenated and used as the input of the first enhanced convolutional block attention mechanism, thereby obtaining the output of the first enhanced convolutional block attention mechanism. The output of the first enhanced convolutional block attention mechanism is used as the input to the max pooling module to obtain the output of the max pooling module; The output of the max pooling module is used as the input of the second dense spatiotemporal feature decoupling block to obtain the output of the second dense spatiotemporal feature decoupling block; The input and output of the second dense spatiotemporal feature decoupling block are concatenated and used as the input of the second enhanced convolutional block attention mechanism to obtain the output of the second enhanced convolutional block attention mechanism. The output of the second enhanced convolutional block attention mechanism is used as the input of the third dense spatiotemporal feature decoupling block to obtain the output of the third dense spatiotemporal feature decoupling block; The input and output of the third dense spatiotemporal feature decoupling block are concatenated and used as the input to the third enhanced convolutional block attention mechanism, thereby obtaining the output of the second enhanced convolutional block attention mechanism. In this invention, each dense spatiotemporal feature decoupling block includes a backbone feature extraction stream and a spatiotemporal decoupling feature extraction stream. The specific data processing procedure is as follows: The data from the input dense spatiotemporal feature decoupling block is processed by two traditional 3D convolutional blocks of the backbone feature extraction stream to obtain the backbone feature output; The output of the first traditional 3D convolutional block of the main feature extraction stream is used as the input of the branch feature extraction stream, and the branch feature output is obtained through the three convolutional blocks. The main feature output and the branch feature output are added together and then concatenated with the input of the dense spatiotemporal feature decoupling block to obtain the final feature.

[0011] In this invention, the enhanced convolutional block attention mechanism comprises two attention mechanisms: an enhanced channel attention mechanism and a spatial attention mechanism. These are used in a concatenated manner for feature processing, and the specific processing procedure is as follows: First, the enhanced channel attention mechanism will be the first component to process features, with different weights set for each channel dimension; the detailed calculation process is as follows: (9); (10); (11); in, For the input features of the enhanced convolutional block attention mechanism, The function is used to process the data. The subsequent data is weighted to obtain the channel weight matrix. Then With features Element-wise multiplication is performed to obtain the final output of the enhanced channel attention mechanism module. ; Secondly, the spatial attention mechanism will serve as another feature processing component following the enhanced channel attention mechanism, and the output of the enhanced channel attention mechanism component will be... This will be used as input to the spatial attention mechanism module to obtain the spatial weight matrix. The detailed calculation process is as follows: (12); (13); (14); in, The function is used to process the function. The subsequent features are weighted to obtain spatial feature weights. Finally With features Perform element-wise multiplication to obtain the final output. In addition, the function This means that the concatenation operation is performed first, followed by a convolution kernel with a size of [missing value]. Convolution operation, and These represent max pooling and average pooling, respectively.

[0012] In this invention, the processing procedure of the feature enhancement module is as follows: First, temporal pooling and global pooling operations are used to extract important features, thereby enhancing the representation capability of gait features; Then, a fully connected layer is used to integrate effective information, and a non-linear activation function is used to enhance the model's representation and generalization capabilities. Finally, BNNeck is adopted to reduce overfitting, accelerate model training, reduce gradient vanishing or exploding problems, and enhance the robustness and generalization ability of the model.

[0013] In this invention, the loss is calculated using the triplet loss function and the cross-entropy loss function for multiple lateral gait features, and then combined for parameter optimization in order to obtain better model parameters.

[0014] In this invention, the loss is calculated using the triplet loss function and the cross-entropy loss function for multiple lateral gait features, as follows: Will and The loss value for each level feature component is calculated using the triple loss function and the cross-entropy loss function, respectively. It is the output after processing by time pooling, global average pooling, global max pooling, and a fully connected layer. The final outputs of BNNeck are summed and then subjected to a joint loss to optimize the model parameters. The loss function is expressed as: (20); By performing triplet loss calculations on each lateral gait feature, the calculation formula is obtained: (twenty one); Where anc. represents the anchor point, pos. represents the positive sample, and neg. represents the negative sample. Represents the number of triples with all non-zero loss terms; Q represents the level component scale of each feature, which is also... Q-dimensional scale; P, K The final result consists of the number of subjects in each batch and the number of sequence samples for each subject. There are three triples; m represents the margin parameter of the triple loss function; in formula (21) This represents the distance difference from the anchor point in each triplet to the positive and negative examples, which can be formally expressed as: (twenty two); in, The q-th level component feature represents the gait sequence sample of the i-th subject, and is also the anchor feature in the current triplet, while formula (22) The similarity metric function uses Euclidean geometric distance, and the formula is as follows: (twenty three); The loss value is calculated for each component feature, and the average loss is finally used as the cross-entropy loss for each sample. The specific calculation formula is as follows: (twenty four); in, N This represents the number of samples in each batch. Q Represents the number of horizontal component features. K Represents the number of categories. Represents the sign function, when the sample i The true category and k If they are the same, take 1; otherwise, take 0. Representative observation sample i The q Each level component feature belongs to the category k The predicted probability.

[0015] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention designs a spatiotemporal feature decoupling network framework based on dynamic dense connections, applying the concept of feature reuse to our designed network. This allows shallow gait features to be directly transferred to deep networks, enabling the network to better focus on detailed and abstract gait features, enhancing feature richness and improving model robustness. Furthermore, our proposed method achieves excellent recognition performance with less resource consumption, thus reducing resource overhead.

[0016] (2) The initial feature processing module in this invention includes the structure of the original input data, the preprocessing method, and the initial feature extraction block. This module can reduce the amount of computation and remove invalid information, extract the region of interest of moving human body from the input video sequence (reducing network parameters and computation), and perform data normalization processing (accelerating network convergence, avoiding gradient vanishing or exploding, enhancing the robustness of the model and improving the generalization performance of the model). In addition, the initial feature extraction block can convert the input data into a feature representation that the network can understand, thereby providing more meaningful information for the subsequent network. Furthermore, it can also reduce the time dimension, reduce the number of network parameters, and reduce resource consumption.

[0017] (3) The dynamic dense spatiotemporal decoupling feature extraction module and the enhanced convolutional block attention mechanism in this invention: The dense spatiotemporal decoupling feature extraction module uses parallel structures and low-rank convolutions to obtain gait spatiotemporal correlation features and spatiotemporal decoupling features, thereby alleviating the problem of insufficient information expression ability, enabling the network to make fuller use of spatial structure information and temporal correlation information, and the dynamic growth rate of the dynamic dense spatiotemporal decoupling feature extraction module can help the network obtain more gait feature information, improve the generalization ability of the model and the final feature representation ability. The enhanced convolutional block attention mechanism can better focus on the detailed features of gait, which is conducive to mining key features such as stride, contour, and arm swing to improve the representation ability of the final features.

[0018] (4) The feature enhancement processing module in this invention can convert the features extracted by the network into multiple lateral gait features, remove certain redundant information, and help the loss function to optimize network parameters. Attached Figure Description

[0019] Figure 1 This is a framework diagram of the present invention.

[0020] Figure 2 This is a schematic diagram of the dense spatiotemporal feature decoupling block structure of the present invention.

[0021] Figure 3 This is a schematic diagram illustrating the principle of the enhanced convolutional block attention mechanism of the present invention.

[0022] Figure 4 This is a schematic diagram illustrating the principle of the feature enhancement processing module of the present invention.

[0023] Figure 5 This is a schematic diagram showing the comparison of batch size and accuracy of the present invention on the CASIA-B dataset.

[0024] Figure 6 This is a bar chart illustrating the impact of the model layer number setting and model parameter quantity on the experimental results.

[0025] Figure 7 This invention illustrates the impact of different DST Blocks and Attention Blocks on the model's recognition accuracy. Detailed Implementation

[0026] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0027] Combination Figure 1-7 A dynamic, densely connected spatiotemporal feature decoupling network for recognizing cross-view gait is described in detail below: (I) Dynamically Densely Connected Spatiotemporal Feature Decoupling Network Framework Figure 1 This section presents our proposed framework based on dynamically densely connected spatiotemporal feature decoupling. First, the Initial Feature Processing (IFPM) module uses two 3D convolutions to extract preliminary features from the data, while simultaneously reducing the temporal dimension to minimize computational resource consumption. Second, a dense spatiotemporal feature decoupling block and concatenation operation enable the sharing of features between shallow and deep networks, addressing the issue of insufficient representational power. An enhanced convolutional block attention mechanism is used to focus the network on more important gait features. Then, in the feature enhancement processing module, temporal pooling (TP), global average pooling (GAP), global max pooling (GMP), split fully connected layers (SFC), and batch normalized neck (BNNeck) structures are employed to map the five-dimensional feature into multiple lateral features, and batch normalization is performed to enhance representational power and model generalization ability. Finally, the triple loss function and cross-entropy loss function are used to calculate the loss for each of the multiple lateral features, and these are jointly used for model parameter optimization to obtain better model parameters.

[0028] (II) Dynamic Dense Spatiotemporal Decoupling Feature Extraction Module We propose a Dynamic Dense Spatiotemporal Decoupling Feature Extraction Module (DDST-FEM), which comprises two modules: the Dense Spatiotemporal Feature Decoupling Block (DST Block) (its structure is as follows). Figure 2 (as shown) and Enhanced Convolutional Block Attention (E-CBAM) (its structure is as shown) Figure 3 (As shown).

[0029] The dynamic dense spatiotemporal decoupling feature extraction module comprises three dense spatiotemporal feature decoupling blocks, three enhanced convolutional block attention mechanisms, and a max-pooling module. Considering the low channel dimension of each block in DenseNet, which leads to the loss of detailed information in gait feature extraction, we redesigned this module by introducing a dynamic growth rate to minimize computational resource consumption while minimizing information loss. Specifically, this invention replaces the fixed number of modules in DenseNet with a dynamically changing number of modules, thereby helping the network acquire more effective gait information. Furthermore, the dense spatiotemporal feature decoupling blocks are designed to address the problem of insufficient gait feature representation, while the enhanced convolutional block attention mechanisms are designed to address the model's insufficient ability to focus on key information. More detailed internal structures and design motivations will be described below.

[0030] Assuming that shallow spatiotemporal features are obtained after the initial feature processing module. This feature will serve as the input to the dynamic dense spatiotemporal decoupling feature extraction module, and will be processed through 3 DST-A Blocks, 3 E-CABM Blocks, and 1 M3D Block to obtain the output data. : (1); in, Represents Initial Block (e.g.) Figure 1 The output of ) is also the input of DDST-FEM. This represents a dynamic, dense spatiotemporal decoupling feature extraction module. This is the final feature output of the module.

[0031] (III) Dense Spatiotemporal Feature Decoupling Block The Dense Spatiotemporal Feature Extraction Block (DST Block) comprises two branches. First, it extracts different features using a main feature extraction stream (containing two convolutional blocks) and a spatiotemporally decoupled feature extraction stream (containing three convolutional blocks). Then, the outputs of the two branches are summed. Finally, the input is concatenated with the summed outputs of the two branches to obtain the output of the DST Block. Figure 2As shown, the dense spatiotemporal feature extraction module was designed to solve two problems. First, it aims to achieve feature reuse, allowing shallow features to be obtained and used in deep networks. Specifically, gait recognition, as a task highly focused on detailed features, is more sensitive to details such as the subject's contour, arm swing, and stride. Shallow feature maps in deep learning are relatively large, thus containing more detailed features (more advantageous for the network to acquire detailed features), while deep feature maps are relatively small, thus containing more abstract semantic features (more advantageous for classification). The idea of ​​feature reuse allows shallow features to be fed into deep networks, achieving the sharing of detailed and abstract semantic features. Therefore, we designed a dense spatiotemporal feature extraction block, which concatenates the input and output of each block through a concatenation operation to pass shallow detailed features to deep networks, achieving feature reuse. The second problem addressed is the extraction of spatiotemporally decoupled features. Traditional 3DConv may introduce irrelevant information and have insufficient feature representation capabilities, while also incurring significant computational overhead due to excessive module stacking. The design of the P3D[6] module utilizes low-rank convolution to obtain features, but this module does not focus on the extraction of spatiotemporal related features. Therefore, this invention designs a dense spatiotemporal feature extraction block based on the P3D module. The main branch is used to extract spatiotemporal related features, while the branches are used to obtain decoupled spatiotemporal features. Finally, the representation power of the features is enhanced by adding them together.

[0032] like Figure 2 As shown, this invention designs, and through experiments, selects the DST-A Block as the dense spatiotemporal feature extraction block in the DD-ST Net method, taking the first DST-A Block as an example ( Figure 1 If the DST Block1 is used, then the input data is the output of the initialization module. The DST-A Block contains two different feature extraction branches, and these two branches are analyzed separately.

[0033] First, for the main feature extraction stream, the input data is ,and The main feature extraction stream passes through two traditional 3D convolutional blocks ( Figure 2 (d) thus obtaining the main output The detailed calculation process of the intermediate layer can be expressed as follows: (2); (3); in, The representative size is convolutional layers, The representative size is The convolutional layer. It is after convolution kernel size is The output obtained from the convolutional layer, Then it is After convolution kernel size is The backbone output obtained from the convolutional layer, , .

[0034] Secondly, for the branch feature extraction flow, the input data is , It will pass through two low-rank convolutions of the branch feature extraction flow ( Figure 2 (d) branches), thus obtaining the branch feature output. The detailed calculation process is as follows: (4); in The kernel size represents the convolution kernel size. The convolutional layers are used to extract gait spatial features. It is the output after passing through a spatially low-rank convolutional layer. .

[0035] (5); in The kernel size represents the convolution kernel size. The convolutional layers are used to extract gait temporal features. It is the output after passing through a spatially low-rank convolutional layer. .

[0036] (6); in The kernel size represents the convolution kernel size. The convolutional layers are mainly used for dimensionality reduction of the channel dimension, ultimately obtaining the gait spatiotemporal decoupling features after branch feature extraction flow. , .

[0037] Finally, the outputs of the main branch and branches are summed to improve the representational power of the features. Then, the input and the summed features are concatenated to obtain the final feature representation. The specific calculation process is as follows: (7); in, This represents an element-wise addition operation. This represents a concatenation operation (the concatenation dimension is located in the channel dimension), and the final output feature. .

[0038] (iv) Enhanced Convolutional Block Attention Mechanism The E-CBAM module designed in this invention includes two attention mechanisms: Enhanced Channel Attention (E-CAM) and Spatial Attention (SAM). Considering that traditional CBAM modules lose some gait feature information, we remove max pooling and average pooling within the channel attention mechanism to better adapt to the gait feature map (e.g., ...). Figure 3 Furthermore, spatial attention mechanisms can help models more effectively capture long-range dependencies and local information (such as...) when processing gait data. Figure 3 Simultaneously employing both attention mechanisms can improve the model's focus on gait walking targets, thereby enhancing the interpretability of deep learning models. Specifically, from a human visual perspective, continuously focusing on the entire scene may prevent the identification of important information. Therefore, humans focus on certain more important information through observation, which is the attention mechanism in the human visual system. Gait, as a behavior activity that emphasizes detailed features, requires attention to the outline of the human walking body, arm swing amplitude, stride length, etc., which is key to achieving better recognition performance. In addition, the CAM module in the traditional CBAM mechanism uses pooling to reduce the feature map size. This may achieve better recognition results in other recognition tasks, but for gait, which requires attention to detailed features, this may lead to the loss of gait feature information. Therefore, to obtain more representative gait features, we redesigned this module, such as... Figure 3 As shown.

[0039] The E-CBAM module includes two attention mechanisms, E-CAM and SAM, and performs feature processing in a concatenated manner (e.g., Figure 3 Taking the first E-CBAM module as an example for analysis, the input data is... Output after E-CBAM The overall calculation formula is as follows: (8); Furthermore, a more detailed introduction to each of the two modules will be provided. First, E-CAM will serve as the first component to perform feature processing, obtaining features in the DST-A block. This will be used as input to E-CAM, allowing different weights to be set for each channel dimension. The detailed calculation process is as follows: (9); (10); (11); in, The function is used to process the data. The subsequent data is weighted to obtain the channel weight matrix. (Formula 10), then With features Element-wise multiplication is performed to obtain the final E-CAM module output. .

[0040] Secondly, SAM will serve as another feature processing component after E-CAM, and the output of the E-CAM component... This will be used as input to the SAM module to obtain the spatial weight matrix. The detailed calculation process is as follows: (12); (13); (14); in, The function is used to process the function. The subsequent features are weighted to obtain spatial feature weights. (Formula 13), finally... With features Perform element-wise multiplication to obtain the final output. Furthermore, the function This means that the concatenation operation is performed first, followed by a convolution kernel with a size of [missing value]. The convolution operation (Equation 12). and These represent max pooling and average pooling, respectively.

[0041] (V) Feature Enhancement Processing Module The feature enhancement module includes temporal pooling, global average pooling, global max pooling, fully connected layers, and a BNNeck component. Considering that the features obtained through the dynamic dense spatiotemporal decoupling feature extraction module contain redundant information, we designed a feature enhancement module to process the features obtained by the network. This processing involves some changes in feature dimensions (such as…). Figure 4 The specific dimensional changes are shown below. We adopted this design for several important purposes. First, the use of temporal pooling and global pooling operations is primarily to extract more important features and enhance the representational power of gait features. Second, the use of fully connected layers is to integrate effective information and enhance the model's representation and generalization capabilities through non-linear activation functions. Finally, the use of BNNeck is to reduce overfitting, accelerate model training, mitigate gradient vanishing or exploding problems, and enhance the model's robustness and generalization ability.

[0042] The detailed calculation process of the feature enhancement processing module is as follows: Figure 4 As shown, the overall framework adopts a sequential and parallel structure. The input data for this module is... Figure 1 The Feature section, i.e. (Formula 1). The detailed calculation process will be given below, and the overall calculation is as follows: (15); in, Refers to the entire feature processing module. It is the final output after passing through BNNeck. .

[0043] Features obtained by the feature extraction module The final output will be obtained through processing by four modules. The detailed calculation process is as follows: (16); (17); (18); (19); in, Represents time-dimension pooling, the resulting output , This represents element-wise addition. and The outputs represent global average pooling and global max pooling. , This represents a decoupled fully connected layer used for feature mapping operations, and the output is... Finally, after Get output .

[0044] (vi) Loss Function For the output and The loss value for each level feature component is calculated using the triple loss function and the cross-entropy loss function respectively. Finally, the values ​​are summed to obtain a joint loss for optimizing the model parameters. The loss function can be expressed as: (20); Influenced by the success of the triple loss function in the pedestrian re-identification task [8], this invention adopts the batch all version of the triple loss function as one of the joint loss functions. This loss function is used in the model training stage. Each optimization sample consists of an anchor point (anc.), positive sample (pos.), and negative sample (neg.). The anchor point has the same label as the positive sample and a different label than the negative sample. In order to make fuller use of the processed gait features, triple loss is calculated for each level feature component, and the calculation formula can be obtained as follows: (twenty one); in, Represents the number of triples with all non-zero loss terms; Q represents the level component scale of each feature, which is also... The Q-dimensional scale; (P, K) represents the number of subjects in each batch and the number of sequence samples per subject (a total of PK sequence samples are input into the network), which can then form the final structure. There are three triples; m represents the margin parameter of the triple loss function. In formula (21) This represents the distance difference from the anchor point in each triplet to the positive and negative examples, which can be formally expressed as: (twenty two); in, The q-th level component feature represents the a-th gait sequence sample of the i-th subject, and is also the anchor feature in the current triplet. and and They have the same meaning, representing positive and negative example features respectively. And formula (22) The similarity metric function uses Euclidean geometric distance, and the formula is as follows: (twenty three); The cross-entropy loss function is widely used in classification tasks in deep learning. Its mathematical form is relatively simple, easy to understand and implement, and it can better avoid the vanishing gradient problem during gradient backpropagation, contributing to stable network training. Therefore, this invention also selects the cross-entropy loss function as one of the members of the joint loss function. Furthermore, to better utilize the final gait level feature components, this invention calculates the loss value for each feature component, ultimately using the average loss as the cross-entropy loss for each sample. The specific calculation formula is as follows: (twenty four); in, N This represents the number of samples in each batch. Q Represents the number of horizontal component features.K Represents the number of categories. Represents the sign function (0 or 1), when the sample i The true category and k If they are the same, take 1; otherwise, take 0. Representative observation sample i The q Each level component feature belongs to the category k The predicted probability.

[0045] (vii) Experimental Data (7.1) Introduction to the dataset We selected two popular human behavior recognition datasets to validate the proposed method, namely CASIA-B[9] and OUMVLP

[10] .

[0046] The CASIA-B dataset, captured by the Institute of Automation, Chinese Academy of Sciences, has become the most widely used gait dataset. It contains RGB and silhouette images of 124 individuals, totaling 13,680 video sequences, making it a large-scale, multi-view, publicly available gait dataset. This dataset considers three different walking conditions: normal walking (NM), backpack walking (BG), and walking with a jacket (CL). Furthermore, the dataset was captured from 11 perspectives, ranging from 0° to 180° (with one perspective every 18°). Each subject ultimately has 6 video sequences (NM#01-06), 2 (CL#01-02), and 2 (BG#01-02) for each perspective. To ensure fair performance comparisons with current methods, we strictly adhere to the evaluation protocols of

[11] ,

[12] , and [1]. The protocol establishes rules for dividing the training and test sets, including few-shot training (ST), medium-shot training (MT), and large-shot training (LT). We adopted large-shot training, using all data from 74 individuals to train the model, while the data from the remaining 50 individuals will be used to construct the image library and probe set for accuracy testing. For the testing phase, data numbered NM#01-04 were used to generate the image library set, while data numbered NM#05-06, BG#01-02, and CL#01-02 were used to generate the probe set.

[0047] OUMVLP is a large-scale, multi-view gait dataset, currently the largest publicly available gait dataset, including 10,307 subjects. The viewing angles range from 0° to 90° (intervals of 15°) and from 180° to 270° (intervals of 15°), totaling 14 viewing angles and containing two sequences (numbered #00-01). Due to the sheer size of the OUMVLP dataset and considering the high computational cost, this invention selects 512 subjects from the dataset according to a common dataset partitioning ratio. Of these, 256 subjects are used for training, and the remaining 256 are used for testing. Data from sequence #00 is used to generate the image library, and data from sequence #01 is used to generate the probe set.

[0048] (7.2) Experimental environment configuration and parameter settings For all experiments, the gait sequences were resized and cropped, with the frame length set to 30, and each frame's height and width set to 64. No data augmentation operations were performed, such as horizontal flipping, random occlusion, or rotation. The margin parameter of the triplet loss function was set to 0.2, and SGD was used as the model optimizer. The initial learning rate (lr) and weight decay parameters were set to 0.1 and 0.0005, respectively. For the CASIA-B dataset, data from 74 subjects were used as the training set, and the batch size was set to 24 during training. P × K =4×6), and the maximum number of iterations is set to 80K. For the OUMVLP dataset, considering the expensive computational cost, in order to verify the generality of the method of this invention, the data of 256 subjects are selected as the training set, and the batch size is set to 24 ( P × K =6×4), and the maximum number of iterations was set to 120K. For the final testing phase, Euclidean distance was used as the final similarity measurement tool. Detailed experimental environment and computer configuration are shown in Table 1.

[0049] Table 1 Experimental Environment and Configuration

[0050] (7.3) Experimental results on the CASIA-B dataset In this section, the proposed method DD-ST Net will be compared with several state-of-the-art methods on the CASIA-B dataset in terms of recognition accuracy, including GEINet

[13] (ICB 2016), CNN-LB

[14] (TPAMI2017), GaitNet

[15] (TPAMI 2022), ACL[3] (TIP 2020), GaitSet

[16] (TPAMI 2022), GaitPart

[17] (CVPR 2020), MvGAN

[18] (TIP 2021), MT3D[1] (MM 2020) and GaitBase[2] (CVPR 2023). For the specific comparison, this invention conducted a more systematic and comprehensive experiment to compare the results, that is, experiments were conducted on CASIA-B under all cross-viewpoint and cross-walking conditions (excluding the same viewpoint). Specifically, the model was first trained using all data from 74 subjects to obtain the feature extraction model. Then, data from another 50 subjects labeled NM#01-04 were used as the image library set, while data labeled NM#05-06, BG#01-02, and CL#01-02 were used as the probe set for evaluating method performance. Finally, Euclidean geometric distance was used to calculate the feature similarity between the probes and the image library. The experimental results in Table 2 show that, compared with multiple methods, DD-ST Net outperforms most methods on average, reaching the current mainstream recognition accuracy level. The method of this invention achieved average recognition accuracies of 96.12%, 91.61%, and 76.44% under NM, BG, and CL conditions, respectively. Compared with the methods listed in Table 2, the method of this invention achieves excellent recognition performance, fully demonstrating the feasibility of DD-ST Net.

[0051] Table 2 shows the average grade 1 accuracy (%) of cross-views of different probe views on CASIA-B, excluding identical views.

[0052]

[0053] From another perspective, the proposed method can achieve excellent Rank-1 recognition accuracy with a smaller batch size. To more intuitively demonstrate the advantages of the proposed method with a smaller batch size, this invention selects several of the most advanced gait recognition methods and compares their batch size and accuracy on the CASIA-B dataset under three walking conditions (NM, BG, and CL) and under normal walking conditions only (NM). The methods include: ACL, GaitSet, GaitPart, MT3D, 3DLocal[5], GaitGL

[19] , CSTL

[20] , GaitSlice[4], and OpenGait[2]. Figure 5 It can be observed that the method of the present invention can achieve excellent recognition performance with small batch sizes. Under the three walking conditions, the method of the present invention can achieve an average accuracy of 88.1% with a batch size of 24. Figure 5 (a)), while under normal walking conditions alone, an average accuracy rate of 96.1% can be achieved ( Figure 5 (b) This demonstrates that the method proposed in this invention can achieve excellent cross-view gait recognition accuracy in a greener and lower-carbon manner (with less resource consumption).

[0054] Table 3 shows the average Rank-1 accuracy (%) of cross-view OMVLP (excluding identical views).

[0055] (7.4) Experimental results on the OUMVLP dataset In order to verify the generalization ability of the method of the present invention, the OUMVLP dataset was selected as the generalization verification dataset. However, considering that the OUMVLP dataset is too large and the computational cost is expensive, only a portion of the data was selected for verification. Specifically, a total of 512 people's data were selected for the experiment, of which all data of 256 people were used for model training, and the other 256 people were used for model performance testing. All 14 views were used for performance testing. In order to evaluate the performance of the method of the present invention more comprehensively and systematically, the present invention selected several well-known methods for comparison, including: GEINet (ICB 2016), DULE

[21] (CVPR 2021), GaitSet (TPAMI 2022), and GaitPart (CVPR 2020). Table 3 shows the comparison of the accuracy of the method of the present invention with other methods. The method of the present invention surpasses most methods and reaches the level of the current mainstream methods, which also proves that the method of the present invention has a certain generalization ability.

[0056] (7.5) Ablation test Our proposed method includes network setup and several key components: DSTBlock, E-CBAM Block, and BNNeck. This section will explore the effectiveness of these network setups and components, and further analyze the network setup and components based on experimental results.

[0057] Table 4 shows the average Rank-1 recognition accuracy (%) in the DD-ST Net ablation experiments on the CASIA-B dataset.

[0058] (7.5.1) Impact of Network Depth on Experimental Results. To investigate the impact of network depth on recognition performance and the number of model parameters, we conducted two sets of comparative experiments, including Group Base (3-layer DST Block and E-CBAM Block) and Group 1 (4-layer DST Block and E-CBAM Block) as shown in Table 4. Through observation... Figure 6 The experimental results in (a) show that deeper network layers did not necessarily lead to better performance for DD-ST Net. In contrast, Group 1, with its shallow network structure, achieved an average recognition rate 0.8% higher than Group under NM conditions, 1.3% higher under BG conditions, and 2.8% higher under CL conditions, with an average Rank-1 recognition accuracy 1.6% higher across all three cases. Furthermore, observations... Figure 6 The comparison bar chart in (b) shows that the number of model parameters in Group 1 is significantly increased compared to Group Base.

[0059] Table 5 shows the average Rank-1 recognition accuracy (%) under different network growth rates.

[0060] (7.5.2) Impact of Dynamic Growth Rate on Experimental Results. To investigate the impact of channel count on model performance, we designed two different sets of experiments to verify the effect of different network widths on model accuracy. Table 5 shows the impact of different model growth rates on model performance. It can be seen that using a dynamic model growth rate results in a larger number of channels. Group B, compared to A, achieves a 1.9% higher recognition rate in the NM case, a 2.9% higher rate in the BG case, and a 3.8% higher rate in the CL case. The average Rank-1 recognition accuracy is 2.8% higher across all three cases. The reason for this is that in deep learning, to a certain extent, a wider network width leads to better performance, and width refers to the number of channels. Increasing the number of channels provides more information and feature representation space, allowing the model to better capture the complexity and diversity of the input data, thus improving its expressive power. Our designed dynamic growth rate ensures that each DSTBlock in the model has a sufficient number of channels, guaranteeing that the model can acquire more information. The original DenseNet network did not use a dynamic growth rate, resulting in information loss. Therefore, using a dynamic growth rate reduces information loss and achieves better recognition performance.

[0061] (7.5.3) Impact of DST Block on Experimental Results. To investigate the impact of using or combining two different modules on model performance, we designed four different ablation experiments, including Group Base, 2, 3, and 4 in Table 4. Group Base used DST-A Block, Group 2 used DST-B Block, Group 3 used both DST-A and DST-B Blocks, and Group 4 did not use either DST-A or DST-B Blocks, only the trunk branch (…). Figure 2 (a)). To more intuitively observe the impact of different DST Blocks on model performance, this invention plotted a bar chart to compare recognition accuracy. Figure 7(a) Observing the experimental accuracy results in the histogram, we found that the DST-ABlock achieved the best recognition performance, while the hybrid DST-A and DST-B Block achieved the lowest accuracy. Specifically, the Rank-1 recognition accuracy of the model using the DST-A Block was 0.9%, 1.7%, and 4.5% higher than the original 3D convolutional module and the DST-B Block under NM, BG, and CL walking conditions, respectively, and 1.2%, 1.8%, and 4.4%. The design of the DST-A Block has a crucial impact on the performance of DD-ST Net. This module adopts a dual-branch design, which helps the model obtain spatiotemporal correlation features and spatiotemporal decoupling features. Therefore, the DST-A Block is considered an important component of DD-ST Net to improve the expressive power of features.

[0062] (7.5.4) The impact of the attention mechanism on experimental results. To investigate the impact of different E-CBAM settings on model performance, we designed five different ablation experiments, including Group Base, 5, 6, 7, and 8 in Table 4. Group Base used the E-CBAM module, Group 5 used the E-CAM module, Group 6 used the E-SAM module, Group 7 used the original CBAM module, and Group 8 did not use either E-CBAM or CBAM modules. To more intuitively observe the comparison of accuracy among these groups, this invention plotted a histogram of Rank-1 recognition accuracy for these groups (…). Figure 7 (b) We can see that the average recognition accuracy using E-CBAM is 0.5%, 4.1%, 11.6%, and 0.9% higher than the other groups (arranged in order of group), respectively. Attention mechanisms play an indispensable role in deep learning. Through attention mechanisms, models can better capture key information and adapt to changes in input data, thereby improving model robustness and ultimately enhancing performance and efficiency. Therefore, E-CBAM is considered a crucial component of the model, helping it to better focus on key gait information and improve recognition performance.

[0063] (7.5.5) The impact of BNNeck on experimental results. To verify the impact of this component on the method of this invention, we designed two sets of ablation experiments, including Group Base (using the BNNeck module) and Group 9 (without the BNNeck module) in Table 4. By observing the experimental results in Table 4, we found that Group 9 with the BNNeck layer had an average recognition rate that was 0.1% higher than Group under NM conditions, 0.6% higher under BG conditions, and 2.0% higher under CL conditions. The average Rank-1 recognition accuracy was 1.0% higher under the three conditions. BNNNeck[7] combines the concepts of batch normalization and dimensionality reduction, which can improve the expressive power of features, reduce overfitting, and enhance the robustness of the model. Therefore, BNNeck is used as one of the components of the model to improve the recognition performance.

[0064] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A system for decoupling the spatiotemporal features of dynamically densely connected gait across different viewpoints, characterized in that, include: The initial feature processing module extracts preliminary features from the data and reduces the time dimension to minimize computational resource consumption. The dynamic dense spatiotemporal decoupling feature extraction module includes a dense spatiotemporal feature decoupling block and an enhanced convolutional block attention mechanism. After the initial feature processing module obtains shallow spatiotemporal features as input, the dense spatiotemporal feature decoupling block realizes the sharing of shallow network features and deep network features, enhancing the representation power of the features; at the same time, the enhanced convolutional block attention mechanism is used to improve the model's attention to gait walking targets. The feature enhancement processing module includes temporal pooling, global average pooling, global max pooling, fully connected layers, and the BNNeck component. It transforms the features extracted by the network into multiple lateral gait features to enhance the representation ability and model generalization ability, and removes redundant information. The dynamic dense spatiotemporal decoupling feature extraction module includes three dense spatiotemporal feature decoupling blocks, three enhanced convolutional block attention mechanisms, and a max pooling module. The specific data processing procedure of the dynamic dense spatiotemporal decoupling feature extraction module is as follows: The output of the initial feature processing module is used as the input of the first dense spatiotemporal feature decoupling block to obtain the output of the first dense spatiotemporal feature decoupling block; The input and output of the first dense spatiotemporal feature decoupling block are concatenated and used as the input of the first enhanced convolutional block attention mechanism to obtain the output of the first enhanced convolutional block attention mechanism. The output of the first enhanced convolutional block attention mechanism is used as the input to the max pooling module to obtain the output of the max pooling module; The output of the max pooling module is used as the input of the second dense spatiotemporal feature decoupling block to obtain the output of the second dense spatiotemporal feature decoupling block; The input and output of the second dense spatiotemporal feature decoupling block are concatenated and used as the input of the second enhanced convolutional block attention mechanism to obtain the output of the second enhanced convolutional block attention mechanism. The output of the second enhanced convolutional block attention mechanism is used as the input of the third dense spatiotemporal feature decoupling block to obtain the output of the third dense spatiotemporal feature decoupling block; The input and output of the third dense spatiotemporal feature decoupling block are concatenated and used as the input of the third enhanced convolutional block attention mechanism to obtain the output of the second enhanced convolutional block attention mechanism. Each dense spatiotemporal feature decoupling block includes a backbone feature extraction stream and a spatiotemporal decoupling feature extraction stream. The specific data processing procedure is as follows: The data from the input dense spatiotemporal feature decoupling block is processed by two traditional 3D convolutional blocks of the backbone feature extraction stream to obtain the backbone feature output; The output of the first traditional 3D convolutional block of the main feature extraction stream is used as the input of the branch feature extraction stream, and the branch feature output is obtained through the three convolutional blocks. The main feature output and the branch feature output are added together and then concatenated with the input of the dense spatiotemporal feature decoupling block to obtain the final feature.

2. The dynamic dense connection spatiotemporal feature decoupling system for identifying cross-view gait according to claim 1, characterized in that, The enhanced convolutional block attention mechanism comprises two attention mechanisms: enhanced channel attention and spatial attention. These are used in a concatenated manner for feature processing. The specific processing steps are as follows: First, the enhanced channel attention mechanism will be the first component to process features, with different weights set for each channel dimension; the detailed calculation process is as follows: （9）； （10）； （11）； in, For the input features of the enhanced convolutional block attention mechanism, The function is used to process the data. The subsequent data is weighted to obtain the channel weight matrix. Then With features Element-wise multiplication is performed to obtain the final output of the enhanced channel attention mechanism module. ; Secondly, the spatial attention mechanism will serve as another feature processing component following the enhanced channel attention mechanism, and the output of the enhanced channel attention mechanism component will be... This will be used as input to the spatial attention mechanism module to obtain the spatial weight matrix. The detailed calculation process is as follows: （12）； （13）； （14）； in, The function is used to process the function. The subsequent features are weighted to obtain spatial feature weights. Finally With features Perform element-wise multiplication to obtain the final output. In addition, the function This means that the concatenation operation is performed first, followed by a convolution kernel with a size of [missing value]. Convolution operation, and These represent max pooling and average pooling, respectively.

3. The dynamic dense connection spatiotemporal feature decoupling system for identifying cross-view gait according to claim 1, characterized in that, The processing procedure of the feature enhancement module is as follows: First, temporal pooling and global pooling operations are used to extract important features, thereby enhancing the representation capability of gait features; Then, a fully connected layer is used to integrate effective information, and a non-linear activation function is used to enhance the model's representation and generalization capabilities. Finally, BNNeck is adopted to reduce overfitting, accelerate model training, reduce gradient vanishing or exploding problems, and enhance the robustness and generalization ability of the model.

4. The dynamic dense connection spatiotemporal feature decoupling system for identifying cross-view gait according to claim 1, characterized in that, The loss is calculated using the triplet loss function and the cross-entropy loss function for multiple lateral gait features, and then combined for parameter optimization in order to obtain better model parameters.

5. The dynamic dense connection spatiotemporal feature decoupling system for identifying cross-view gait according to claim 4, characterized in that, The loss is calculated using the triplet loss function and the cross-entropy loss function for multiple lateral gait features, as follows: Will and The loss value for each level feature component is calculated using the triple loss function and the cross-entropy loss function, respectively. It is the output after processing by time pooling, global average pooling, global max pooling, and a fully connected layer. The final outputs of BNNeck are summed and then subjected to a joint loss to optimize the model parameters. The loss function is expressed as: （20）； By performing triplet loss calculations on each lateral gait feature, the calculation formula is obtained: （21）； Where anc. represents the anchor point, pos. represents the positive sample, and neg. represents the negative sample. Represents the number of triples with all non-zero loss terms; Q represents the level component scale of each feature, which is also... Q-dimensional scale; P, K The final result consists of the number of subjects in each batch and the number of sequence samples for each subject. There are three triples; m represents the margin parameter of the triple loss function; in formula (21) This represents the distance difference from the anchor point in each triplet to the positive and negative examples, which can be formally expressed as: （22）； in, The q-th level component feature represents the gait sequence sample of the i-th subject, and is also the anchor feature in the current triplet, while formula (22) The similarity metric function uses Euclidean geometric distance, and the formula is as follows: （23）； The loss value is calculated for each component feature, and the average loss is finally used as the cross-entropy loss for each sample. The specific calculation formula is as follows: （24）； in, N This represents the number of samples in each batch. Q Represents the number of horizontal component features. K Represents the number of categories. Represents the sign function, when the sample i The true category and k If they are the same, take 1; otherwise, take 0. Representative observation sample i The q Each level component feature belongs to the category k The predicted probability.

Images