Dynamic dense connection spatial-temporal feature decoupling network for recognizing cross-view gait

Through dynamic and dense connection of spatiotemporal feature decoupling network, the problem of insufficient expression of spatiotemporal decoupling feature in cross-view gait recognition is solved, and the sharing and feature enhancement of shallow and deep features are achieved, which improves recognition performance and model robustness.

CN119049129AActive Publication Date: 2024-11-29HEFEI UNIV
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411166831.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-11-29
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

The existing cross-view gait recognition method based on convolutional neural networks has problems such as insufficient spatial and temporal decoupling feature expression, excessive network parameters, insufficient redundant information and feature expression capabilities, and ignores the sharing of shallow and deep features. .

Method used

A dynamically densely connected spatiotemporal feature decoupling network is designed to identify gait across perspectives. The sharing of shallow and deep features is achieved through dense spatiotemporal feature decoupling blocks and enhanced convolutional block attention mechanisms is achieved, and in the feature enhancement processing module Batch normalization and multiple lateral feature mappings are performed to enhance characterization and model generalization capabilities.

Benefits of technology

It improves the richness of gait features and model robustness, reduces computing resource consumption, improves recognition performance, and effectively utilizes the spatio-temporal correlation information of gait.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049129A_ABST
    Figure CN119049129A_ABST
Patent Text Reader

Abstract

The invention discloses a dynamic dense connection spatio-temporal feature decoupling network for identifying cross-view gaits, which relates to the field of computer vision and comprises an initial feature processing module, a dynamic dense spatio-temporal decoupling feature extraction module and a feature enhancement processing module. Sharing of shallow network features and deep network features is realized by using dense spatial-temporal feature decoupling blocks and splicing operation, so that the problem of insufficient characterization capability is solved; meanwhile, a convolutional block attention enhancing mechanism is used for enabling the network to pay attention to more important gait features; and finally, the five-dimensional features are mapped and processed into a plurality of transverse features in a feature enhancement processing module, and batch standardization is performed on the features to enhance the representation capability and the model generalization capability, so that the correlation between shallow and deep features can be mined, and the problem of insufficient information expression capability is relieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a dynamic densely connected spatiotemporal feature decoupling network for identifying cross-view gait. Background Art

[0002] Biometric technology refers to the technology that uses the inherent physiological and behavioral characteristics of the human body to identify individuals through the close combination of computers, optics, acoustics, biosensors and biostatistics principles. Gait recognition refers to the technology of identifying individuals through biometric characteristics such as the outline, stride, and arm swing of the human body. It has been successfully applied to user identification, sports science and impact analysis. Due to the advantages of gait features such as easy collection, long distance, non-contact, and difficult to disguise, existing gait recognition methods use gait features instead of traditional biometric features, and have achieved exciting performance and received widespread attention.

[0003] In the past two decades, many gait recognition methods have been developed to solve the above problems. Before 2015, the research on gait recognition mainly focused on manually extracting gait features, then generating templates through modeling, and then realizing gait recognition through template matching, but the traditional method did not achieve impressive results. Since the deep learning-based gait recognition method was proposed in 2015, the traditional machine learning method has gradually been abandoned, and more and more attention has been focused on the deep learning-based gait recognition method.

[0004] Since the manual feature method based on traditional machine learning does not obtain sufficient global and temporal information, it has certain limitations. At present, most of the methods based on deep learning use the original gait sequence data as input data, then extract the spatiotemporal features of gait through convolutional neural networks, and finally predict the category through feature similarity measurement methods. Therefore, it can pay more attention to the spatiotemporal information of gait. The gait recognition methods based on ordered sequences are mainly divided into decoupled spatiotemporal modeling and model construction using 3D convolution. Decoupled spatiotemporal modeling generally refers to separating the time dimension from the spatial dimension for feature extraction. The usual practice is to first use a two-dimensional convolutional network to extract the spatial features of each gait frame, and then use temporal modeling to extract the gait information in the temporal dimension. For example, Zhang et al. used a long short-term memory network as a temporal attention model to obtain the temporal information of gait. Li et al. also made different modeling for the spatiotemporal information of gait and proposed a gait recognition method called GaitSlice. In terms of space, GaitSlice uses a spatial feature extractor to obtain the spatial features of gait, and in terms of time, it introduces a residual frame attention mechanism to obtain key gait frames, and finally uses maximum aggregation to obtain the temporal features of gait.

[0005] In addition, in order to better utilize the spatiotemporal correlation information, three-dimensional convolutional networks are gradually being applied to the field of gait recognition. For example, Lin et al. proposed a new multi-time scale gait recognition method based on 3D convolution. MT3D has two feature extraction streams, small time scale and large time scale, and uses 3DCNN to extract spatiotemporal information of gait information of two different time scales respectively, and then uses frame pooling operation to fuse the information of the two scales, and finally uses Euclidean distance to calculate the distance between features to predict the category. Huang et al. proposed a new point of view. They believe that different parts of the human body will show different movement patterns when walking, and proposed a gait recognition method called three-dimensional local convolutional neural network. This method divides the gait frame into different local ordered sequences according to the parts of the human body (such as legs, arms, etc.), and then uses 3DCNN to extract spatiotemporal features. Finally, all features are spliced ​​to calculate feature similarity to predict the category.

[0006] However, the existing cross-view gait recognition methods based on convolutional neural networks still have the following problems. First, most of the previous work on gait recognition uses a single spatiotemporal modeling method and stacks a series of modules to obtain spatiotemporal features. The method of directly performing spatiotemporal modeling and then extracting features will ignore the spatiotemporal decoupling characteristics of gait, making it difficult to fully learn the spatiotemporal decoupling feature expression of gait sequence data, which has certain limitations. Another method of modeling time and space separately can obtain the spatiotemporal decoupling characteristics of gait, but it often ignores the spatiotemporal correlation information of gait, which will lead to insufficient representation of gait feature information. In addition, the use of traditional 3D convolution modules for modeling will also lead to some problems. On the one hand, it may lead to too many network parameters, resulting in huge consumption of computing resources. On the other hand, it may introduce some unnecessary redundant information or cause insufficient feature expression capabilities. Finally, previous technologies often ignore the sharing problem of shallow and deep network features. The shallow feature map is relatively large and has more detailed features (which is more advantageous for the network to obtain detailed features). In contrast, the deep feature map is relatively small and contains more abstract semantic features (which is more advantageous for classification). However, most previous methods only consider deep features, and thus often ignore some more detailed features, which are important information for distinguishing the identity of the subjects.

[0007] Based on this, the present invention aims to provide a dynamic densely connected spatiotemporal feature decoupling network for identifying cross-view gait, which can mine the correlation between shallow and deep features and alleviate the problem of insufficient information expression ability. Summary of the invention

[0008] In order to make up for the shortcomings of the existing technical problems, the purpose of the present invention is to provide a dynamic densely connected spatiotemporal feature decoupling network for identifying cross-view gait, using dense spatiotemporal feature decoupling blocks and splicing operations to achieve the sharing of shallow network features and deep network features, so as to solve the problem of insufficient representation ability. At the same time, the enhanced convolution block attention mechanism is used to make the network focus on more important gait features; finally, in the feature enhancement processing module, the five-dimensional feature map is processed into multiple horizontal features and the features are batch normalized to enhance the representation ability and model generalization ability.

[0009] To achieve the above object, the present invention provides the following technical solutions:

[0010] A dynamic densely connected spatiotemporal feature disentangled network for cross-view gait recognition, including:

[0011] Initial feature processing module: The initial feature processing module extracts preliminary features of the data and reduces the time dimension to reduce the loss of computing resources;

[0012] Dynamic dense spatiotemporal decoupling feature extraction module, including dense spatiotemporal feature decoupling block and enhanced convolutional block attention mechanism. After the initial feature processing module, shallow spatiotemporal features are obtained as input. The dense spatiotemporal feature decoupling block realizes the sharing of shallow network features and deep network features to enhance the representation of features. At the same time, the enhanced convolutional block attention mechanism is used to improve the model's attention to the gait walking target.

[0013] The feature enhancement processing module, including temporal pooling, global average pooling, global maximum pooling, fully connected layer and BNNeck component, converts the features extracted by the network into multiple lateral gait features to enhance the representation ability and model generalization ability and remove redundant information.

[0014] In the present invention, the dynamic dense spatiotemporal decoupling feature extraction module includes three dense spatiotemporal feature decoupling blocks, three enhanced convolutional block attention mechanisms and a maximum pooling module; the specific data processing process of the dynamic dense spatiotemporal decoupling feature extraction module is as follows:

[0015] Using the output of the initial feature processing module as the input of the first dense spatiotemporal feature decoupling block to obtain the output of the first dense spatiotemporal feature decoupling block;

[0016] The input of the first dense spatiotemporal feature decoupling block and the output of the first dense spatiotemporal feature decoupling block are concatenated as the input of the first enhanced convolutional block attention mechanism, thereby obtaining the output of the first enhanced convolutional block attention mechanism;

[0017] The output of the attention mechanism of the first enhanced convolutional block is used as the input of the maximum pooling module to obtain the output of the maximum pooling module;

[0018] Using the output of the maximum pooling module as the input of the second dense spatiotemporal feature decoupling block to obtain the output of the second dense spatiotemporal feature decoupling block;

[0019] The input of the second dense spatiotemporal feature decoupling block and the output of the second dense spatiotemporal feature decoupling block are concatenated as the input of the second enhanced convolutional block attention mechanism, thereby obtaining the output of the second enhanced convolutional block attention mechanism;

[0020] The output of the attention mechanism of the second enhanced convolutional block is used as the input of the third dense spatiotemporal feature decoupling block, thereby obtaining the output of the third dense spatiotemporal feature decoupling block;

[0021] The input of the third dense spatiotemporal feature decoupling block is concatenated with the output of the third dense spatiotemporal feature decoupling block as the input of the third enhanced convolutional block attention mechanism, thereby obtaining the output of the second enhanced convolutional block attention mechanism.

[0022] In the present invention, each dense spatiotemporal feature decoupling block includes a backbone feature extraction stream and a spatiotemporal decoupling feature extraction stream. The specific data processing process is as follows:

[0023] The data input into the dense spatiotemporal feature decoupling block is processed by two traditional 3D convolution blocks in the backbone feature extraction flow to obtain the backbone feature output;

[0024] The output of the first traditional 3D convolution block of the trunk feature extraction flow is used as the input of the branch feature extraction flow, and the branch feature output is obtained through three convolution blocks;

[0025] The trunk feature output is added to the branch feature output and then concatenated with the input of the dense spatiotemporal feature decoupling block to obtain the final feature.

[0026] In the present invention, the enhanced convolutional block attention mechanism includes two attention mechanisms, namely, the enhanced channel attention mechanism and the spatial attention mechanism, which are connected in series to perform feature processing. The specific processing process is as follows:

[0027] First, the enhanced channel attention mechanism will be used as the first component for feature processing, setting different weights for the channel dimension; the detailed calculation process is as follows:

[0028]

[0029]

[0030] in, To enhance the input features of the convolutional block attention mechanism, the Sigmoid function is used to assign weights to the data after MLP, and then the channel weight matrix Mc is obtained, and then Mc is combined with the feature Perform element-wise multiplication to obtain the final enhanced channel attention mechanism module output

[0031] Secondly, the spatial attention mechanism will serve as another feature processing component after the enhanced channel attention mechanism, enhancing the output of the channel attention mechanism component It will be used as the input of the spatial attention mechanism module to obtain the spatial weight matrix. The detailed calculation process is as follows:

[0032] f(x,y)=C 7×7×7 (Concat(x,y)) (12)

[0033]

[0034] Among them, the Sigmoid function is used to weight the features after function f, so as to obtain the spatial feature weight Ms, and finally Ms is combined with the feature Perform element-wise multiplication to obtain the final output In addition, function f represents a concatenation operation followed by a convolution operation with a convolution kernel size of (7×7×7). MaxPool and AvgPool represent maximum pooling and average pooling, respectively.

[0035] In the present invention, the processing process of the feature enhancement processing module is as follows:

[0036] First, temporal pooling and global pooling operations are used to extract important features and enhance the representation ability of gait features.

[0037] Then, a fully connected layer is used to integrate effective information, and a nonlinear activation function is used to enhance the representation and generalization capabilities of the model.

[0038] Finally, BNNeck is used to reduce overfitting, accelerate model training, reduce gradient vanishing or exploding problems, and enhance the robustness and generalization capabilities of the model.

[0039] In the present invention, the losses of multiple lateral gait features are calculated using a triplet loss function and a cross entropy loss function respectively, and are combined for parameter optimization in the hope of obtaining better model parameters.

[0040] In the present invention, the losses are calculated using the triplet loss function and the cross entropy loss function for multiple lateral gait features respectively as follows:

[0041] Will and The triple loss function and cross entropy loss function are used to calculate the loss value of each horizontal feature component. It is the output after time pooling, global average pooling, global maximum pooling and full connection layer processing. It is the final output of BNNeck, and finally the joint loss is added to optimize the model parameters. The loss function is expressed as:

[0042] Loss = L tri +L ce (20)

[0043] The triplet loss is calculated for each lateral gait feature, and the calculation formula is obtained:

[0044]

[0045] Among them, anc. represents the anchor point, pos. represents the positive sample, neg. represents the negative sample, N tri represents the number of all non-zero loss term triplets; Q represents the horizontal component scale of each feature, which is also F out The Q-dimensional scale of the training set is: P, K is the number of subjects in each batch and the number of sequence samples of each subject (a total of PK sequence samples are input into the network), which can eventually form PK (PK-K) (K-1) triplets; m represents the margin parameter of the triple loss function; dist in formula (21) represents the distance difference from the anchor point to the positive example and the negative example in each triplet, which can be expressed in the form of:

[0046]

[0047] in, represents the qth horizontal component feature in the ath gait sequence sample of the ith subject, and is also the anchor feature in the current triple group. The D in formula (22) represents the similarity measurement function, and the Euclidean geometric distance is used as the similarity measurement function. The formula is as follows:

[0048] D(A,B)=||AB|| (23)

[0049] The loss value is calculated for each component feature, and the average loss is finally used as the cross entropy loss of each sample. The specific calculation formula is as follows:

[0050]

[0051] Among them, N represents the number of samples in each batch, Q represents the number of horizontal component features, K represents the number of categories, and y i,c,k represents the sign function, which takes 1 when the true category of sample i is the same as k, otherwise it takes 0; p i,c,k Represents the predicted probability that the qth level component feature of observation sample i belongs to category k.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] (1) This paper designs a network framework based on dynamic dense connection and spatiotemporal feature decoupling, and applies the idea of ​​feature reuse to the network we designed, so that the shallow gait features are directly transferred to the deep network, so that the network can better focus on the detailed gait features and abstract gait features, enhance the richness of features, and improve the robustness of the model. In addition, the method we proposed can obtain excellent recognition performance with less resource loss, reducing resource loss.

[0054] (2) The initial feature processing module in the present invention includes the structure, preprocessing method and initial feature extraction block of the original input data. The module can reduce the amount of calculation and remove invalid information, extract the region of interest of the moving human body from the input video sequence (reduce network parameters and calculation amount), and perform data normalization processing (accelerate network convergence, avoid gradient disappearance or explosion, enhance the robustness of the model and improve the generalization performance of the model); in addition, the initial feature extraction block can convert the input data into a feature representation that can be understood by the network, thereby providing more meaningful information for the subsequent network; in addition, the time dimension can also be reduced to reduce the number of network parameters and reduce resource loss.

[0055] (3) The dynamic dense spatiotemporal decoupling feature extraction module of the present invention has a dense spatiotemporal feature decoupling block and an enhanced convolutional block attention mechanism; the dense spatiotemporal decoupling feature extraction module uses a parallel structure and low-rank convolution to obtain gait spatiotemporal correlation features and spatiotemporal decoupling features, thereby alleviating the problem of insufficient information expression ability, allowing the network to more fully utilize spatial structure information and temporal correlation information, and the dynamic growth rate of the dynamic dense spatiotemporal decoupling feature extraction module can help the network obtain more gait feature information, improve the generalization ability of the model and the final feature representation. The enhanced convolutional block attention mechanism can better focus on the detailed features of the gait, which is conducive to mining key features such as stride, contour, and arm swing to improve the representation ability of the final feature.

[0056] (4) The feature enhancement processing module of the present invention can convert the features extracted by the network into multiple lateral gait features, remove certain redundant information, and help the loss function to optimize the network parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 It is a framework diagram of the present invention.

[0058] Figure 2 This is a schematic diagram of the dense spatiotemporal feature decoupling block structure of the present invention.

[0059] Figure 3 Schematic diagram of the principle of the enhanced convolutional block attention mechanism of the present invention.

[0060] Figure 4It is a schematic diagram of the principle of the feature enhancement processing module of the present invention.

[0061] Figure 5 Schematic diagram of the comparison between batch size and accuracy of the present invention on the CASIA-B dataset.

[0062] Figure 6 It is a bar chart showing the influence of the model layer number setting and model parameter quantity on the experimental results of the present invention.

[0063] Figure 7 This is the impact of different DST Blocks and Attention Blocks on the model recognition accuracy of the present invention. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0065] Combination Figure 1-7 A dynamic densely connected spatiotemporal feature decoupling network for cross-view gait recognition is specifically described as follows:

[0066] (I) Dynamic dense connection spatiotemporal feature decoupling network framework

[0067] Figure 1 The framework diagram of the network based on dynamic dense connection spatiotemporal feature decoupling proposed by us is shown. First, two three-dimensional convolutions in the initial feature processing module (IFPM) are used to extract preliminary features of the data, and the time dimension is reduced to reduce the loss of computing resources; secondly, dense spatiotemporal feature decoupling blocks and splicing operations are used to realize the sharing of shallow network features and deep network features, so as to solve the problem of insufficient representation ability. At the same time, the enhanced convolution block attention mechanism is used to make the network focus on more important gait features; then in the feature enhancement processing module, temporal pooling (TP), global average pooling (GAP), global maximum pooling (GMP), separated fully connected layer (SFC) and batch normalization neck (BNNeck) structures are used to process the five-dimensional feature map into multiple lateral features and batch normalize the features to enhance the representation ability and model generalization ability. Finally, the triple loss function (Triple LossFunction) and cross-entropy loss function (Cross-Entropy Loss Function) are used to calculate the loss of multiple lateral features, and the parameters of the model are optimized jointly in the hope of obtaining better model parameters.

[0068] (II) Dynamic dense spatiotemporal decoupling feature extraction module

[0069] We propose a dynamic dense spatiotemporal decoupling feature extraction module (DDST-FEM), which consists of two modules: dense spatiotemporal feature decoupling block (DST Block) (its structure is shown in Figure 2 As shown in the figure) and the enhanced convolutional block attention mechanism (E-CBAM) (whose structure is shown in the figure) Figure 3 shown).

[0070] The dynamic dense spatiotemporal decoupling feature extraction module includes three dense spatiotemporal feature decoupling blocks, three enhanced convolutional block attention mechanisms and a maximum pooling module. Considering that the channel dimension of each block of DenseNet is too low, this will lead to the loss of detailed information in the extracted gait features. Therefore, we redesigned the module and introduced a dynamic growth rate to ensure that the loss of computing resources is reduced while minimizing information loss. Specifically, the present invention replaces the fixed number of module channels of DenseNet with a dynamically changing number of module channels to help the network obtain more effective gait information. In addition, the design of the dense spatiotemporal feature decoupling block is to solve the problem of insufficient representation of gait features, and the design of the enhanced convolutional block attention mechanism is to solve the problem of insufficient ability of the model to focus on key information. More specific internal structures and design motivations will be introduced below.

[0071] Assume that after the initial feature processing module, shallow spatiotemporal features are obtained This feature will be used as the input of the dynamic dense spatiotemporal decoupling feature extraction module, and will be processed by 3 DST-A Blocks, 3 E-CABM Blocks and 1 M3D Block to obtain the output data Y F :

[0072] Y F ←DDST-FEM(X f )(1)

[0073] Among them, X f Represents Initial Block (such as Figure 1 ) is also the input of DDST-FEM. DDST-FEM(·) stands for dynamic dense spatiotemporal decoupling feature extraction module. It is the final feature output of this module.

[0074] (III) Dense spatiotemporal feature decoupling block

[0075] The dense spatiotemporal feature extraction block contains two branches. First, the backbone feature extraction flow (including two convolution blocks) and the spatiotemporal decoupling feature extraction flow (including three convolution blocks) are used to extract different features. Then the outputs of the two branches are added together. Finally, the input is concatenated with the added output of the two branches to obtain the output of the DST Block, as shown in Figure 2 As shown in the figure. The dense spatiotemporal feature extraction module is designed to solve two problems. The first is to achieve feature reuse so that shallow features can also be obtained and used in the deep network. Specifically, gait, as a recognition task that pays great attention to detail features, is more sensitive to detail features such as the subject's profile, arm swing, and stride. The shallow feature map in deep learning is relatively large, so it has more detail features (more advantageous for the network to obtain detail features). In contrast, the deep feature map is relatively small, so it contains more abstract semantic features (more advantageous for classification). The idea of ​​feature reuse can feed shallow features into the deep network to achieve the sharing of detail features and abstract semantic features. Therefore, we designed a dense spatiotemporal feature extraction block, which splices the input and output of each block through the splicing operation to pass the shallow detail features to the deep network to achieve feature reuse. Another problem to be solved is the extraction of spatiotemporal decoupled features. Traditional 3DConv may introduce some irrelevant information and insufficient feature expression capabilities, and it will also cause serious computational overhead due to too many module stacking. The design of the P3D[6] module utilizes low-rank convolution to obtain features, but the module does not focus on the extraction of spatiotemporal correlation features. Therefore, the present invention designs a dense spatiotemporal feature extraction block based on the P3D module. The trunk is used to extract spatiotemporal correlation features, while the branches are used to obtain decoupled spatiotemporal features. Finally, the representation of the features is enhanced by adding them together.

[0076] like Figure 2 As shown, the present invention designs, after experiments, the present invention selects DST-ABlock as the dense spatiotemporal feature extraction block in the DD-STNet method, taking the first DST-ABlock as an example ( Figure 1 DSTBlock1), the input data is the output X of the initialization module f DST-ABlock contains two different feature extraction branches, and these two branches are analyzed separately.

[0077] First, for the backbone feature extraction flow, the input data is X f ∈R B×C×T×H×W , while X f The main feature extraction flow will pass through two traditional 3D convolution blocks ( Figure 2 (d)), thus obtaining the main output Y f ″, and the detailed calculation process of the middle layer can be expressed as:

[0078] Y f ′=C 1×1×1 (X f ) (2)

[0079] Y f=C 3×3×3 (Y f ′) (3)

[0080] Among them, C 1×1×1 represents a convolutional layer of size (1×1×1), C 3×3×3 Y represents a convolutional layer of size (3×3×3). f ′ is the output of the convolution layer with a convolution kernel size of (1×1×1), Y f ″ is Y f ′The main output obtained after the convolution layer with a convolution kernel size of (3×3×3),

[0081] Secondly, for the branch feature extraction flow, the input data is Y f ′,Y f ′ will pass through two low-rank convolutions of the branch feature extraction flow ( Figure 2 (d) branch), thus obtaining the branch feature output Y f ″′, the detailed calculation process is as follows:

[0082]

[0083] Among them C 1×3×3 represents the convolution layer with a convolution kernel size of (1×3×3), which is used to extract gait spatial features. f1 ′ is the output of the spatial low-rank convolutional layer,

[0084]

[0085] Among them C 3×1×1 Represents a convolution layer with a convolution kernel size of (3×1×1), which is used to extract gait temporal features. is the output of the spatial low-rank convolutional layer,

[0086]

[0087] Among them C 1×1×1 Represents a convolution layer with a convolution kernel size of (1×1×1), which is mainly used to reduce the channel dimension and finally obtain the spatiotemporal decoupling features of gait after the branch feature extraction flow

[0088] Finally, the outputs of the trunk and branches are added to improve the representation of the features, and then the input and the added features are concatenated to obtain the final feature representation. The specific calculation process is as follows:

[0089]

[0090] in, represents the element-by-element addition operation, Concat(·) represents the concatenation operation (the concatenation dimension is in the channel dimension), and the final output feature

[0091] (IV) Enhanced convolutional block attention mechanism

[0092] The E-CBAM module designed in this paper contains two attention mechanisms, namely the enhanced channel attention mechanism (E-CAM) and the spatial attention mechanism (SAM). Considering that the traditional CBAM module will lose some gait feature information, we remove the maximum pooling and average pooling in the channel attention to better adapt to the gait feature map (such as Figure 3 ); In addition, the spatial attention mechanism can help the model capture long-range dependencies and local information more effectively when processing gait data (such as Figure 3 ). The simultaneous use of these two attention mechanisms can improve the model's attention to the gait walking target, thereby enhancing the interpretability of the deep learning model. Specifically, from the perspective of human vision, continuous attention to the entire scene may prevent people from finding important information, so humans will focus on certain more important information through observation, and this is the attention mechanism in the human visual system. Gait, as a behavioral activity that pays more attention to detailed features, focuses on the outline of human walking, arm swing amplitude, stride size and other parts. It is the key to obtain better recognition performance. In addition, the CAM module in the traditional CBAM mechanism uses pooling to reduce the size of the feature map, which may achieve better recognition results in other recognition tasks, but gait is a recognition task that needs to pay attention to detailed features, which may lead to the loss of gait feature information. Therefore, in order to obtain more representative gait features, we redesigned the module, such as Figure 3 shown.

[0093] The E-CBAM module includes two attention mechanisms, E-CAM and SAM, and uses a serial approach to process features (e.g. Figure 3 ). Take the first E-CBAM module as an example for analysis, and the input data is Output after E-CBAM The overall calculation formula is as follows:

[0094]

[0095] In addition, the two modules are introduced in more detail. First, E-CAM will be the first component to be feature processed. The features obtained in the DST-A block It will be used as the input of E-CAM to set different weights for the channel dimension. The detailed calculation process is as follows:

[0096]

[0097] Among them, the Sigmoid function is used to assign weights to the data after MLP, and then the channel weight matrix Mc (Formula 10) is obtained, and then Mc is combined with the feature Perform element-wise multiplication to obtain the final E-CAM module output

[0098] Secondly, SAM will be another feature processing component after E-CAM. The output of E-CAM component It will be used as the input of the SAM module to obtain the spatial weight matrix. The detailed calculation process is as follows:

[0099] f(x,y)=C 7×7×7 (Concat(x,y)) (12)

[0100]

[0101] Among them, the Sigmoid function is used to weight the features after function f, so as to obtain the spatial feature weight Ms (Formula 13), and finally Ms is combined with the feature Perform element-wise multiplication to obtain the final output In addition, function f represents a concatenation operation followed by a convolution operation with a convolution kernel size of (7×7×7) (Formula 12), and MaxPool and AvgPool represent maximum pooling and average pooling, respectively.

[0102] (V) Feature Enhancement Processing Module

[0103] The feature enhancement processing module includes time pooling, global average pooling, global maximum pooling, fully connected layers, and BNNeck components. Considering that the features obtained by the dynamic dense spatiotemporal decoupling feature extraction module contain redundant information, we designed a feature enhancement processing module to process the features obtained by the network. The processing process involves some changes in feature dimensions (such as Figure 4), the specific dimension changes are shown below. We adopt this design for several important purposes. First, the time pooling and global pooling operations are mainly used to extract more important features and enhance the representation ability of gait features. Then, the fully connected layer is used to integrate effective information and enhance the representation and generalization ability of the model through nonlinear activation functions. Finally, BNNeck is used to reduce overfitting, accelerate model training, reduce the problem of gradient disappearance or explosion, and enhance the robustness and generalization ability of the model.

[0104] The detailed calculation process of the feature enhancement processing module is as follows Figure 4 As shown in Figure 1, the overall framework adopts a sequential and parallel structure. The input data of this module is Figure 1 The Feature part, that is, Y F (Formula 1). The detailed calculation process will be given below, and the overall calculation is as follows:

[0105] F out ←FEPM(Y F )(15)

[0106] Among them, FEPM refers to the entire feature processing module, F out is the final output of BNNeck.

[0107] The feature Y obtained by the feature extraction module F The final output will be obtained through the processing of four modules. The detailed calculation process is as follows:

[0108] Y F ′=TP(Y F ) (16)

[0109]

[0110] Y F ″′=SFC(Y F ″) (18)

[0111] F out =BNNeck(Y F ″′) (19)

[0112] Among them, TP represents time dimension pooling, and the output obtained is represents element-by-element addition operation, GAP and GMP represent global average pooling and global maximum pooling, and the output is SFC stands for Separate Fully Connected, which is used for feature mapping operations. The output Finally, the output F is obtained through BNNeck out .

[0113] (VI) Loss Function

[0114] For output and The triple loss function and cross entropy loss function are used to calculate the loss value of each horizontal feature component, and finally the joint loss is added to optimize the model parameters. The loss function can be expressed as:

[0115] Loss = L tri +L ce (20)

[0116] Influenced by the success of the triple loss function in the pedestrian re-identification task [8], the present invention adopts the batch all version of the triple loss function as one of the loss functions of the joint loss. This loss function is used in the model training stage, and each optimized sample consists of an anchor point (anc.), a positive sample (pos.) and a negative sample (neg.). The anchor point has the same label as the positive sample and a different label from the negative sample. In order to make fuller use of the processed gait features, a triple loss calculation is performed on each horizontal feature component, and the calculation formula can be obtained:

[0117]

[0118] Among them, N tri represents the number of all non-zero loss term triplets; Q represents the horizontal component scale of each feature, which is also F out The Q-dimensional scale of the model; (P, K) is the number of subjects in each batch and the number of sequence samples of each subject (a total of PK sequence samples are input into the network), which can eventually form PK (PK-K) (K-1) triplets; m represents the margin parameter of the triple loss function. The dist in formula (21) represents the distance difference from the anchor point to the positive example and the negative example in each triplet, which can be formally expressed as:

[0119]

[0120] in, Represents the qth horizontal component feature in the ath gait sequence sample of the i-th subject, and is also the anchor feature in the current triple group. and and have the same meaning, representing positive example features and negative example features respectively. D in formula (22) represents the similarity measurement function. The present invention adopts Euclidean geometric distance as the similarity measurement function, and the formula is as follows:

[0121] D(A,B)=||AB|| (23)

[0122] The cross entropy loss function is widely used in deep learning classification tasks. Its mathematical form is relatively simple, easy to understand and implement, and can better avoid the problem of gradient vanishing during gradient back propagation, which is helpful for stable network training. Therefore, the present invention also selects the cross entropy loss function as one of the members of the joint loss function. At the same time, in order to better utilize the final gait level feature component, the present invention will calculate the loss value of each component feature, and finally use the average loss as the cross entropy loss of each sample. The specific calculation formula is as follows:

[0123]

[0124] Among them, N represents the number of samples in each batch, Q represents the number of horizontal component features, K represents the number of categories, and y i,c,k represents a sign function (0 or 1), which takes 1 when the true category of sample i is the same as k, otherwise it takes 0. i,c,k Represents the predicted probability that the qth level component feature of observation sample i belongs to category k.

[0125] (VII) Experimental data

[0126] (7.1) Dataset Introduction

[0127] We selected two popular human action recognition datasets to validate the proposed method, namely CASIA-B[9] and OUMVLP

[10] .

[0128] The CASIA-B dataset was taken by the Institute of Automation, Chinese Academy of Sciences. It has become the most widely used gait dataset. It contains RGB and silhouette images of 124 people, a total of 13,680 sequence videos, and is a large-scale multi-view public gait dataset. The dataset considers three different walking conditions, namely normal walking (NM), walking with a backpack (BG), and walking with a coat (CL). In addition, the dataset was shot from 11 perspectives, ranging from 0° to 180° (with one perspective every 18°). In the end, each subject has 6 (numbered NM#01-06), 2 (numbered CL#01-02), and 2 (numbered BG#01-02) sequence videos at each perspective. In order to compare the performance with the current methods fairly, we strictly follow the evaluation protocols of

[11] ,

[12] , and [1]. The protocol sets the rules for dividing the training set and the test set, including small sample training (ST), medium sample training (MT), and large sample training (LT). We use large sample training, that is, all the data of 74 people are used to train the model, and the data of the remaining 50 people will be used to build the library and probe set for accuracy testing. For the test phase, the data numbered NM#01-04 is used to generate the library set, while the data numbered NM#05-06, BG#01-02 and CL#01-02 are used to generate the probe set.

[0129] OUMVLP is a multi-view large sample dataset. It is currently the largest public gait dataset, including 10,307 subjects. The shooting angle range includes 0°~90° (interval of 15°) and 180°~270° (interval of 15°), a total of 14 angles, including two sequences (numbered #00-01). Since the OUMVLP dataset is too large and considering the expensive computing cost, the present invention selects a total of 512 subjects in the dataset according to the general dataset division ratio, of which 256 subjects are used for training and the remaining 256 subjects are used for testing, and the data numbered #00 is used for the generation of the gallery set, and the data numbered #01 is used for the generation of the probe set.

[0130] (7.2) Experimental environment configuration and parameter setting

[0131] For all experimental processes, the gait sequence is resized and cropped, the frame length of the sample is set to 30, the height of each frame is set to 64, and the width is set to 44. At the same time, the present invention does not take any data enhancement operations, such as horizontal flipping, random occlusion, rotation, etc. The margin parameter of the triple loss function is set to 0.2, SGD is used as the optimizer of the model, and the initial learning rate (Learning Rate, lr) and weight decay (weight_decay) parameters of the optimizer are set to 0.1 and 0.0005. For the CASIA-B data set, the present invention uses the data of 74 subjects as the training set, and the batch size (batch size) is set to 24 (P×K=4×6) during training, and the maximum number of iterations is set to 80K times. For the OUMVLP data set, considering the expensive computational overhead cost, in order to verify the universality of the method of the present invention, the present invention selects the data of 256 subjects as the training set, sets the batch size (batch size) to 24 (P×K=6×4), and sets the maximum number of iterations to 120K times. For the final test phase, Euclidean distance is used as the final similarity measurement tool. The detailed experimental environment and computer configuration are in Table 1.

[0132] Table 1 Experimental environment and configuration

[0133]

[0134] (7.3) Experimental results on the CASIA-B dataset

[0135] In this section, the proposed method DD-ST Net will be compared with several state-of-the-art methods on the CASIA-B dataset in terms of recognition accuracy, including GEINet

[13] (ICB 2016), CNN-LB

[14] (TPAMI2017), GaitNet

[15] (TPAMI 2022), ACL[3](TIP 2020), GaitSet

[16] (TPAMI 2022), GaitPart

[17] (CVPR 2020), MvGAN

[18] (TIP 2021), MT3D[1](MM 2020) and GaitBase[2](CVPR 2023). In view of the specific situation of the comparison, the present invention conducts a more systematic and comprehensive experiment to compare the results, that is, experiments are conducted under all cross-view and cross-walk conditions on CASIA-B (excluding the same view). Specifically, all the data of 74 subjects are first used to train the model to obtain the feature extraction model. Then, the data of another 50 subjects with labels NM#01-04 will be used as the gallery set, and the data with labels NM#05-06, BG#01-02 and CL#01-02 will be used as the probe set to evaluate the performance of the method. Finally, the Euclidean geometric distance is used to calculate the feature similarity between the probe and the gallery. The experimental results in Table 2 show that the average performance of DD-ST Net compared with multiple methods exceeds most methods and reaches the current mainstream recognition accuracy level. The method of the present invention achieved an average recognition accuracy of 96.12%, 91.61% and 76.44% under the conditions of NM, BG and CL, respectively. Compared with the methods listed in Table 2, the method of the present invention obtains excellent recognition performance, which fully proves the feasibility of DD-ST Net.

[0136] Table 2 Cross-view average Grade 1 accuracy (%) for different probe views on CASIA-B, excluding identical views.

[0137]

[0138] Analyzing the method proposed in the present invention from another perspective, it is found that the method of the present invention can achieve excellent Rank-1 recognition accuracy under a smaller batch size. In order to more intuitively demonstrate the advantages of the method of the present invention under a smaller batch size, the present invention selects several of the most advanced gait recognition methods on the CASIA-B dataset to compare batch sizes and accuracy under three walking conditions (NM, BG and CL) and only under normal walking conditions (NM). The methods include: ACL, GaitSet, GaitPart, MT3D, 3DLocal[5], GaitGL

[19] , CSTL

[20] , GaitSlice[4] and OpenGait[2]. Observation Figure 5 It can be found that the proposed method can obtain excellent recognition performance in the case of small batch size. Under the three walking conditions, the proposed method can obtain an average accuracy of 88.1% when the batch size is 24 ( Figure 5 (a)), while an average accuracy of 96.1% can be obtained under normal walking conditions only ( Figure 5 (b)). This shows that the method proposed in the present invention can achieve excellent cross-view gait recognition accuracy in a greener and lower-carbon manner (less resource loss).

[0139] Table 3 Cross-view average Rank-1 accuracy (%) of OUMVLP (excluding the same view)

[0140]

[0141] (7.4) Experimental results on the OUMVLP dataset

[0142] In order to verify the generalization of the method of the present invention, the OUMVLP dataset was selected as a generalization verification dataset. However, considering that the OUMVLP dataset is too large and the computational overhead is expensive, part of the data was finally selected for verification. Specifically, a total of 512 people's data were selected for the experiment, of which all the data of 256 people were used for model training, and the other 256 people were used for model performance testing, and all 14 views were used for performance testing. In order to more comprehensively and systematically evaluate the performance of the method of the present invention, the present invention selected several well-known methods for comparison, including: GEINet (ICB 2016), DULE

[21] (CVPR 2021), GaitSet (TPAMI 2022), and GaitPart (CVPR 2020). Table 3 shows the comparison of the accuracy of the method of the present invention with other methods. The method of the present invention exceeds most methods and reaches the level of current mainstream methods, which also proves that the method of the present invention has a certain generalization ability.

[0143] (7.5) Ablation experiment

[0144] The proposed method includes the network settings and several key components, including DSTBlock, E-CBAM Block and BNNeck. In this section, we will explore the effectiveness of these network settings and components, and further analyze the network settings and components based on the experimental results.

[0145] Table 4 Average Rank-1 recognition accuracy (%) of DD-ST Net ablation experiment on CASIA-B dataset

[0146]

[0147] (7.5.1) The influence of network depth on experimental results. In order to explore the influence of network depth on recognition performance and model parameter quantity, we conducted two groups of control experiments, including Group Base (3-layer DST Block and E-CBAM Block) and Group 1 (4-layer DST Block and E-CBAM Block) in Table 4. Figure 6 From the experimental results of (a), we can find that deeper network layers do not bring better performance to DD-ST Net. In contrast, the average recognition rate of Group 1 using a shallow network structure is 0.8% higher than that of Group under NM conditions, 1.3% higher under BG conditions, and 2.8% higher under CL conditions. The average Rank-1 recognition accuracy of the three conditions is 1.6% higher. In addition, we observe Figure 6 From the comparison bar chart of model parameter quantities in (b), it can be found that the number of model parameters of Group 1 has increased significantly compared to that of Group Base.

[0148] Table 5 Average Rank-1 recognition accuracy using different network growth rates (%)

[0149]

[0150] (7.5.2) The impact of dynamic growth rate on experimental results. In order to explore the impact of the number of channels on model performance, we designed two different sets of experiments in the hope of verifying the impact of different network widths on model accuracy. Table 5 shows the impact of different model growth rates on model performance. It can be seen that the use of dynamic model growth rate can obtain a larger number of channels. At the same time, Group B has a 1.9% higher recognition rate than A in the NM condition, 2.9% higher in the BG condition, and 3.8% higher in the CL condition. The average Rank-1 recognition accuracy of the three conditions is 2.8% higher. Analysis of the reason is that in deep learning, the wider the network is within a certain range, the better the performance will be, and the width refers to the number of channels. When the number of channels increases, it can provide more information and feature representation space, which can enable the model to better capture the complexity and diversity of the input data, so the expression ability will also be improved to a certain extent. The dynamic growth rate we designed can make the number of channels of each DST Block of the model more sufficient, which ensures that the model can obtain more information, while the original DenseNet network does not use a dynamic growth rate, which leads to information loss. Therefore, we use a dynamic growth rate to reduce information loss and obtain better recognition performance.

[0151] (7.5.3) The impact of DST Block on experimental results. In order to explore the impact of using two different modules or using them together on model performance, we designed four different ablation experiments, including Group Base, 2, 3, and 4 in Table 4. Group Base uses DST-A Block, Group 2 uses DST-B Block, Group 3 uses both DST-A and DST-B Block, and Group 4 does not use DST-A and DST-B Block, but only uses the trunk branch ( Figure 2 (a)). In order to more intuitively observe the impact of different DST Blocks on model performance, the present invention draws a bar chart to compare the recognition accuracy ( Figure 7(a)), observing the experimental accuracy results of the bar graph, we found that the best recognition effect was obtained by using DST-ABlock, while the accuracy of the mixed DST-A and DST-B Block was the lowest. Among them, the Rank-1 recognition accuracy of the model using DST-A Block is 0.9%, 1.7%, 4.5% and 1.2%, 1.8%, 4.4% higher than the original 3D convolution module and DST-B Block under NM, BG, and CL walking conditions, respectively. The design of DST-A Block has a crucial impact on the performance of DD-ST Net. The module adopts a dual-branch design, which can help the model obtain spatiotemporal correlation features and spatiotemporal decoupling features. Therefore, DST-A Block is used as one of the important components of DD-ST Net to improve the expressiveness of features.

[0152] (7.5.4) The impact of attention mechanism on experimental results. In order to explore the impact of different E-CBAM settings on the performance of the model, we designed five different ablation experiments, including Group Base, 5, 6, 7 and 8 in Table 4, where Group Base uses the E-CBAM module, Group 5 uses the E-CAM module, Group 6 uses the E-SAM module, Group 7 uses the original CBAM module, and Group 8 does not use the E-CBAM or CBAM module. In order to more intuitively observe the comparison of the accuracy of these groups, the present invention draws the Rank-1 recognition accuracy bar chart of these groups ( Figure 7 (b)), we can find that the average recognition accuracy of E-CBAM is 0.5%, 4.1%, 11.6% and 0.9% higher than that of the other groups (arranged in order of groups). The attention mechanism plays an indispensable role in deep learning. The attention mechanism can help the model better capture key information and better adapt to changes in input data to improve the robustness of the model, thereby improving the performance and efficiency of the model. Therefore, E-CBAM is used as one of the important components of the model to help the model better focus on key gait information and improve recognition performance.

[0153] (7.5.5) Impact of BNNeck on experimental results. In order to verify the impact of this component on the method of the present invention, we designed two groups of ablation experiments, including Group Base (using BNNeck module) and Group 9 (not using BNNeck module) in Table 4. By observing the experimental results in Table 4, we found that the average recognition rate of Group 9 using the BNNeck layer was 0.1% higher than that of Group under NM conditions, 0.6% higher under BG conditions, and 2.0% higher under CL conditions. The average Rank-1 recognition accuracy of the three conditions was 1.0% higher. BNNNeck[7] combines the concepts of batch normalization and dimensionality reduction to improve the expressiveness of features, reduce overfitting, and enhance the robustness of the model. Therefore, BNNeck is used as one of the components of the model to improve the recognition performance.

[0154] What is described above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

[0155] References:

[0156] 1.Lin,BB,et al.Gait Recognition with Multiple-Temporal-Scale 3DConvolutional Neural Network.in 28th ACM International Conference onMultimedia(MM).2020.Electr Network.

[0159] 2.Fan, C., et al., OpenGait: Revisiting Gait Recognition Toward BetterPracticality. 2023 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023: p.9707-9716.

[0161] 3.Zhang,Y.Q.,et al.,Cross-View Gait Recognition by DiscriminativeFeature Learning.Ieee Transactions on Image Processing,2020.29:p.1001-1015.

[0162] 4.Li,H.K.,et al.,GaitSlice:A gait recognition model based on spatio-temporal slice features.Pattern Recognition,2022.124.

[0163] 5.Huang,Z.a.X.,Dixiu and Shen,Xu and Tian,Xinmei and Li,Houqiang andHuang,Jianqiang and Hua,Xian-Sheng,3D Local Convolutional Neural Networks forGait Recognition,in 2021IEEE / CVF International Conference on Computer Vision(ICCV).2021.p.14900-14909.

[0165] 6.Xing,X.L.,et al.,Complete canonical correlation analysis withapplication to multi-view gait recognition.Pattern Recognition,2016.50:p.107-117.

[0166] 7.Sun,Y.,et al.Beyondpart models:Person retrieval with refinedpartpooling(and astrong convolutional baseline).in Proceedings ofthe Europeanconference on computer vision(ECCV).2018.

[0167] 8.Hermans,A.,L.Beyer,and B.Leibe,In Defense ofthe Triplet Loss forPerson Re-Identification.ArXiv,2017.abs / 1703.07737.

[0168] 9.Shiqi,Y.,T.Daoliang,and T.Tieniu,A framework for evaluating theeffect ofview angle,clothing and carrying condition on gaitrecognition.200618th International Conference on Pattern Recognition,2006:p.4pp.-4pp.

[0169] 10.Takemura,N.,et al.,Multi-view large population gait dataset andits performance evaluation for cross-view gait recognition.IPSJ Transactionson Computer Vision and Applications,2018.10(1):p.4(14pp.)-4(14pp.).

[0170] 11.He,Y.W.,et al.,Multi-Task GANs for View-Specific Feature Learningin Gait Recognition.Ieee Transactions on Information Forensics and Security,2019.14(1):p.102-113.

[0171] 12.Chao,H.,et al.GaitSet:Regarding gait as a set for cross-view gaitrecognition.in 33rd AAAI Conference on Artificial Intelligence,AAAI 2019,31stAnnual Conference on InnovativeApplications ofArtificial Intelligence,IAAI2019 and the 9th AAAI Symposium on EducationalAdvances in ArtificialIntelligence,EAAI 2019,January 27,2019-February 1,2019.2019.Honolulu,HI,United states:AAAI Press.

[0172] 13.Shiraga,K.,et al.GEINet:View-Invariant Gait Recognition Using aConvolutionalNeural Network.in 2016 International Conference on Biometrics(ICB).2016.

[0173] 14.Wu,Z.F.,et al.,A Comprehensive Study on Cross-View Gait BasedHumanIdentification with Deep CNNs.Ieee Transactions on Pattern Analysis andMachine Intelligence,2017.39(2):p.209-226.

[0174] 15.Zhang,Z.Y.,et al.,On Learning Disentangled Representations forGait Recognition.Ieee Transactions on Pattern Analysis and MachineIntelligence,2022.44(1):p.345-360.

[0175] 16.Chao,H.,et al.,GaitSet:Cross-View Gait Recognition ThroughUtilizing Gait As aDeep Set.IEEE Transactions on Pattern Analysis and MachineIntelligence,2022.44(7):p.3467-3478.

[0176] 17.Chao,F.,et al.GaitPart:Temporal Part-Based Model for GaitRecognition.in 2020IEEE / CVF Conference on Computer Vision andPatternRecognition(CVPR).Proceedings.2020.

[0177] 18.Chen,X.,et al.,Multi-View Gait Image Generation for Cross-ViewGait Recognition.Ieee Transactions on Image Processing,2021.30:p.3041-3055.

[0178] 19.Lin,B.,S.Zhang,and X.Yu.Gait Recognition via Effective Global-Local FeatureRepresentation and Local Temporal Aggregation.in 2021 IEEE / CVFInternational Conferenceon Computer Vision(ICCV).2021.

[0179] 20.Huang,X.,et al.Context-Sensitive Temporal Feature Learning forGait Recognition.in2021 IEEE / CVF International Conference on Computer Vision(ICCV).2021.

[0180] 21.Zhang,S.,Y.Wang,and A.Li.Cross-View Gait Recognition with DeepUniversalLinear Embeddings.in 2021 IEEE / CVF Conference on Computer Vision andPatternRecognition(CVPR).2021.

Claims

1. A dynamic densely connected spatiotemporal feature decoupling network for cross-view gait recognition, characterized by: include: Initial feature processing module: The initial feature processing module extracts preliminary features of the data and reduces the time dimension to reduce the loss of computing resources; Dynamic dense spatiotemporal decoupling feature extraction module, including dense spatiotemporal feature decoupling block and enhanced convolutional block attention mechanism. After the initial feature processing module, shallow spatiotemporal features are obtained as input. The dense spatiotemporal feature decoupling block realizes the sharing of shallow network features and deep network features to enhance the representation of features. At the same time, the enhanced convolutional block attention mechanism is used to improve the model's attention to the gait walking target. The feature enhancement processing module, including temporal pooling, global average pooling, global maximum pooling, fully connected layer and BNNeck component, converts the features extracted by the network into multiple lateral gait features to enhance the representation ability and model generalization ability and remove redundant information.

2. The dynamic densely connected spatiotemporal feature decoupling network for identifying cross-view gait according to claim 1, characterized in that: The dynamic dense spatiotemporal decoupling feature extraction module includes three dense spatiotemporal feature decoupling blocks, three enhanced convolutional block attention mechanisms and a maximum pooling module; the specific data processing process of the dynamic dense spatiotemporal decoupling feature extraction module is as follows: Using the output of the initial feature processing module as the input of the first dense spatiotemporal feature decoupling block to obtain the output of the first dense spatiotemporal feature decoupling block; The input of the first dense spatiotemporal feature decoupling block and the output of the first dense spatiotemporal feature decoupling block are concatenated as the input of the first enhanced convolutional block attention mechanism, thereby obtaining the output of the first enhanced convolutional block attention mechanism; The output of the attention mechanism of the first enhanced convolutional block is used as the input of the maximum pooling module to obtain the output of the maximum pooling module; Using the output of the maximum pooling module as the input of the second dense spatiotemporal feature decoupling block to obtain the output of the second dense spatiotemporal feature decoupling block; The input of the second dense spatiotemporal feature decoupling block and the output of the second dense spatiotemporal feature decoupling block are concatenated as the input of the second enhanced convolutional block attention mechanism, thereby obtaining the output of the second enhanced convolutional block attention mechanism; The output of the attention mechanism of the second enhanced convolutional block is used as the input of the third dense spatiotemporal feature decoupling block, thereby obtaining the output of the third dense spatiotemporal feature decoupling block; The input of the third dense spatiotemporal feature decoupling block is concatenated with the output of the third dense spatiotemporal feature decoupling block as the input of the third enhanced convolutional block attention mechanism, so as to obtain the output of the second enhanced convolutional block attention mechanism.

3. The dynamic densely connected spatiotemporal feature decoupling network for identifying cross-view gait according to claim 1, characterized in that: Each dense spatiotemporal feature decoupling block includes a backbone feature extraction stream and a spatiotemporal decoupling feature extraction stream. The specific data processing process is as follows: The data input into the dense spatiotemporal feature decoupling block is processed by two traditional 3D convolution blocks in the backbone feature extraction flow to obtain the backbone feature output; The output of the first traditional 3D convolution block of the trunk feature extraction flow is used as the input of the branch feature extraction flow, and the branch feature output is obtained through three convolution blocks; The trunk feature output is added to the branch feature output and then concatenated with the input of the dense spatiotemporal feature decoupling block to obtain the final feature.

4. The dynamic densely connected spatiotemporal feature decoupling network for identifying cross-view gait according to claim 1, characterized in that: The enhanced convolutional block attention mechanism includes two attention mechanisms, namely the enhanced channel attention mechanism and the spatial attention mechanism, which use a series connection method to process features. The specific processing process is as follows: First, the enhanced channel attention mechanism will be used as the first component for feature processing, setting different weights for the channel dimension; the detailed calculation process is as follows: in, To enhance the input features of the convolutional block attention mechanism, the Sigmoid function is used to assign weights to the data after MLP, and then the channel weight matrix Mc is obtained, and then Mc is combined with the feature Perform element-wise multiplication to obtain the final enhanced channel attention mechanism module output Secondly, the spatial attention mechanism will serve as another feature processing component after the enhanced channel attention mechanism, enhancing the output of the channel attention mechanism component It will be used as the input of the spatial attention mechanism module to obtain the spatial weight matrix. The detailed calculation process is as follows: f(x,y)=C 7×7×7 (Concat(x,y)) (12) Among them, the Sigmoid function is used to weight the features after function f, so as to obtain the spatial feature weight Ms, and finally Ms is combined with the feature Perform element-wise multiplication to obtain the final output In addition, function f represents a concatenation operation followed by a convolution operation with a convolution kernel size of (7×7×7). MaxPool and AvgPool represent maximum pooling and average pooling, respectively.

5. The dynamic densely connected spatiotemporal feature decoupling network for identifying cross-view gait according to claim 1, characterized in that: The processing process of the feature enhancement processing module is as follows: First, temporal pooling and global pooling operations are used to extract important features and enhance the representation ability of gait features. Then, a fully connected layer is used to integrate effective information, and a nonlinear activation function is used to enhance the representation and generalization capabilities of the model. Finally, BNNeck is used to reduce overfitting, accelerate model training, reduce gradient vanishing or exploding problems, and enhance the robustness and generalization capabilities of the model.

6. The dynamic densely connected spatiotemporal feature decoupling network for identifying cross-view gait according to claim 1, characterized in that: The losses of multiple lateral gait features are calculated using the triplet loss function and the cross entropy loss function respectively, and they are jointly used for parameter optimization in the hope of obtaining better model parameters.

7. The dynamic densely connected spatiotemporal feature decoupling network for identifying cross-view gait according to claim 6, characterized in that: The losses are calculated using the triplet loss function and the cross entropy loss function for multiple lateral gait features as follows: Will and The triple loss function and cross entropy loss function are used to calculate the loss value of each horizontal feature component. It is the output after time pooling, global average pooling, global maximum pooling and full connection layer processing. It is the final output of BNNeck, and finally the joint loss is added to optimize the model parameters. The loss function is expressed as: Loss=L tri +L ce (20) The triplet loss is calculated for each lateral gait feature, and the calculation formula is obtained: Among them, anc. represents the anchor point, pos. represents the positive sample, neg. represents the negative sample, N tri represents the number of all non-zero loss term triplets; Q represents the horizontal component scale of each feature, which is also F out The Q dimension scale; P, K are the number of subjects in each batch and the number of sequence samples of each subject, which ultimately constitutes PK(PK-K)(K-1) triplets; m represents the margin parameter of the triple loss function; dist in formula (21) represents the distance difference from the anchor point to the positive example and the negative example in each triplet, which is formally expressed as: in, represents the qth horizontal component feature in the ath gait sequence sample of the ith subject, and is also the anchor feature in the current triple group. The D in formula (22) represents the similarity measurement function, and the Euclidean geometric distance is used as the similarity measurement function. The formula is as follows: D(A,B)=||AB|| (23) The loss value is calculated for each component feature, and the average loss is finally used as the cross entropy loss of each sample. The specific calculation formula is as follows: Among them, N represents the number of samples in each batch, Q represents the number of horizontal component features, K represents the number of categories, and y i,c,k represents the sign function, which takes 1 when the true category of sample i is the same as k, otherwise it takes 0; p i,c,k Represents the predicted probability that the qth level component feature of observation sample i belongs to category k.

Citation Information

Patent Citations

  • Cross-view gait recognition method based on spatio-temporal information enhancement and multi-scale saliency feature extraction

    CN113947814A

  • Multi-feature fusion gait recognition method based on space-time dimension screening

    CN115439930A

  • Cross-view gait recognition method based on adversarial domain adaptive learning

    CN116704612A

  • Multi-gait recognition method based on multi-feature fusion

    CN117456608A

  • Gait recognition system based on multi-scale 3D convolution and space-time attention

    CN117809373A