Pulse neural network driven space-frequency domain pedestrian gait feature extraction method

Abstract

The present application relates to the technical field of computer image processing, and specifically provides a kind of space-frequency domain pedestrian gait feature extraction method based on impulse neural network driving, gait image is extracted using space-frequency domain pedestrian gait feature extraction model;Space-frequency domain pedestrian gait feature extraction model includes: main network, multi-stage time domain global feature interaction enhancement module and discriminative feature aggregation module;Main network includes N stages connected in turn, each stage includes an air-frequency feature fusion module for fusing spatial domain features and frequency domain features;Multi-stage time domain global feature interaction enhancement module extracts and enhances spatial domain global features and time domain global features;Discriminative feature aggregation module aggregates the output of main network with the output of multi-stage time domain global feature interaction enhancement module.This method can improve the model's ability to capture gait micro-detail features and enhance its adaptability to complex scenarios.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer image processing, and more particularly to a method for extracting gait features of pedestrians in space-frequency domain based on a pulse neural network. BACKGROUND

[0002] Gait recognition technology uses the unique walking pattern of an individual to achieve identity authentication, which is an important direction in the field of biometric recognition. This technology has shown great potential in intelligent security, crime prevention, judicial authentication, and dynamic crowd tracking in many key fields.

[0003] There are still some important obstacles to overcome in the practical application of this technology. For example, the recognition difficulty caused by target object occlusion, sudden changes in environmental lighting conditions, and differences in viewing angles, as well as the complexity of the background environment, can all affect the recognition accuracy. In addition, factors such as the individual's own health status, emotional fluctuations, and changes in walking speed can also cause natural fluctuations in gait features, which requires the recognition algorithm to have higher environmental adaptability and robustness.

[0004] In 2016, Shiraga et al. proposed GEINet, which uses a six-layer convolutional network to extract features from gait energy maps (GEI). Since then, various deep learning models have been applied to gait recognition. Nair et al. developed the DBNGait model based on deep Bayesian networks. The GaitSet model designed an efficient and lightweight network architecture, achieving a cross-view recognition accuracy of 70.3% on the CASIA-B dataset under the condition of clothing change. The GaitBase model used a residual network architecture to deepen the network, significantly improving the recognition performance on outdoor datasets; the subsequent DeepGaitV2-2D further increased the network depth and verified the positive correlation between recognition accuracy and model depth on outdoor datasets. The BigGait model introduced a self-supervised learning module, extracting general features through a general feature extraction module (LVM), and then converting these general features into features specific to gait recognition using a gait recognition expert module (GRE).

[0005] The rapid development of spiking neural networks (SNNs) has led to their expansion into multiple fields. In gait recognition, the GaitSpike model was the first to combine traditional SNNs with gait recognition algorithms. However, there is relatively little research on SNN-based gait recognition methods, and new frameworks need to be designed to promote the further development of such networks in the field of gait recognition.

[0006] In visual tasks such as target detection, pose estimation and machine time series signal processing, networks based on frequency domain (such as Fourier transform or wavelet transform) have been applied. These methods convert data into frequency domain space for filtering and feature extraction. For example, in the field of pose estimation, the discrete Fourier transform (DTC) is used to extract the frequency domain features of the entire time series as supplementary information of the joint space position and motion details. These frequency domain features are combined with the analysis results of the joint space domain of a single frame and input into the Transformer for cross-frame time dependence modeling. In terms of attention mechanism, the GFN model proposes a learnable frequency filter for efficient hybrid visual markers, and the AFNO further improves the performance of the frequency domain marker mixer through operator learning. However, the above-mentioned frequency domain based methods have not been fully explored and applied in the field of gait recognition.

[0007] The core defects of existing gait recognition technology are:

[0008] (1) Insufficient dimension information mining of spatial and frequency domain features: existing methods rely more on spatial domain contour features, and do not mine frequency domain dynamic information (such as step frequency harmonic components and joint motion spectrum), resulting in loss of motion details and difficulty in distinguishing key identity markers such as step length variation and step frequency difference;

[0009] (2) Lack of high and low frequency feature extraction and fusion mechanism: there is a lack of special module to separate low frequency and high frequency details (joint micro-motion), resulting in insufficient adaptability of the algorithm to occlusion and view angle changes;

[0010] (3) Potential of spiking neural network application in gait recognition field has not been released: the spiking time coding characteristics of SNN are naturally adapted to gait dynamic modeling, but existing explorations (such as GaitSpike) have not designed architecture combined with spatial domain characteristics, and the spiking coding capability of space and time has not been fully utilized;

[0011] (4) Inefficient global feature interaction in spatial and temporal domains: there is no mechanism to fuse global spatial and global frequency domain features across dimensions, resulting in weak correlation between space and time features;

[0012] (5) Multi-stage feature fusion method is not robust: multi-scale time series features (such as gait cycle and single step micro-motion) are not fused hierarchically, resulting in a sharp drop in recognition rate in complex scenes. SUMMARY

[0013] In order to overcome the defects and shortcomings of the prior art, the purpose of the present application is to provide a spatial and frequency domain pedestrian gait feature extraction method based on a spiking neural network. The method fuses spatial and frequency domain features, models high and low frequencies respectively, takes advantage of the spiking neural network, and performs global spatial and temporal feature interaction to improve the ability of the model to capture gait micro-detail features and enhance the adaptability to complex scenes.

[0014] To achieve the above objectives, the present invention is implemented through the following technical solution: a spatial frequency domain pedestrian gait feature extraction method based on a spiking neural network, which uses a spatial frequency domain pedestrian gait feature extraction model to extract gait features from gait images;

[0015] The spatial-frequency domain pedestrian gait feature extraction model includes: a backbone network, a multi-stage temporal-domain global feature interaction enhancement module, and a discriminative feature aggregation module. The backbone network comprises N sequentially connected stages, each stage including a spatial-frequency feature fusion module for fusing spatial and frequency domain features. The multi-stage temporal-domain global feature interaction enhancement module extracts and enhances spatial and temporal global features from the output feature maps Si, i=1,2,…,N of each stage of the backbone network, to obtain the output feature map S. T_total The discriminative feature aggregation module combines the backbone network output with the feature map S output by the multi-stage temporal global feature interaction enhancement module. T_total Aggregation is performed to obtain the gait features finally output by the spatial frequency domain pedestrian gait feature extraction model.

[0016] Preferably, each space-frequency feature fusion module includes a spatial domain extraction branch and a frequency domain extraction branch; each space-frequency feature fusion module divides the feature map input to the space-frequency feature fusion module into two feature maps I1 and I2; the spatial domain extraction branch extracts spatial features from feature map I1 through two convolutional kernels; the frequency domain extraction branch performs a Fourier transform on feature map I2 and then separates the high-frequency and low-frequency components after the Fourier transform through two convolutional kernels, and then inputs them into the corresponding convolutional networks after inverse transform to obtain the low-frequency feature U in the spatial domain. 0L and high-frequency features U 0H High-frequency feature U 0H The high-frequency component weights W and low-frequency features U are obtained through a pulsating neural network LIF. 0L After adding the spatial features extracted by the spatial extraction branch, the dual-branch low-frequency fusion feature U1 is obtained; the dual-branch low-frequency fusion feature U1 is multiplied by the low-frequency component weight (1-W) to obtain the fused low-frequency feature U. 1L The low-frequency fusion feature U1 of the dual-branch system is multiplied by the high-frequency component weight W and then combined with the high-frequency feature U. 0H Add them together to obtain the fused high-frequency feature U. 1H Then, the fused low-frequency features U 1L and high-frequency features U 1H Further addition and fusion are performed to obtain the hybrid output feature map U2;

[0017] The feature map I is processed through a channel attention mechanism to obtain channel weights, which are then multiplied with the weights obtained from the spiking neural network LIF to obtain the updated channel attention weights W. C The channel attention weight W CThe feature map U based on the attention mechanism is obtained by multiplying the mixed output feature map U2 C After normalization, the updated features are input into the FFN feedforward neural network module, and finally added to the feature map I through the residual link module to obtain the final output feature map Si of the spatial-frequency feature fusion module.

[0018] Preferably, the feature maps I1 and I2 are I1, I2 = split(I); wherein split is a split function.

[0019] The low-frequency fusion feature U1 of the two-path branch is:

[0020] U1 = Conv2D ×2 (I1) + Conv2D(IFFT([Conv2D ×2 (FFT(I2)) LF ]) ) ;

[0021] Wherein, Conv2D ( ) is a 2D convolution; Conv2D ×2 ( ) is a 2D convolution of two convolution kernels; FFT ( ) is a Fourier transform; IFFT ( ) is an inverse Fourier transform; [ ] LF represents a low-frequency component;

[0022] The high-frequency component weight W is:

[0023] W = LIF(Conv2D(IFFT([Conv2D ×2 (FFT(I2)) HF ]) ) ;

[0024] Wherein, LIF ( ) is a spiking neural network; [ ] HF represents a high-frequency component;

[0025] The fused low-frequency feature U 1L and the high-frequency feature U 1H are respectively:

[0026] U 1L = U1 × (1-W) ;

[0027] U 1H = U1 × W + Conv2D(IFFT([Conv2D ×2 (FFT(I2)) HF ]) ) ;

[0028] The mixed output feature map U2 is:

[0029] U2 = Conv2D(BN(Conv2D(U 1L ) + BN(Conv2D(U1H ))) ;

[0030] wherein BN( ) is a batch normalization operation;

[0031] channel attention weight W C is:

[0032] W C = Tanh(attention(I) ×LIF(attention(I))) ;

[0033] wherein attention( ) is an attention mechanism; Tanh( ) is an activation function;

[0034] feature map U C based on the attention mechanism is:

[0035] U C =U2×W C ;

[0036] The final output feature map Si of the spatial-frequency feature fusion module is:

[0037] Si = FFN(Norm(U C ))+I;

[0038] wherein FFN ( ) is a feedforward neural network module; Norm( ) is a normalization operation.

[0039] Preferably, the multi-stage time-domain global feature interaction enhancement module comprises a time-domain global feature fusion module, a spatial-domain global feature enhancement network branch, and a time-domain global feature enhancement network branch;

[0040] The output feature map Si of each stage of the backbone network is calculated by spatial-domain pooling and time-domain pooling, respectively, to obtain spatial-domain global feature S si and time-domain global feature S Ti ; The spatial-domain global feature S si and the time-domain global feature S Ti are jointly input into the time-domain global feature fusion module to obtain spatial-time domain global feature S STi ; The spatial-domain global feature S si of each stage is merged with the spatial-time domain global feature S STi in the time dimension to form merged feature (S si , S STi );

[0041] The merged feature (S si , S STi) are respectively input into the spatial domain global feature enhancement network branch; the spatial domain global feature enhancement network branch is composed of N spatial domain enhancement convolution groups; the merged feature (S si , S STi ) of the current stage is added to the output feature map (S s i-1 , S ST i-1 ) of the spatial domain enhancement convolution group of the previous stage, and then is input into the spatial domain enhancement convolution group of the current stage for feature extraction to obtain the output feature map (S s i , S ST i ) of the spatial domain enhancement convolution group of the current stage.

[0042] The time domain global feature maps S Ti of each stage are respectively input into the time domain global feature enhancement network branch; the time domain global feature enhancement network branch is composed of N time domain enhancement convolution groups; the time domain global feature map S Ti of the current stage is added to the output feature map S T i-1 of the time domain enhancement convolution group of the previous stage, and then is input into the time domain enhancement convolution group of the current stage for feature extraction to obtain the output feature map S T i of the time domain enhancement convolution group of the current stage.

[0043] The output feature map (S s N , S ST N ) of the spatial domain enhancement convolution group of the last stage is divided into features S s N , S ST N ; the feature S s N is taken as the spatial domain global feature S s(N+1) , and the output feature map S T N of the time domain enhancement convolution group of the last stage is taken as the time domain global feature S T(N+1) , which are jointly input into the time domain global feature fusion module to obtain the output space-time domain global feature S ST(N+1) ; the feature S ST N is fused with the feature map S ST_temp ; the feature map S ST_temp is weighted and fused with the feature map S T N through the weight obtained by the activation function to obtain the final output feature map S T_total of the multi-stage time domain global feature interaction enhancement module.

[0044] Preferably, in the multi-stage time-domain global feature interaction enhancement module, the calculation method is as follows:

[0045] Initialize S s 0 = 0; S ST 0 = 0; S T 0 = 0;

[0046] The spatial domain global feature S si and the time domain global feature S Ti are respectively:

[0047] ;

[0048] ;

[0049] Wherein, T is the number of time frames, C, H, and W are respectively the channel number, height, and width of the feature map Si;

[0050] The output feature map (S s i ,S ST i ) of each stage spatial domain enhancement convolution group is:

[0051] (S s i ,S ST i ) = Conv2D ×2 (concat (S s i-1 ,S ST i-1 ) + concat (S si ,S STi ) ;

[0052] The output feature map S T i of each stage time domain enhancement convolution group is:

[0053] S T i = Conv2D ×2 (S T i-1 + S Ti );

[0054] The feature map S ST_temp is:

[0055] S ST_temp = Conv2D ×2 (f st (S sN ,S T N )+S ST N );

[0056] The final output feature map S of the multi-stage temporal global feature interaction enhancement module T_total for:

[0057] S T_total =sigmoid(S ST_temp ) ×sigmoid(S T N ) ×S ST_temp +S T N .

[0058] Preferably, the spatial global feature S si and temporal global features S Ti The common inputs are into the temporal global feature fusion module: spatial global feature S si After feature extraction via two layers of 3D convolution, a temporal attention score is obtained through a temporal attention mechanism module consisting of two layers of 3D convolution and activated in the temporal dimension. This temporal attention score is then compared with the spatial global feature S. si The fused features S are obtained by performing dot product weighting. si_temp Temporal global features S Ti After max pooling and average pooling, it is combined with the feature S si_temp The features are added together, and the spatiotemporal global features S output by the temporal global feature fusion module are obtained through two layers of 3D convolution. STi .

[0059] Preferably, the spatiotemporal global feature S output by the temporal global feature fusion module STi for:

[0060] S STi =f st (S si ,S Ti )

[0061] =Conv3D ×2 (Softmax(Conv3D ×2 (R(S si ))) ×Conv3D ×2 (R(S si ))+maxpool(S Ti )+meanpool(S Ti ));

[0062] Among them, f stdenotes a time domain global feature fusion module; R( ) is a reshape operation; Conv3D ×2 denotes two layers of 3D convolution; Softmax( ) is an activation function; maxpool( ) and meanpool( ) are spatial maximum pooling and average pooling operations, respectively.

[0063] Preferably, the output feature map SN of the last stage of the backbone network is subjected to a time domain pooling operation to obtain a feature map S m ; in the discriminative feature aggregation module, the feature map S m is aggregated with the output feature map S T_total of the multi-stage time domain global feature interaction enhancement module, and after feature fusion through convolution, is input into an HPM operation to obtain the output S out of the discriminative feature aggregation module:

[0064] S out = HPM(Conv2D ×2 (Cancat(S T_total , S m )));

[0065] wherein, HPM( ) is a block operation; Conv2D ×2 ( ) is a 2D convolution of two convolution kernels; and Cancat( ) is an aggregation operation.

[0066] A readable storage medium, wherein the storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the pulse neural network driven space-frequency domain pedestrian gait feature extraction method.

[0067] A computer device, comprising a processor and a memory for storing a program executable by the processor, and when the processor executes the program stored in the memory, the pulse neural network driven space-frequency domain pedestrian gait feature extraction method is realized.

[0068] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0069] 1. The method of the present application significantly improves the performance of the model in extracting gait features, thereby improving the accuracy and robustness of gait recognition: first, through the spatial-frequency feature fusion module, the present application can efficiently separate and fuse high and low frequency features, strengthen the main structure features and edge texture details, thereby improving the richness and accuracy of feature expression; second, the design of the multi-stage time domain global feature interaction enhancement module enables the model to gradually extract and enhance the global features of the spatial and time domains, further improving the model's ability; in addition, the discriminative feature aggregation module fuses the output feature map of the backbone network with the feature map enhanced by the multi-stage time domain global feature interaction enhancement module, and further enhances the discriminative feature through HPM operation, so that the model can maintain excellent recognition performance in complex scenarios;

[0070] 2. The spatial-frequency feature fusion module of the present application effectively fuses the spatial and frequency domain features in the gait image, improving the accuracy and robustness of gait recognition; the spatial-frequency feature fusion module uses a pulse neural network screening mechanism to separate and fuse high and low frequency features, strengthen the edge texture details, and enhance the detail sensitivity, thereby improving the expression ability of gait features;

[0071] 3. The multi-stage time domain global feature interaction enhancement module of the present application realizes the interactive enhancement of spatial domain global features and time domain global features through cyclic progressive calculation, further improving the extraction ability of gait features; the multi-stage time domain global feature interaction enhancement module uses an interactive attention mechanism module to perform time sequence weighted fusion on the feature map, obtaining more robust gait features;

[0072] 4. The time domain global feature fusion module of the present application accurately captures the time dynamic information in the gait sequence through a time sequence attention mechanism, and fuses the global spatial domain features with the global time domain features, enhancing the model's sensitivity to gait changes, so that gait recognition can maintain high performance at different speeds and rhythms;

[0073] 5. The discriminative feature aggregation module of the present application fuses the output feature map of the backbone network with the feature map enhanced by the multi-stage time domain global feature interaction enhancement module, obtaining more rich and accurate gait features; the discriminative feature aggregation module uses fixed block operation and 2D convolution group for feature fusion, improving the accuracy and efficiency of gait recognition. BRIEF DESCRIPTION OF DRAWINGS

[0074] Figure 1 is the structure block diagram of the spatial-frequency domain pedestrian gait feature extraction model of the present application;

[0075] Figure 2 is the structure block diagram of the spatial-frequency feature fusion module of the present application;

[0076] Figure 3 is a structural block diagram of the multi-stage time domain global feature interaction enhancement module of the present application;

[0077] Figure 4 is a structural block diagram of the time domain global feature fusion module of the present application;

[0078] Figure 5 is a structural block diagram of the discriminative feature aggregation module of the present application. DETAILED DESCRIPTION

[0079] The present application will be further described in detail below in combination with the accompanying drawings and specific embodiments.

[0080] Embodiment One

[0081] The embodiment is a method for extracting spatial-frequency domain pedestrian gait features based on a pulse neural network driver, which uses a spatial-frequency domain pedestrian gait feature extraction model to extract features from gait images.

[0082] As shown in Figure 1 , the spatial-frequency domain pedestrian gait feature extraction model includes a backbone network, a multi-stage time domain global feature interaction enhancement module, and a discriminative feature aggregation module.

[0083] The backbone network includes N stages connected in sequence; before the first stage, it also includes a stem module; after the last stage, it also includes a time domain pooling module (TP). Each stage includes an spatial-frequency feature fusion module for fusing spatial domain features and frequency domain features. In this embodiment, the backbone network includes four stages; each stage is composed of a 2D residual module and an spatial-frequency feature fusion module in series; the channel number of the feature map output by each stage is C, 2C, 4C, 8C in sequence, and the size of the feature map is (H, W), (1 / 2H, 1 / 2W), (1 / 4H, 1 / 4W), (1 / 4H, 1 / 4W) respectively. In actual application, the number of stages of the backbone network can also be two, three, five, six, or even more.

[0084] The spatial-frequency feature fusion module is composed of an spatial-frequency dual-domain decoupling processing branch, a high-low frequency feature extraction and dynamic fusion mechanism, a cross-domain channel attention interaction based on a pulse neural network, and a feedforward neural network. The spatial-frequency feature fusion module mainly separates and fuses high and low frequency features through the weights obtained by the dual-branch structure and the pulse neural network screening mechanism, the low frequency component retains the main structure features, the high frequency component enhances the edge texture details, and the channel attention mechanism path enhances the detail sensitivity.

[0085] As shown in Figure 2As shown, each space-frequency feature fusion module includes a spatial domain extraction branch and a frequency domain extraction branch; each space-frequency feature fusion module divides the feature map input to the space-frequency feature fusion module into two feature maps I1, I2: I1, I2 = split(I); where split is the splitting function.

[0086] The spatial domain extraction branch extracts spatial features from feature map I1 using two convolutional kernels; the frequency domain extraction branch extracts high- and low-frequency components from feature map I2 after Fourier transform using two convolutional kernels, and then inputs them into the corresponding convolutional networks after inverse transform to obtain the low-frequency features U in the spatial domain. 0L and high-frequency features U 0H High-frequency feature U 0H The high-frequency component weights W are obtained through the spiking neural network LIF:

[0087] W=LIF(Conv2D(IFFT([Conv2D ×2 (FFT(I2))] HF ))) ;

[0088] Wherein, LIF() represents a spiking neural network; [ ] HF Represents high-frequency components;

[0089] Low-frequency characteristics U 0L After adding the spatial features extracted by the spatial extraction branch, the dual-branch low-frequency fusion feature U1 is obtained:

[0090] U1=Conv2D ×2 (I1)+ Conv2D(IFFT([Conv2D ×2 (FFT(I2))] LF )) ;

[0091] Where Conv2D() is a 2D convolution; Conv2D ×2 ( ) represents a 2D convolution with two kernels; FFT() represents the Fourier transform; IFFT() represents the inverse Fourier transform; [ ] LF This represents the low-frequency component.

[0092] The fused low-frequency feature U1 is obtained by multiplying the low-frequency component weight (1-W) by the dual-branch low-frequency fusion feature U1. 1L The low-frequency fusion feature U1 of the dual-branch system is multiplied by the high-frequency component weight W and then combined with the high-frequency feature U. 0H Add them together to obtain the fused high-frequency feature U. 1H The fused low-frequency feature U 1L and high-frequency features U 1H They are respectively:

[0093] U1L = U1 x (1 - W) ;

[0094] U 1H = U1 x W + Conv2D(IFFT([Conv2D ×2 (FFT(I2)) HF )) ) ;

[0095] The fused low-frequency feature U 1L and the high-frequency feature U 1H are further added and fused to obtain a mixed output feature map U2:

[0096] U2 = Conv2D(BN(Conv2D(U 1L ))+ BN(Conv2D(U 1H ))) ;

[0097] Wherein, BN( ) is a batch normalization operation.

[0098] The feature map I is subjected to a channel attention mechanism to obtain channel weights, and then multiplied by the weights obtained by the pulse neural network LIF to obtain updated channel attention weights W C :

[0099] W C = Tanh(attention(I) x LIF(attention(I))) ;

[0100] Wherein, attention( ) is an attention mechanism; Tanh( ) is an activation function.

[0101] The channel attention weights W C are multiplied by the mixed output feature map U2 to obtain an attention mechanism-based feature map U C :

[0102] U C = U2 x W C ;

[0103] After the normalization operation, the updated feature is obtained by inputting the FFN feedforward neural network module, and finally added to the feature map I through the residual link module, so that the feature map I provides cross-layer feature supplement to avoid information loss, and the final output feature map Si of the space-frequency feature fusion module is obtained:

[0104] Si = FFN(Norm(U C ))+ I;

[0105] Wherein, FFN ( ) is a feedforward neural network module; Norm( ) is a normalization operation.

[0106] The multi-stage time-domain global feature interaction enhancement module extracts and enhances the spatial domain global feature and the time-domain global feature from the output feature maps Si of each stage of the backbone network, i = 1, 2,..., N, to obtain output feature maps S T_total .

[0107] As shown in Figure 3 , the multi-stage time-domain global feature interaction enhancement module includes a time-domain global feature fusion module, a spatial domain global feature enhancement network branch, and a time-domain global feature enhancement network branch.

[0108] First, S s 0 = 0; S ST 0 = 0; S T 0 = 0; the output feature maps Si of each stage of the backbone network are calculated by spatial domain pooling and time-domain pooling to obtain the spatial domain global feature S si and the time-domain global feature S Ti :

[0109] ;

[0110] ;

[0111] The spatial domain global feature S si and the time-domain global feature S Ti are jointly processed by the time-domain global feature fusion module to obtain the space-time domain global feature S STi .

[0112] The feature dimension of the space-time domain global feature S STi is consistent with the feature dimension of the output feature map Si of the corresponding stage; the spatial domain global feature S si of each stage is merged with the space-time domain global feature S STi in the time dimension to form merged features (S si , S STi );

[0113] The merged features (S si , S STi ) of each stage are input into the spatial domain global feature enhancement network branch; the spatial domain global feature enhancement network branch is composed of N stage spatial domain enhancement convolution groups; the merged features (S si , S STi ) of the current stage are combined with the output feature maps (S s i-1 , S ST i-1) are added and input to the current stage spatial domain enhancement convolution group for feature extraction to obtain the output feature map (S s i , ST i :

[0114] (S s i , ST i ) = Conv2D ×2 (concat(S s i-1 , ST i-1 ) + concat(S si , STi )

[0115] The time domain global feature maps S Ti of each stage are input to a time domain global feature enhancement network branch respectively; the time domain global feature enhancement network branch is composed of N stage time domain enhancement convolution groups; the time domain global feature map S Ti of the current stage is added to the output feature map S T i-1 of the time domain enhancement convolution group of the previous stage, and then input to the current stage time domain enhancement convolution group for feature extraction to obtain the output feature map S T i of the current stage time domain enhancement convolution group:

[0116] S T i = Conv2D ×2 (S T i-1 + S Ti )

[0117] The output feature map (S s N , S ST N ) of the last stage spatial domain enhancement convolution group is divided into features S s N , S ST N ; the feature S s N is taken as the spatial domain global feature S s(N+1) , and the output feature map S T N of the last stage time domain enhancement convolution group is taken as the time domain global feature S T(N+1) , which are jointly input into a time domain global feature fusion module to obtain the output space-time domain global feature S ST(N+1) , and the feature SST N The feature map S is obtained by fusion. ST_temp :

[0118] S ST_temp =Conv2D ×2 (f st (S s N ,S T N )+S ST N );

[0119] Feature map S ST_temp With feature map S T N The weights obtained through the activation function are weighted and fused to obtain the final output feature map S of the multi-stage temporal global feature interaction enhancement module. T_total :

[0120] S T_total =sigmoid(S ST_temp ) ×sigmoid(S T N ) ×S ST_temp +S T N .

[0121] like Figure 4 As shown, the spatial global feature S si and temporal global features S Ti The common inputs are into the temporal global feature fusion module: spatial global feature S si After feature extraction via two layers of 3D convolution, a temporal attention score is obtained through a temporal attention mechanism module consisting of two layers of 3D convolution and activated in the temporal dimension. This temporal attention score is then compared with the spatial global feature S. si The fused features S are obtained by performing dot product weighting. si_temp Temporal global features S Ti After max pooling and average pooling, it is combined with the feature S si_temp The features are added together, and the spatiotemporal global features S output by the temporal global feature fusion module are obtained through two layers of 3D convolution. STi :

[0122] S STi =f st (S si ,S Ti )

[0123] =Conv3D ×2 (Softmax(Conv3D ×2 (R(Ssi ))) x Conv3D ×2 (R(S si ) + maxpool(S Ti ) + meanpool(S Ti ));

[0124] where f st () denotes the time domain global feature fusion module; R() is the reshape operation; Conv3D ×2 () is two layers of 3D convolution; Softmax() is an activation function; maxpool() and meanpool() are spatial maximum pooling and average pooling operations, respectively; here i = 1, 2, …, N, N + 1.

[0125] The output feature map SN of the last stage of the backbone network is subjected to a time domain pooling operation (TP) to obtain a feature map S m .

[0126] As shown in Figure 5 , the discriminative feature aggregation module aggregates the output feature map S m of the backbone network and the output feature map S T_total of the multi-stage time domain global feature interaction enhancement module, and inputs the feature fusion result obtained through convolution into the HPM operation to obtain the output S out of the discriminative feature aggregation module as the final output of the spatial and spectral domain pedestrian gait feature extraction model:

[0127] S out = HPM(Conv2D ×2 (Cancat(S T_total , S m )));

[0128] where HPM() is a block operation; Conv2D ×2 () is a 2D convolution with two convolution kernels; and Cancat() is an aggregation operation.

[0129] The spatial and spectral domain pedestrian gait feature extraction model is trained as follows: the model data input is a gait silhouette sequence with a size of (B, T, C, H, W), where B is the batch size (this parameter is commonly used in deep learning models), T is the number of time frames, C is the number of channels, H is the height of the feature map, and W is the width of the feature map. The data is subjected to the stem module and the backbone network, respectively; meanwhile, the output feature map of each stage is input into the multi-stage time domain global feature interaction enhancement module to obtain an enhanced feature map. The output feature maps of the backbone network and the multi-stage time domain global feature interaction enhancement module are input into the discriminative feature aggregation module to obtain the final gait feature vector. The L tripletTriplet loss and L ce The contrast loss is supervised training, and a final space-frequency domain pedestrian gait feature extraction model is obtained.

[0130] The method of the present application significantly improves the performance of the model in extracting gait features, thereby improving the accuracy and robustness of gait recognition: first, through the space-frequency feature fusion module, the present application can efficiently separate and fuse high and low frequency features, strengthen the main structure features and edge texture details, thereby improving the richness and accuracy of feature expression; second, the design of the multi-stage time domain global feature interaction enhancement module enables the model to gradually extract and enhance the global features of the space domain and the time domain, further improving the model capability; in addition, the discriminative feature aggregation module fuses the output feature map of the backbone network with the feature map enhanced by the multi-stage time domain global feature interaction enhancement module, and performs HPM operation, further enhancing the discriminative of the features, so that the model can also maintain excellent recognition performance in complex scenes.

[0131] Embodiment two

[0132] The embodiment is a readable storage medium, wherein the readable storage medium stores a computer program, and the computer program, when executed by a processor, causes the processor to execute the space-frequency domain pedestrian gait feature extraction method based on the pulse neural network driving described in embodiment one.

[0133] Embodiment three

[0134] The embodiment is a computer device, comprising a processor and a memory for storing a program executable by the processor, and when the processor executes the program stored in the memory, the space-frequency domain pedestrian gait feature extraction method based on the pulse neural network driving described in embodiment one is realized.

[0135] The above embodiments are the preferred embodiments of the present application, but the embodiments of the present application are not limited by the above embodiments, and any changes, modifications, substitutions, combinations and simplifications made without departing from the spirit and principles of the present application should be equivalent replacement methods, and all are included in the protection scope of the present application.

Claims

1. A method for extracting pedestrian gait features in the spatial frequency domain based on a spiking neural network, characterized in that: A spatial frequency domain pedestrian gait feature extraction model is used to extract gait features from gait images; The spatial-frequency domain pedestrian gait feature extraction model includes: a backbone network, a multi-stage temporal global feature interaction enhancement module, and a discriminative feature aggregation module. The backbone network comprises N sequentially connected stages, each stage including a spatial-frequency feature fusion module for fusing spatial and frequency domain features. The multi-stage temporal global feature interaction enhancement module extracts and enhances spatial and temporal global features from the output feature maps Si, i=1,2…N of each stage of the backbone network, to obtain the output feature map S. T_total The discriminative feature aggregation module combines the backbone network output with the feature map S output by the multi-stage temporal global feature interaction enhancement module. T_total Aggregation is performed to obtain the gait features finally output by the spatial frequency domain pedestrian gait feature extraction model; Each spatial-frequency feature fusion module includes a spatial domain extraction branch and a frequency domain extraction branch. Each module segments the feature map input to the spatial-frequency feature fusion module into two feature maps I1 and I2. The spatial domain extraction branch extracts spatial features from feature map I1 using two convolutional kernels. The frequency domain extraction branch performs a Fourier transform on feature map I2, then separates the high- and low-frequency components using two convolutional kernels. These components are then inversely transformed and input into the corresponding convolutional networks to obtain the low-frequency feature U in the spatial domain. 0L and high-frequency features U 0H High-frequency feature U 0H The high-frequency component weights W and low-frequency features U are obtained through a pulsating neural network LIF. 0L After adding the spatial features extracted by the spatial extraction branch, the dual-branch low-frequency fusion feature U1 is obtained; the dual-branch low-frequency fusion feature U1 is multiplied by the low-frequency component weight (1-W) to obtain the fused low-frequency feature U. 1L The low-frequency fusion feature U1 of the dual-branch system is multiplied by the high-frequency component weight W and then combined with the high-frequency feature U. 0H Add them together to obtain the fused high-frequency feature U. 1H Then, the fused low-frequency features U 1L and high-frequency features U 1H Further addition and fusion are performed to obtain the hybrid output feature map U2; The feature map I is processed through a channel attention mechanism to obtain channel weights, which are then multiplied with the weights obtained from the spiking neural network LIF to obtain the updated channel attention weights W. C The channel attention weight W C Multiplying the mixed output feature map U2 with the attention-based feature map U2 yields the attention-based feature map U. C After normalization, the updated features are input into the FFN feedforward neural network module. Finally, the features are added to the feature map I through the residual linking module to obtain the final output feature map Si of the space-frequency feature fusion module. The multi-stage temporal global feature interaction enhancement module includes a temporal global feature fusion module, a spatial global feature enhancement network branch, and a temporal global feature enhancement network branch; the spatial global feature enhancement network branch consists of N stages of spatial enhancement convolutional groups; the temporal global feature enhancement network branch consists of N stages of temporal enhancement convolutional groups. The calculation method in the multi-stage temporal global feature interaction enhancement module is as follows: Initialize the output feature map S of the spatial enhancement convolution group s 0 =0、S ST 0 =0; Output feature map S of the temporally enhanced convolutional group T 0 =0; Spatial global features S si and temporal global features S Ti They are respectively: ； ； Where T is the number of time frames, and C, H, and W are the number of channels, height, and width of feature map Si, respectively; Output feature maps of spatial enhancement convolutional groups at each stage (S s i ,S ST i )for: (S s i ,S ST i )=Conv2D ×2 (concat(S s i-1 ,S ST i-1 )+concat(S si ,S STi ) ; Output feature maps S of temporal augmentation convolutional groups at each stage T i for: S T i =Conv2D ×2 (S T i-1 +S Ti ); Feature map S ST_temp for: S ST_temp =Conv2D ×2 (f st (S s N ,S T N )+S ST N ); The final output feature map S of the multi-stage temporal global feature interaction enhancement module T_total for: S T_total =sigmoid(S ST_temp ) ×sigmoid(S T N ) ×S ST_temp +S T N 。 2. The method for extracting pedestrian gait features in the spatial frequency domain based on a spiking neural network according to claim 1, characterized in that: The feature maps I1 and I2 are: I1,I2=split(I); where split is the segmentation function; The dual-branch low-frequency fusion feature U1 is: U1=Conv2D ×2 (I1)+ Conv2D(IFFT([Conv2D ×2 (FFT(I2))] LF )) ; Where Conv2D() is a 2D convolution; Conv2D ×2 ( ) represents a 2D convolution with two kernels; FFT() represents the Fourier transform; IFFT() represents the inverse Fourier transform; [ ] LF Represents low-frequency components; The weight W of the high-frequency components is: W=LIF(Conv2D(IFFT([Conv2D ×2 (FFT(I2))] HF ))) ; Wherein, LIF() represents a spiking neural network; [ ] HF Represents high-frequency components; The fused low-frequency feature U 1L and high-frequency features U 1H They are respectively: U 1L =U1×(1-W); U 1H =U1×W+Conv2D(IFFT([Conv2D ×2 (FFT(I2))] HF )) ; The mixed output feature map U2 is: U2=Conv2D(BN(Conv2D(U 1L ))+BN(Conv2D(U 1H ))) ; Where BN() is the batch normalization operation; Channel attention weight W C for: W C = Tanh(attention(I) ×LIF(attention(I))) ; Where attention() is the attention mechanism; Tanh() is the activation function; Feature map U based on attention mechanism C for: IN C =U2×W C ; The final output feature map Si of the space-frequency feature fusion module is: Si = FFN(Norm(U C ))+I; Here, FFN() is the feedforward neural network module; Norm() is the normalization operation.

3. The method for extracting pedestrian gait features in the spatial frequency domain based on a spiking neural network according to claim 1, characterized in that: The multi-stage temporal global feature interaction enhancement module includes a temporal global feature fusion module, a spatial global feature enhancement network branch, and a temporal global feature enhancement network branch. The output feature maps Si of each stage of the backbone network are used to calculate the spatial global features S through spatial pooling and temporal pooling, respectively. si and temporal global features S Ti Spatial global features S si and temporal global features S Ti The common inputs are fed into the temporal global feature fusion module to obtain the spatiotemporal global feature S. STi ; The spatial global features S of each stage si With the global features in the space-time domain S STi Merging into merged features (S) along the time dimension si , S STi ); Merging characteristics at each stage (S) si , S STi The inputs are respectively fed into the spatial global feature enhancement network branch; The spatial global feature enhancement network branch consists of N stages of spatial enhancement convolutional groups; the merged features (S) of the current stage si , S STi The output feature map (S) of the previous stage spatial enhancement convolution group s i-1 ,S ST i-1 The sums are then fed into the current stage's spatial enhancement convolutional group for feature extraction to obtain the output feature map (S) of the current stage's spatial enhancement convolutional group. s i ,S ST i ); Temporal global feature map S at each stage Ti The inputs are respectively fed into the temporal global feature enhancement network branch; The temporal global feature enhancement network branch consists of N stages of temporal enhancement convolutional groups; the temporal global feature map S of the current stage. Ti Compared with the output feature map S of the previous stage temporal enhancement convolution group T i-1 The sums are then fed into the current stage's temporal augmentation convolutional group for feature extraction, yielding the output feature map S of the current stage's temporal augmentation convolutional group. T i ; The output feature map (S) of the final stage spatial enhancement convolution group s N ,S ST N Segmented into features S s N ,S ST N ; With feature S s N As a global feature in the spatial domain S s(N+1) The output feature map S of the final stage temporal enhancement convolution group T N As a time-domain global feature S T(N+1) The common input temporal global feature fusion module obtains the output spatiotemporal global feature S. ST(N+1) , with feature S ST N The feature map S is obtained by fusion. ST_temp Feature map S ST_temp With feature map S T N The weights obtained through the activation function are weighted and fused to obtain the final output feature map S of the multi-stage temporal global feature interaction enhancement module. T_total .

4. The method for extracting pedestrian gait features in the spatial frequency domain based on a spiking neural network according to claim 3, characterized in that: Spatial global features S si and temporal global features S Ti The common inputs are into the temporal global feature fusion module: spatial global feature S si After feature extraction via two layers of 3D convolution, a temporal attention score is obtained through a temporal attention mechanism module consisting of two layers of 3D convolution and activated in the temporal dimension. This temporal attention score is then compared with the spatial global feature S. si The fused features S are obtained by performing dot product weighting. si_temp Temporal global features S Ti After max pooling and average pooling, it is combined with the feature S si_temp The features are added together, and the spatiotemporal global features S output by the temporal global feature fusion module are obtained through two layers of 3D convolution. STi .

5. The method for extracting pedestrian gait features in the spatial frequency domain based on a spiking neural network according to claim 4, characterized in that: The spatiotemporal global feature S output by the temporal global feature fusion module STi for: S STi =f st (S si ,S Ti ) =Conv3D ×2 (Softmax(Conv3D ×2 (R(S si ))) ×Conv3D ×2 (R(S si ))+maxpool(S Ti )+meanpool(S Ti )); Among them, f st ( ) represents the temporal global feature fusion module; R() is the reshape operation; Conv3D ×2 ( ) represents two layers of 3D convolution; Softmax() is the activation function; maxpool() and meanpool() are the spatial max pooling and average pooling operations, respectively.

6. The method for extracting pedestrian gait features in the spatial frequency domain based on a spiking neural network according to claim 1, characterized in that: The feature map S is obtained by performing temporal pooling on the output feature map SN of the final stage of the backbone network. m In the feature aggregation module, the feature map S is... m The feature map S is output by the multi-stage temporal global feature interaction enhancement module. T_total After aggregation and feature fusion via convolution, the results are input into the HPM operation to obtain the output S of the discriminative feature aggregation module. out : S out =HPM(Conv2D ×2 (Cancat(S T_total ,S m ))); HPM() is the block operation; Conv2D ×2 ( ) represents a 2D convolution with two kernels; Cancat() represents an aggregation operation.

7. A readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the spatial frequency domain pedestrian gait feature extraction method based on a spiking neural network as described in any one of claims 1-6.

8. A computer device comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the spatial frequency domain pedestrian gait feature extraction method based on spiking neural network driven by any one of claims 1-6.

Images