A Gait Recognition Method Based on Event Cameras

Abstract

This invention discloses a gait recognition method based on an event camera. The method first segments a complete asynchronous pulse signal generated by a neural sensor in the time domain according to different scales, and then passes these segments through a time frame generation module and a spatial frame generation module to obtain corresponding time frames and spatial frames. The time frames are processed by extracting temporal features to obtain features containing temporal information, and then fused using a 3DCNN. The spatial frames are processed by extracting spatial features to obtain features containing spatial information. Specifically, the time and spatial frames (one channel each) are concatenated and fed into a ViT network to obtain the spatiotemporal features of the image. Finally, the features from the three parts are concatenated to obtain the total pedestrian gait features of the entire video sequence. This invention solves the problem that traditional gait recognition algorithms based on RGB video have stringent requirements for environmental and camera conditions, and can be widely applied in video surveillance, security inspection, criminal investigation, and other fields.

Description

Technical Field

[0001] This invention belongs to the field of video target recognition technology, and relates to neural vision sensors and biometric recognition algorithms in videos, specifically a gait recognition method based on an event camera. Background Technology

[0002] Gait recognition technology is a relatively new biometric authentication technology that identifies individuals by their walking patterns. Compared to other biometric technologies, gait recognition offers advantages such as non-contact, long-range operation and resistance to spoofing. In the field of intelligent video surveillance, it has a greater advantage than image recognition.

[0003] Neural vision sensors, also known as event cameras, are a new type of visual perception system that simulates the structure and function of the human retina. They have advantages such as high temporal resolution, high dynamic range, low data redundancy, and low power consumption. In recent years, they have shown great promise in fields such as autonomous driving, drone visual navigation, industrial inspection, and visual monitoring (especially in conditions involving high-speed motion and extreme lighting).

[0004] For gait recognition technology, deep learning-based methods can leverage the strengths of traditional methods and overcome the problem of intra-class variations being greater than inter-class variations due to factors such as cross-viewpoint differences, clothing changes, and walking speed, making them relatively easy to implement. Hardware improvements have also addressed the issues of excessively large parameters and slow training speeds in deep learning network models. However, gait recognition algorithms are highly demanding in terms of environmental requirements, necessitating uniform target movement, suitable ambient lighting, and short-term camera capture. This limits their applicability to certain special environments, such as surveillance or high-speed moving targets. Current algorithms based on traditional RGB images are struggling to overcome these limitations, thus requiring a new image acquisition method and improvements to existing algorithms.

[0005] Therefore, combining event cameras with gait recognition technology is a current necessity and an important step in the future development of event camera-based computer vision algorithms. Gait recognition algorithms based on event cameras can fully leverage the advantages of event cameras, enabling recognition tasks to be performed in environments with harsh lighting conditions, variable target movement speeds, and high camera power consumption requirements, while ensuring high recognition accuracy. Summary of the Invention

[0006] The problem this invention aims to solve is that traditional gait recognition algorithms based on RGB cameras have high requirements for camera and environmental conditions, making them unsuitable for harsh environments and resulting in low recognition accuracy.

[0007] To address the aforementioned problems, this invention provides a gait recognition method based on an event camera. This method converts the event stream generated by the event camera into a frame-like format, enabling neural networks to process the data; it extracts features from both temporal and spatial perspectives using a multi-scale and multi-branch approach, ensuring the completeness and comprehensiveness of the features; and it utilizes 3D convolution combined with cross-entropy and triplet loss functions to make feature comparisons more accurate. The method includes the following steps:

[0008] 1) The original event stream is uniformly divided into blocks of different scales along the time dimension, and the event packets of different scales obtained from the block division are input into the time frame generation module to obtain multiple time frames with different numbers of channels; at the same time, the original event stream is input into the spatial frame generation module to obtain spatial frames.

[0009] 2) Then, the time frame and spatial frame are input into the time feature extraction module and the spatial feature extraction module respectively to obtain time features and spatial features. Furthermore, the single-channel time frame and spatial frame are simultaneously input into the spatiotemporal feature extraction module to obtain spatiotemporal features.

[0010] 3) For the extracted temporal features, feature fusion is performed using 3D convolution operations to obtain the total temporal features; then the total temporal features are concatenated with the spatial features and spatiotemporal features to obtain the total pedestrian gait features; finally, gait feature matching is performed, and the pedestrian sequence number to which the target video belongs is found using Euclidean similarity.

[0011] Furthermore, in step 1), the original event stream is divided into 8 equal parts, 4 equal parts, 2 equal parts, and no division in the time domain according to different scales, and a time frame is generated using the time frame generation module. At the same time, the original event stream is generated into a spatial frame using the spatial frame generation module. Specifically:

[0012] 1.1) Construct 4 branches. These 4 branches divide the original event stream into 8 equal parts, 4 equal parts, 2 equal parts, and no division in the time domain according to different scales to obtain event packets of different scales. At the end of the event stream, some redundant points are removed to ensure that the event stream is completely divided.

[0013] 1.2) On each branch, each event packet is sequentially processed through the time frame generation module to obtain a time frame. Each event packet generates a single-channel time frame, and the multiple single-channel time frames generated by multiple event packets on a branch are merged into a multi-channel time frame. Therefore, after these four branches are processed through the time frame generation module, 8-channel, 4-channel, 2-channel, and single-channel time frames are obtained sequentially.

[0014] 1.3) Simultaneously, the original event stream is input into the spatial frame generation module, and a single-channel spatial frame is generated using the frequency cumulative image representation method;

[0015] Furthermore, in step 2), feature extraction is performed on the time frame and the spatial frame to obtain temporal features, spatial features, and spatiotemporal features, specifically:

[0016] 2.1) On each branch, the time frame is input into the time feature extraction module to obtain time features. The time features obtained from multiple branches are of the same size.

[0017] 2.2) Input the single-channel spatial frame into the spatial feature extraction module to obtain spatial features;

[0018] 2.3) The single-channel time frame and spatial frame obtained in 1.2) and 1.3) are simultaneously input into the spatiotemporal feature extraction module to obtain spatiotemporal features. The processing procedure of the spatiotemporal feature extraction module is as follows: First, the single-channel time frame and spatial frame are simultaneously input into two 3×3 convolutional layers for feature extraction. Then, the extracted feature map is divided into blocks to obtain 16 feature blocks with the same length and width. Finally, these feature blocks are simultaneously fed into ViT (Vision of Transformer) to obtain spatiotemporal features.

[0019] Furthermore, in step 3), gait recognition is performed based on the total gait features of pedestrians, specifically as follows:

[0020] 3.1) The four time features obtained in 2.1) are concatenated along the channel dimension;

[0021] 3.2) Input the spliced ​​features into a 3D convolutional network consisting of two layers of 3D convolutions, and use 3D convolution to perform feature fusion to obtain the total temporal features;

[0022] 3.3) The total temporal features are concatenated with the spatial features obtained in 2.2) and the spatiotemporal features obtained in 2.3) along the channel dimension to obtain the total pedestrian gait features;

[0023] 3.4) The similarity of the total pedestrian gait features obtained from different samples is calculated. In this model, Euclidean distance is used as the measure of feature similarity. The calculation process of feature similarity S is shown in Equation (1):

[0024]

[0025] Among them, f i and f j These are the feature vectors obtained after inputting different samples. For the eigenvector f i with f j The Euclidean distance in the feature space after normalization.

[0026] The features with the closest feature similarity are matched as gait features belonging to the same pedestrian.

[0027] 3.5) The model is trained simultaneously using cross-entropy loss and triplet loss, with the final loss being a weighted sum of the two. The final loss L com Represented as:

[0028] L com =αL cse +βL tri (2)

[0029] Where α = 0.5, β = 0.5.

[0030] Cross-entropy loss L cse Represented as:

[0031]

[0032] Where N represents the total number of input samples during training, x is the total pedestrian gait feature output by the model, p(·) is the probability value of the current feature belonging to the target label, and q(·) is the probability value of the current feature not belonging to the target label, i.e., q(x) = 1 - P(x).

[0033] Triple loss L tri It can be represented as:

[0034]

[0035] Where N represents the total number of input samples during training. Indicates the current sample, Indicates and Samples from the same pedestrian tag, Indicates and For samples with different labels, f(·) represents the feature extraction operation corresponding to the model, α is the boundary of the triplet loss, set to 0.3, and the operation [γ]... + It equals max(γ,0).

[0036] This invention provides a gait recognition method based on an event camera. The method first segments an asynchronous pulse signal generated by a neural visual sensor into different scales in the time domain, and then passes these segments through a time frame generation module and a spatial frame generation module to obtain corresponding time frames and spatial frames. The time frames are processed by a time feature extraction module to obtain features containing time information, and then fused using a 3DCNN. The spatial frames are processed by a spatial feature extraction module to obtain features containing spatial information. The time and spatial frames from a single channel are then concatenated and input into a ViT (Vision-Based Image Processing) module to obtain the spatiotemporal features of the image. Finally, the time features, spatial features, and spatiotemporal features are concatenated to obtain the total pedestrian gait features of the entire video sequence. This invention solves the problem of traditional gait recognition algorithms being highly demanding in terms of environmental and camera conditions. This invention has low computational complexity, high recognition efficiency, fast recognition speed, accurate recognition results, and good algorithm robustness. Attached Figure Description

[0037] Figure 1 This is a flowchart of the gait recognition method based on an event camera according to the present invention.

[0038] Figure 2 This is a model diagram of the gait recognition method based on an event camera according to the present invention. Detailed Implementation

[0039] This invention provides a gait recognition method based on an event camera. The method uses an event stream obtained from pedestrian gait captured by an event camera as input. First, an asynchronous pulse signal generated by a neural visual sensor is divided into 8 equal parts, 4 equal parts, 2 equal parts, and no division in the time domain according to different scales. These are then processed by a time frame generation module and a spatial frame generation module to obtain corresponding time frames and spatial frames. The time frames are processed by a time feature extraction module to obtain time features containing time information, and then fused using a 3DCNN to obtain total time features. The spatial frames are processed by a spatial feature extraction module to obtain spatial features containing spatial information. The single-channel time frames and spatial frames are concatenated and input into ViT to obtain the spatiotemporal features of the image. Finally, the time features, spatial features, and spatiotemporal features are concatenated to obtain the total pedestrian gait features of the video sequence. This invention includes the following steps:

[0040] 1) Acquiring test videos. Since there is currently no unified event camera dataset for gait recognition tasks, the CASIA-B dataset was transformed using the Event Camera Simulator (ESIM). ESIM tightly integrates the rendering engine and the event simulator, allowing the event simulator to adaptively acquire frames based on the dynamics of the visual signals.

[0041] First, the original CASIA-B video was increased in frame count from 25 to 60 using a frame interpolation algorithm. Then, the video was cropped. The cropping range was from when the pedestrian's body completely entered the frame until any part of their body left the frame. Each frame of the video was then sequentially input into ESIM for simulation. The generated dataset's directory tree remained consistent with CASIA-B.

[0042] 2) The asynchronous pulse signal generated by the event camera is in quadruple form, which cannot be processed directly. Therefore, a suitable method is needed to process the event stream. Here, we convert the signal from a gait video captured by the event camera into a frame sequence format similar to that captured by a traditional RGB camera. This format preserves both the temporal and spatial information contained in the signal and can be used as input to a neural network for easier processing. Specifically:

[0043] 2.1) Generation of Time Frames. Four branches are constructed. These four branches sequentially divide the original event stream in the time domain according to different scales: 8 equal parts, 4 equal parts, 2 equal parts, and no division, obtaining event packets of different scales. Some redundant points can be removed at the end of the event stream to ensure that the event stream is completely divided. This process can be represented as:

[0044] P n =C n (ε) (5)

[0045] Where ε represents the original event stream, C n This indicates dividing the original event stream into n equal parts, where n = 1, 2, 4, 8, P n This represents the set of n event packets obtained by dividing the original event stream into n equal parts on the corresponding branch.

[0046] The definition of a related event is: two consecutive events occurring at a pixel, if the time difference between their occurrences is no greater than [missing value]. These are considered related events. The time information in the event packet can be extracted by recording the time difference of related events at each pixel. The time information T of pixel (x, y) is... x,y Calculated from the time of occurrence of the relevant events at that pixel. Event packet p nm (n = 1, 2, 4, 8, m = 1, 2, ..., n) represents the m-th event packet obtained when the event packet is divided into n equal parts. nm The time information T of pixel (x,y) nm The formula for calculating (x,y) is:

[0047]

[0048] Among them, t i It is the time when the i-th event occurs at pixel (x,y), and N represents the event packet P.nm The total number of events occurring at pixel (x, y). Set it to 2.

[0049] By recording the time difference between two consecutive events occurring at a pixel, T nm (x, y) can aggregate information from all related events at that position within the event packet. For two consecutive events at the same position, the smaller the time difference, the stronger the correlation; the larger the time difference, the weaker the correlation. Therefore, a logarithmic function is chosen. If the time difference between two consecutive events is greater than... Then it is assumed that the two events are unrelated, or that one of the events is noise.

[0050] For event packet P nm Construct a single-channel time frame F nm The value F at its pixel (x,y) nm (x,y) is:

[0051] F nm (x,y)=T nm (x,y)+0.5(T nm (x-1,y)+T nm (x+1,y)+T nm (x,y-1)+T nm (x,y+1))+0.25(T nm (x-1, y-1)+T nm (x+1,y-1)+T nm (x-1,y+1)+T nm (x+1,y+1)) (7)

[0052] Single-channel time frame F nm The value of each pixel is a weighted sum of the temporal information of that location and its eight surrounding locations, with the four adjacent locations having a larger weight and the four diagonally opposite locations having a smaller weight.

[0053] In the branch that divides the original event stream into n equal parts, n channel time frames are generated. for:

[0054]

[0055] Among them, F ni This represents the i-th single-channel time frame generated after dividing the original event stream into n equal parts, and ⊕ represents performing a cat operation on the single-channel time frame in the channel dimension.

[0056] After passing through the time frame generation module, each of the four branches generates one time frame, with channel numbers of 1, 2, 4, and 8 respectively, represented as follows: n = 1, 2, 4, 8;

[0057] 2.2) Spatial Frame Generation. The original event stream is input into the spatial frame generation module. Using the frequency accumulation image representation method, the events in the event stream are accumulated in the spatial domain to generate a single-channel spatial frame F. spa The meaning of each pixel within a frame represents the number of times an event occurs at that location. Spatial frame F spa The value at pixel (x, y) represents the total number of events occurring at pixel (x, y) in the event stream. The specific formula is as follows:

[0058] F spa (x,y)=num(x,y) (9)

[0059] Where num(·) represents a counter that records the number of events that occur at pixel (x,y).

[0060] 3) Feature extraction is performed on the temporal and spatial frames to obtain temporal features, spatial features, and spatiotemporal features, specifically:

[0061] 3.1) In the branch that divides the original event stream into n equal parts, the n channel time frames Input the time feature extraction module to obtain time features The temporal feature extraction module uses a ResNet50 network with all fully connected layers removed. The specific process is as follows:

[0062]

[0063] in, R represents an n-channel time frame, and R(·) represents a ResNet50 network with all fully connected layers removed.

[0064] The temporal features of each branch are of the same size. Each temporal feature extraction module is independent and does not share parameters.

[0065] 3.2) Input the spatial frame into the spatial feature extraction module to obtain the spatial features f. spa The spatial feature extraction module uses a ResNet50 network with the last fully connected layer removed and a fully connected layer with a 1024-dimensional output added. The specific process can be represented as follows:

[0066] f spa =F fc (R'(F spa (11)

[0067] Among them, F spa R'(·) represents a spatial frame, R'(·) represents a ResNet50 network with the last fully connected layer removed, and F fc (·) indicates a fully connected layer with a 1024-dimensional output.

[0068] 3.3) Convert single-channel time frames and spatial frame F spa The feature map is concatenated along the channel dimension, and then processed through two 3×3 convolutional layers for feature extraction. The resulting feature map is then segmented into 16 equal-sized feature blocks (80×60 pixels each). These blocks are simultaneously fed into ViT to obtain the spatiotemporal feature f. ts ViT consists of 6 transformer blocks, with the output of the last fully connected layer changed to 1024 dimensions. The specific process can be represented as follows:

[0069]

[0070] in, F represents a single-channel time frame. spa Represents a spatial frame, f 3×3 (·) represents a convolutional layer with a kernel size of 3×3 and a stride of 3×3, D(·) represents a block operation, V(·) represents removing the last fully connected layer of the ViT network, and F' fc (·) indicates a fully connected layer with a 1024-dimensional output, and ⊕ indicates a cat operation on the channel dimension.

[0071] The role of the spatiotemporal feature extraction module is to amplify the differences between classes and solve the problem that when two people walk at the same angle, the frames are quite similar.

[0072] 4) The temporal features obtained from the four branches are fused using 3D convolution, and then passed through a fully connected layer to obtain the total temporal feature f. TEM The process can be represented as:

[0073]

[0074] in, This represents the temporal features generated from the n-channel time frame, where Te1 and Te2 are the feature fusion operations. F″ represents a 3D convolution with a kernel size of 3×3×3 and a stride of 3×3×3. fc (·) indicates a fully connected layer with a 1024-dimensional output, and ⊕ indicates a cat operation on the channel dimension.

[0075] Traditional 2D CNNs do not consider inter-frame motion information in the temporal dimension. Using 3D CNNs can better capture temporal and spatial feature information in videos, resulting in the fused total temporal feature f. TEM Afterwards, with spatial features f spa Spatiotemporal characteristics f ts By concatenating along the channel dimension, the total pedestrian gait feature f is obtained. out This process can be represented as:

[0076]

[0077] Among them, f TEM f represents the overall time characteristic. spa Representing spatial characteristics, f ts ⊕ represents spatiotemporal features, and ⊕ represents multiple one-dimensional features joined together by length.

[0078] 5) Calculate and match the similarity of the obtained total pedestrian gait features. Specifically:

[0079] 5.1) The similarity of the total pedestrian gait features obtained from different samples is calculated. In this model, Euclidean distance is used as the measure of feature similarity. The calculation process of feature similarity S is shown in Equation (15):

[0080]

[0081] Among them, f i and f j These are the feature vectors obtained after inputting different samples. For the eigenvector f i with f j The Euclidean distance in the feature space after normalization.

[0082] The features with the closest feature similarity are matched as gait features belonging to the same pedestrian.

[0083] 5.2) The model is trained simultaneously using cross-entropy loss and triplet loss, with the final loss being a weighted sum of the two. The final loss L com It can be represented as:

[0084] L com =αL cse +βL tri (16)

[0085] Where α = 0.5, β = 0.5.

[0086] Cross-entropy loss L cse definition:

[0087]

[0088] Where N represents the total number of input samples during training, x is the recognition feature output by the model, p(·) is the probability value that the current feature belongs to the target label, and q(·) is the probability value that the current feature does not belong to the target label, i.e., q(x) = 1 - p(x).

[0089] Triple loss L tri definition:

[0090]

[0091] Where N represents the total number of input samples during training. Indicates the current sample, Indicates and Samples from the same pedestrian tag, Indicates and For samples with different labels, f(·) represents the feature extraction operation corresponding to the model, α is the boundary of the triplet loss, set to 0.3, and the operation [γ]... + It equals max(γ,0).

[0092] This invention can be widely used in video surveillance, security inspection, criminal investigation, and other fields. The invention will now be described in detail with reference to the accompanying drawings.

[0093] (1) In an embodiment of the present invention, the input raw event stream is first processed. An asynchronous pulse signal generated by a neural visual sensor is divided into 8 equal parts, 4 equal parts, 2 equal parts, or no division in the time domain according to different scales to obtain multiple event packets.

[0094] (2) Send the obtained event packet into the event frame generation module to obtain the corresponding time frame and spatial frame;

[0095] (2.1) The definition of a related event is: two consecutive events occurring at a pixel, if the time difference between their occurrences is no greater than [missing information]. These are considered related events. In the time frame generation module, the time information in the event packet is first extracted by recording the time difference between related events. For two consecutive events at the same location, the smaller the time difference, the stronger the correlation; the larger the time difference, the weaker the correlation. If the time difference between two consecutive events is greater than... If the two events are unrelated, or one of the events is noise, then the two events are considered to be unrelated. Within each event packet, a time frame is constructed using the time information calculated for each pixel. The value of each pixel within the frame is the weighted sum of the time information of that location and its eight surrounding locations.

[0096] (2.2) Spatial frame generation uses the frequency accumulation image representation method. The number of events in the original event stream is accumulated in the spatial domain to obtain the spatial frame;

[0097] (3) Next, feature extraction is performed on the generated time frame and spatial frame respectively to obtain time features, spatial features and spatiotemporal features containing time information and spatial information;

[0098] (3.1) The generated time frame and spatial frame are sent to the time feature extraction module and the spatial extraction module respectively to obtain time features and spatial features. The time features are then fused using 3DCNN to obtain the total time features.

[0099] (3.2) The time frame and spatial frame of a single channel are concatenated in the channel dimension, passed through 2 convolutional layers, and then the feature map is cut into blocks to obtain multiple feature blocks with the same length and width. These feature blocks are fed into ViT at the same time to obtain spatiotemporal features.

[0100] (3.3) The total temporal features, spatial features, and spatiotemporal features are spliced ​​together in the channel dimension to obtain the total pedestrian gait features;

[0101] (3.4) The similarity of gait recognition features of different samples is calculated using Euclidean similarity;

[0102] (3.5) The model is trained using both cross-entropy loss and triplet loss, and the weighted sum of the two is used as the final loss of the model, so that the final recognition features are more discriminative.

[0103] This method was implemented using the PyCharm compiler and Python programming language on a 12th Gen Intel(R) Core(TM) i7-1260P 2.10GHz and a Win11 64-bit operating system.

[0104] This invention provides a gait recognition method based on event cameras. This method can fully utilize the advantages of event cameras, enabling recognition tasks to be performed in environments with harsh lighting conditions, variable target movement speeds, and high camera power consumption requirements, while ensuring high recognition accuracy. Experiments show that this method can perform pedestrian gait recognition tasks quickly and effectively.

Claims

1. A gait recognition method based on an event camera, characterized in that, For a given set of pedestrian gait videos in a given scene, perform the following operations: 1) The original event stream is uniformly divided into blocks of different scales along the time dimension, and the event packets of different scales obtained from the block division are input into the time frame generation module to obtain multiple time frames with different numbers of channels; at the same time, the original event stream is input into the spatial frame generation module to obtain spatial frames; Time frame generation: Construct 4 branches, which sequentially divide the original event stream in the time domain into 8 equal parts, 4 equal parts, 2 equal parts, and no division according to different scales to obtain event packets of different scales. Redundant points are removed at the end of the event stream to ensure that the event stream is completely divided; Spatial frame generation: The original event stream is input into the spatial frame generation module, and the frequency accumulation image representation method is used to accumulate the events of the event stream in the spatial domain to generate a single-channel spatial frame F. spa The meaning of each pixel within a frame represents the number of times an event occurs at that location; 2) Then, the time frame and spatial frame are input into the time feature extraction module and the spatial feature extraction module respectively to obtain time features and spatial features. The single-channel time frame and spatial frame are input into the spatiotemporal feature extraction module at the same time to obtain spatiotemporal features. 3) For the extracted temporal features, feature fusion is performed using 3D convolution operations to obtain the total temporal features; Then, the total temporal features are concatenated with the spatial and spatiotemporal features to obtain the total pedestrian gait features; finally, gait feature matching is performed, and the pedestrian sequence number to which the target video belongs is found using Euclidean similarity.

2. The gait recognition method based on an event camera according to claim 1, characterized in that, The method for generating the event frame in step 1) is as follows: Step 1.1) Generation of time frames; Construct 4 branches, which sequentially divide the original event stream in the time domain into 8 equal parts, 4 equal parts, 2 equal parts, and no division, respectively, to obtain event packets of different scales. Redundant points are removed at the end of the event stream to ensure that the event stream is completely divided; as shown below: P n =C n (e) (1) Where ε represents the original event stream, C n This indicates dividing the original event stream into n equal parts, where n = 1, 2, 4, 8, P n This represents the set of n event packets obtained by dividing the original event stream into n equal parts on the corresponding branch; The definition of a related event is: two consecutive events occurring at a pixel, if the time difference between their occurrences is no greater than [missing value]. These are considered related events; by recording the time difference of related events at each pixel, the time information in the event packet can be extracted; the time information T of pixel (x,y) x,y Calculated from the time of occurrence of the relevant events at that pixel; event packet p nm This represents the m-th event packet obtained when the packet is divided into n equal parts, where n = 1, 2, 4, 8, m = 1, 2, ..., n; for event packet p nm The time information T of pixel (x,y) nm The formula for calculating (x,y) is: Among them, t i It is the time when the i-th event occurs at pixel (x,y), and N represents the event packet P. nm The total number of events occurring at pixel (x, y). Set to 2; By recording the time difference between two consecutive events occurring at a pixel, T nm (x, y) aggregates information about all relevant events at that position within the event package; a logarithmic function is chosen to be constructed if the time difference between two consecutive events is greater than... The two events are then considered unrelated, or one of the events is noise. For event packet P nm Construct a single-channel time frame F nm The value F at its pixel (x,y) nm (x,y) is: F nm (x,y)=T nm (x,y)+0.5(T nm (x-1,y)+T nm (x+1,y)+T nm (x,y-1)+T nm (x,y+1))+0.25(T nm (x-1,y-1)+T nm (x+1,y-1)+T nm (x-1,y+1)+T nm (x+1,y+1)) (3) Single-channel time frame F nm The value of each pixel is a weighted sum of the temporal information of that position and the surrounding eight positions, with the four adjacent positions having a larger weight and the four diagonally opposite positions having a smaller weight. In the branch that divides the original event stream into n equal parts, n channel time frames are generated. for: Among them, F ni This represents the i-th single-channel time frame generated after dividing the original event stream into n equal parts. This indicates that a single-channel time frame is subjected to a cat operation along the channel dimension; After passing through the time frame generation module, each of the four branches generates one time frame, with channel numbers of 1, 2, 4, and 8 respectively, represented as follows: n = 1, 2, 4, 8; 1.2) Spatial Frame Generation: The original event stream is input into the spatial frame generation module. Using the frequency accumulation image representation method, the events in the event stream are accumulated in the spatial domain to generate a single-channel spatial frame F. spa The meaning of each pixel within a frame represents the number of times an event occurs at that location; spatial frame F spa The value at pixel (x, y) is the total number of events occurring at pixel (x, y) in the event stream; the specific formula is as follows: F spa (x,y)=num(x,y) (5) Where num(·) represents a counter that records the number of events that occur at pixel (x,y).

3. The gait recognition method based on an event camera according to claim 1, characterized in that... In step 2), feature extraction is performed on the generated time frame and spatial frame respectively to obtain features containing both time and spatial information, specifically as follows: 2.1) In the branch that divides the original event stream into n equal parts, the n channel time frames Input the time feature extraction module to obtain time features The temporal feature extraction module uses a ResNet50 network with all fully connected layers removed. The specific process is as follows: in, R represents an n-channel time frame, and R(·) represents a ResNet50 network with all fully connected layers removed. The temporal features of each branch are of the same size; each temporal feature extraction module is independent and does not share parameters. 2.2) Input the spatial frame into the spatial feature extraction module to obtain the spatial features f. spa The spatial feature extraction module uses a ResNet50 network with the last fully connected layer removed and a fully connected layer with a 1024-dimensional output added. The specific process can be represented as follows: f spa =F fc (R'(F spa )) (7) Among them, F spa R'(·) represents a spatial frame, R'(·) represents a ResNet50 network with the last fully connected layer removed, and F fc (·) indicates a fully connected layer with a 1024-dimensional output; 2.3) Convert the single-channel time frame and spatial frame F spa The feature map is concatenated along the channel dimension, and then processed through two 3×3 convolutional layers for feature extraction. The resulting feature map is then segmented into 16 equal-sized feature blocks (80×60 pixels each). These blocks are simultaneously fed into ViT to obtain the spatiotemporal feature f. ts ViT consists of 6 transformer blocks, with the output of the last fully connected layer changed to 1024 dimensions; the specific process can be represented as follows: in, F represents a single-channel time frame. spa Represents a spatial frame, f 3×3 (·) represents a convolutional layer with a kernel size of 3×3 and a stride of 3×3, D(·) represents a block operation, V(·) represents removing the last fully connected layer of the ViT network, and F' fc (·) indicates a fully connected layer with a 1024-dimensional output. This indicates that a cat operation is performed at the channel level; The role of the spatiotemporal feature extraction module is to amplify the differences between classes and solve the problem that when two people walk at the same angle, the frames are quite similar.

4. The gait recognition method based on an event camera according to claim 1, characterized in that... Step 3) involves fusing temporal, spatial, and spatiotemporal features to obtain the total pedestrian gait features, and then performing gait recognition. Specifically: 3.1) The temporal features obtained from the four branches are fused using 3D convolution, and then passed through a fully connected layer to obtain the total temporal feature f. TEM The process can be represented as: in, This represents the temporal features generated from the n-channel time frame, where Te1 and Te2 are the feature fusion operations. F represents a 3D convolution with a kernel size of 3×3×3 and a stride of 3×3×3. fc (·) indicates a fully connected layer with a 1024-dimensional output. This indicates that a cat operation is performed at the channel level; Using 3DCNN can better capture temporal and spatial feature information in videos; thus obtaining the fused total temporal feature f. TEM Afterwards, with spatial features f spa Spatiotemporal characteristics f ts By concatenating along the channel dimension, the total pedestrian gait feature f is obtained. out , means as follows: Among them, f TEM f represents the overall time characteristic. spa Representing spatial characteristics, f ts Indicates spatiotemporal characteristics, This indicates that multiple one-dimensional features are concatenated by length. 3.2) The similarity of the total pedestrian gait features obtained from different samples is calculated, and the Euclidean distance is used as the measure of feature similarity. The calculation process of feature similarity S is shown in Equation (11): Among them, f i and f j These are the feature vectors obtained after inputting different samples. For the eigenvector f i with f j Euclidean distance in the feature space after normalization; The features with the closest feature similarity are matched as gait features belonging to the same pedestrian; 3.3) The model is trained simultaneously using cross-entropy loss and triplet loss, with the final loss being a weighted sum of the two; the final loss L com Represented as: L com =αL cse +βL tri (12) Where α = 0.5, β = 0.5; Cross-entropy loss L cse definition: Where N represents the total number of input samples during training, x is the recognition feature output by the model, p(·) is the probability value of the current feature belonging to the target label, and q(·) is the probability value of the current feature not belonging to the target label, i.e. q(x) = 1 - p(x). Triple loss L tri definition: Where N represents the total number of input samples during training. Indicates the current sample, Indicates and Samples from the same pedestrian tag, Indicates and For samples with different labels, f(·) represents the feature extraction operation corresponding to the model, α is the boundary of the triplet loss, set to 0.3, and the operation [γ]... + It equals max(γ,0).

5. The gait recognition method based on an event camera according to claim 1, 2, 3, or 4, characterized in that, The pedestrian gait videos mentioned above were all captured by event cameras, or captured by RGB cameras and then converted into event streams using an event camera simulator.

Images