A pedestrian detection method, system, chip and device based on event camera

Abstract

The present invention relates to the field of computer vision technology, and in particular to a pedestrian detection method based on an event camera, the method comprising: after the event camera is powered on, obtaining a current event stream within a preset time length through the event camera; preprocessing the current event stream to obtain stacked event frames of each frequency level in multiple frequency levels, and splicing the stacked event frames of multiple frequency levels to obtain a target event frame; subjecting the target event frame to feature processing of a preset number of blocks in a deep learning model in sequence to obtain pedestrian detection frame information of the current event stream, wherein each block of the deep learning model includes a local space feature extraction layer, a global space feature extraction layer, and a time feature fusion layer; sending the pedestrian detection frame information and the visualized event frame of the current event stream to a display screen and / or a memory connected to the event camera. The method utilizes the advantages of the event camera to achieve low latency, low power consumption, and high robustness pedestrian detection.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a pedestrian detection method, system, chip and device based on an event camera. Background Art

[0002] As one of the hot topics in today's science and technology field, autonomous driving technology is actually a fusion of a series of cutting-edge technologies. This fusion includes machine learning, sensor technology, artificial intelligence, computer vision and other technologies. On-board pedestrian detection is a key task in autonomous driving technology, which involves the safety and reliability of the vehicle. If the vehicle cannot accurately detect and respond to the presence of pedestrians, it may cause traffic accidents and even personal injury. Therefore, the research and development of on-board pedestrian detection technology is of great significance to improving the safety and reliability of autonomous driving technology.

[0003] Existing on-board pedestrian detection solutions are generally based on traditional RGB cameras for pedestrian detection. Commonly used methods for pedestrian detection based on traditional RGB cameras include image processing methods based on traditional computer vision technology and neural network methods based on deep learning technology. Image processing-based methods usually use image processing software to process and analyze the input image to detect the position and posture of pedestrians. Common image processing technologies include color segmentation, edge detection, morphological processing, etc. These methods usually require manual setting of some parameters and need to be optimized and adjusted for different scenarios. Neural network-based methods usually use deep learning technology to train the model so that it can automatically learn and recognize the characteristics and patterns of pedestrians. Commonly used neural network structures are mainly convolutional neural networks, such as Fast RCNN, YOLO and other models.

[0004] Compared with detection methods in other scenarios, pedestrian detection methods need to meet the following requirements:

[0005] (1) High precision and high reliability: Pedestrian detection methods for autonomous driving require high precision and high reliability to ensure that pedestrians can be accurately identified and detected in various scenarios to avoid accidents caused by misjudgment or missed detection.

[0006] (2) Real-time: Pedestrian detection methods for autonomous driving need to be performed in real time so that decisions and responses can be made in a short time. Therefore, efficient algorithms and hardware devices need to be selected to ensure that the system can process and respond to data quickly.

[0007] (3) Adaptability and robustness: Onboard pedestrian detection for autonomous driving needs to adapt to a variety of scenarios and environments, such as different lighting conditions, weather conditions, and road conditions. At the same time, it also needs to be robust to avoid misjudgment or missed detection due to noise and other interference factors.

[0008] Compared with traditional RGB cameras, event cameras have obvious advantages in pedestrian detection scenarios of autonomous driving, such as high frame rate, low latency, low power consumption and high dynamic range, and can better meet the business needs of pedestrian detection in autonomous driving. However, compared with traditional RGB cameras, the amount of data generated by event cameras is sparse, which makes the data features at different times vary greatly. The events generated by event cameras have no color information, which increases the difficulty of feature extraction in the pedestrian detection process, resulting in low accuracy of pedestrian detection results. In addition, event cameras record the changes of moving objects and scenes, and need to consider both spatial and temporal dimensions. This further increases the difficulty of feature extraction in the pedestrian detection process, resulting in low accuracy of pedestrian detection results. The pedestrian detection process based on event cameras also has the problem of low model generalization ability, which also leads to low accuracy of pedestrian detection results.

[0009] Therefore, event cameras have the special properties of data sparsity, spatiotemporal nature of features, and generalization ability of models using event cameras, which makes the neural network model based on event cameras far less mature and complete than the neural network model based on traditional RGB cameras. As a result, the existing pedestrian detection methods based on event cameras have the problem of low detection accuracy. Summary of the invention

[0010] The embodiments of the present application provide a pedestrian detection method, system, chip and device based on an event camera to at least solve the above-mentioned technical problems in the prior art.

[0011] In a first aspect, an embodiment of the present invention provides a pedestrian detection method based on an event camera, comprising:

[0012] After the event camera is powered on, the current event stream within a preset time length is obtained through the event camera;

[0013] Preprocessing the current event stream to obtain stacked event frames of each frequency level in multiple frequency levels, and splicing the stacked event frames of the multiple frequency levels to obtain a target event frame;

[0014] The target event frame is sequentially processed by a preset number of blocks in a deep learning model to obtain pedestrian detection frame information of the current event stream, wherein each block of the deep learning model includes a local spatial feature extraction layer, a global spatial feature extraction layer and a temporal feature fusion layer;

[0015] The pedestrian detection frame information and the visualized event frame of the current event stream are sent to a display screen and / or a memory connected to the event camera.

[0016] Based on the same inventive concept, in a second aspect, the present invention further provides a vehicle-mounted pedestrian detection system based on an event camera, which is used to execute the steps of the pedestrian detection method based on an event camera described in the first aspect, and the system comprises:

[0017] An event camera module, used to obtain a current event stream within a preset time period through the event camera after the event camera is powered on;

[0018] An event stream preprocessing module, used to preprocess the current event stream to obtain a stacked event frame of each frequency level in a plurality of frequency levels, and to splice the stacked event frames of the plurality of frequency levels to obtain a target event frame;

[0019] A deep learning algorithm module, used for sequentially subjecting the target event frame to feature processing of a preset number of blocks in a deep learning model to obtain pedestrian detection frame information of the current event stream, wherein each of the blocks in the deep learning model includes a local spatial feature extraction layer, a global spatial feature extraction layer, and a temporal feature fusion layer;

[0020] The detection result output module is used to send the pedestrian detection frame information and the visual event frame of the current event stream to a display screen and / or a memory connected to the event camera.

[0021] Based on the same inventive concept, in a third aspect, the present invention provides a chip, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of a pedestrian detection method based on an event camera when executing the program.

[0022] Based on the same inventive concept, in a fourth aspect, the present invention provides an electronic device, comprising the chip described in the third aspect.

[0023] One or more technical solutions in the embodiments of the present invention have at least the following technical effects or advantages:

[0024] In the pedestrian detection method based on event camera of the embodiment of the present invention, during the operation of the vehicle, the current event stream within the preset time length is first obtained through the event camera to utilize the unique advantages of low latency and low power consumption of the event camera to improve the pedestrian detection effect, solve the image blur caused by the movement of the traditional RGB camera, the recognition failure problem caused in the scene with weak lighting environment and large change amplitude, the low pedestrian detection efficiency and the low accuracy of pedestrian detection results. Then the current event stream is preprocessed to obtain the stacked event frame of each frequency level in multiple frequency levels, and the stacked event frames of multiple frequency levels are spliced ​​to obtain the target event frame, so as to convert the event data obtained by the event camera into the input format and the visual event frame format that meet the input format of the deep learning model. The preprocessing here includes the MFT (Multi Frequency Tensor) multi-frequency event feature representation processing method, so that the deep learning model uses the input of event frames of different frequency levels, so that the model can be more widely applied to different frequency times during reasoning, improve the generalization ability of the model, and increase the robustness to different frequency events. The preprocessing also includes a processing method for generating a visualization event frame from the current event stream, so that the pedestrian detection box information obtained by the deep learning model can be displayed on the visualization event frame.

[0025] Next, the target event frame is processed by the feature of a preset number of blocks in the deep learning model in turn to obtain the pedestrian detection frame information of the current event stream. Here, the target event frame is processed by the local spatial feature extraction layer, the global spatial feature extraction layer and the temporal feature fusion layer of each block of the deep learning model in turn to obtain the pedestrian detection frame information of the current event stream, so as to improve the precision and accuracy of the pedestrian detection frame information in the current event stream and enhance the pedestrian detection effect in the current event stream. In this way, the problem of low recognition accuracy and large difference in recognition results between the previous and next frames caused by reasons such as the sparsity of event camera data can be effectively solved. In addition, feature extraction is performed by considering the two dimensions of space and time for each block of the deep learning model, and the accuracy and robustness of the pedestrian detection process and results are further improved through the correlation of the two dimensions of space and time. Compared with similar methods, the detection method of the embodiment of the present invention has strong anti-motion blur capability, strong anti-illumination change capability, higher recognition result accuracy, high robustness, and can adapt to event inputs of different frequencies. Then, the pedestrian detection frame information and the visual event frame of the current event stream are sent to the display screen and / or memory of the vehicle to display the visual results of pedestrian detection in real time through the display screen, and can also be stored in the memory for subsequent viewing. Therefore, the detection method of the embodiment of the present invention has the advantages of low latency, low power consumption and high robustness of pedestrian detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Also, throughout the accompanying drawings, the same reference figures are used to represent the same components. In the drawings:

[0027] Figure 1 A schematic diagram of the steps of a pedestrian detection method based on an event camera in an embodiment of the present invention is shown;

[0028] Figure 2 A schematic diagram of the structure of obtaining a target event frame in an embodiment of the present invention is shown;

[0029] Figure 3 A schematic diagram of the structure of a deep learning model in an embodiment of the present invention is shown;

[0030] Figure 4 A schematic diagram showing the structure of each block of the deep learning model in an embodiment of the present invention is shown;

[0031] Figure 5 A schematic diagram showing the structure of the HF-SA processing process of the history association frame self-attention layer in an embodiment of the present invention is shown;

[0032] Figure 6 A module schematic diagram of a pedestrian detection system based on an event camera in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0033] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0034] Embodiment 1

[0035] The first embodiment of the present invention provides a pedestrian detection method based on an event camera, such as Figure 1 As shown, including:

[0036] S101, after the event camera is powered on, obtaining the current event stream within a preset time length through the event camera;

[0037] S102, preprocessing the current event stream to obtain stacked event frames of each frequency level in multiple frequency levels, and splicing the stacked event frames of the multiple frequency levels to obtain a target event frame;

[0038] S103, subjecting the target event frame to feature processing of a preset number of blocks in the deep learning model in sequence, to obtain pedestrian detection frame information of the current event stream, wherein each block of the deep learning model includes a local spatial feature extraction layer, a global spatial feature extraction layer, and a temporal feature fusion layer;

[0039] S104: Send the pedestrian detection frame information and the visualized event frame of the current event stream to a display screen and / or a memory connected to the event camera.

[0040] In the pedestrian detection method based on event camera of this embodiment, during the operation of the vehicle, the current event stream within the preset time length is first obtained through the event camera to utilize the unique advantages of low latency and low power consumption of the event camera to improve the pedestrian detection effect, solve the image blur caused by the movement of the traditional RGB camera, the recognition failure problem caused in the scene with weak lighting environment and large change amplitude, the low pedestrian detection efficiency and the low accuracy of pedestrian detection results. Then the current event stream is preprocessed to obtain the stacked event frame of each frequency level in multiple frequency levels, and the stacked event frames of multiple frequency levels are spliced ​​to obtain the target event frame, so as to convert the event data obtained by the event camera into the input format and the visual event frame format that meet the input format of the deep learning model. The preprocessing here includes the MFT (MultiFrequency Tensor) multi-frequency event feature representation processing method, so that the deep learning model uses the input of event frames of different frequency levels, so that the model can be more widely applied to different frequency times during reasoning, improve the generalization ability of the model, and increase the robustness to different frequency events. The preprocessing also includes a processing method for generating a visualization event frame from the current event stream, so that the pedestrian detection box information obtained by the deep learning model can be displayed on the visualization event frame.

[0041] Next, the target event frame is processed by the feature of a preset number of blocks in the deep learning model in turn to obtain the pedestrian detection frame information of the current event stream. Here, the target event frame is processed by the local spatial feature extraction layer, the global spatial feature extraction layer and the temporal feature fusion layer of each block of the deep learning model in turn to obtain the pedestrian detection frame information of the current event stream, so as to improve the precision and accuracy of the pedestrian detection frame information in the current event stream and enhance the pedestrian detection effect in the current event stream. In this way, the problem of low recognition accuracy and large difference in recognition results between the previous and next frames caused by reasons such as the sparsity of event camera data can be effectively solved. Moreover, feature extraction is performed by considering the two dimensions of space and time for each block of the deep learning model, and the accuracy and robustness of the pedestrian detection process and results are further improved through the correlation of the two dimensions of space and time. Compared with similar methods, the detection method of this embodiment has strong anti-motion blur capability, strong anti-illumination change capability, higher recognition result accuracy, high robustness, and can adapt to event inputs of different frequencies. Then, the pedestrian detection frame information and the visual event frame of the current event stream are sent to the display screen and / or memory of the vehicle to display the visual results of pedestrian detection in real time through the display screen, and can also be stored in the memory for subsequent viewing. Therefore, the detection method of this embodiment has the advantages of low latency, low power consumption and high robustness of pedestrian detection.

[0042] Next, combine Figure 1 The specific implementation steps of the pedestrian detection method based on the event camera provided in this embodiment are described in detail:

[0043] The detection method of this embodiment is not only applicable to the field of autonomous driving, but also to the fields of video surveillance, robotics, and AR (Augmented Reality), etc.

[0044] First, execute step S101. After the event camera is powered on, the current event stream within a preset time length is obtained through the event camera. Specifically, after the event camera is powered on, the current event stream within a preset time length is obtained through the event camera. The preset time length can be set according to actual needs and is represented by T. The current event stream includes several current events. Each event is represented by the data format of the following formula (1):

[0045] (1)

[0046] in, It is the horizontal and vertical coordinates of the current event on the pixel plane. is the timestamp, is the polarity of the current time, ∈{0, 1}.

[0047] The detection method of this embodiment is based on the event camera to obtain the current event stream, perform data processing and pedestrian detection analysis on the current event stream, and can effectively utilize the hardware advantages of the event camera to solve the image blur caused by the movement of the traditional RGB camera, the pedestrian recognition failure problem caused by weak lighting environment and large changes, the low pedestrian detection efficiency and the low accuracy of pedestrian detection results. In addition, since the event camera has obvious advantages such as high frame rate, low latency, low power consumption and high dynamic range, the detection method of this embodiment is implemented based on the event camera, so that the detection method of this embodiment also has the advantages of high frame rate, low latency and low power consumption.

[0048] Next, step S102 is executed to pre-process the current event stream to obtain stacked event frames of each frequency level in multiple frequency levels, and the stacked event frames of the multiple frequency levels are spliced ​​to obtain a target event frame.

[0049] Specifically, the frequency level and the number of frequency levels can be set according to actual needs. The preprocessing process includes two parts: one is to generate MFT (Multi Frequency Tensor) multi-frequency tensor. Specifically, at each frequency level, the current event stream is first divided into corresponding sub-event streams according to the frequency level. Then, in each corresponding sub-event stream, the positive polarity events and negative polarity events are counted, and the stacked event frames of the corresponding sub-events are obtained according to the positive polarity events and negative polarity events; after performing statistical operations on each corresponding sub-event, the stacked event frames of each corresponding sub-event are obtained. Then, according to the stacked event frames of each corresponding sub-event, the stacked event frames of the frequency level are obtained, and then the stacked event frames of each frequency level are obtained. The second is to generate a visual event frame of the current event stream according to the current event stream.

[0050] For example, the current event stream is (T, W, H), where T represents the time length, W represents the width of the event camera, and H represents the height of the event camera. The time length corresponding to the current event stream is T. Assume that three frequency levels are set, and the time length corresponding to each frequency level is T, T / 2, and T / 4.

[0051] The preprocessing process includes two parts. One is to generate a visualization event frame of the current event stream based on the current event stream. Specifically, the current event stream (T, W, H) counts positive polarity events and negative polarity events. A positive polarity event or a negative polarity event is a stacked event frame. Based on the positive polarity events and negative polarity events of the current event stream (T, W, H), a visualization event frame of the current event stream is generated.

[0052] The visual event frame counts the positive polarity (+) events and negative polarity (-) events within a fixed time length T. Taking the positive polarity (+) event as an example, it is expressed as follows:

[0053] (2)

[0054] Where δ is the Kronecker delta, and x and y represent the horizontal and vertical coordinates of each pixel, respectively. Here is an example of processing: all places where the positive polarity time h+(x, y) is greater than 0 are displayed using the first color (such as [255, 255, 255]), all places where the negative polarity time h-(x, y) is greater than 0 are displayed using the second color (such as [201, 126, 64]), and the remaining pixels are displayed using the third color (such as [52, 37, 30]). In this way, an RGB three-channel image is obtained, that is, the visual event frame of the current event stream. These three colors can be set and selected according to actual needs.

[0055] The second is to generate MFT (Multi Frequency Tensor). Specifically, Figure 2 As shown, according to the time length corresponding to the first frequency level, the current event stream (T, W, H) is divided into a corresponding sub-event stream (T, W, H). According to the time length corresponding to the second frequency level, the current event (T, W, H) stream is divided into two corresponding sub-event streams, that is, each sub-event stream corresponding to the second frequency level is (T / 2, W, H). According to the time length corresponding to the third frequency level, the current event (T, W, H) stream is divided into four corresponding sub-event streams, that is, each sub-event stream corresponding to the third frequency level is (T / 4, W, H).

[0056] At the first frequency level, a sub-event stream (T, W, H) corresponding to the first frequency level is divided, and the positive polarity events and negative polarity events are counted for the sub-event stream (T, W, H). For the method of obtaining the positive polarity events and the negative polarity events, please refer to formula (2).

[0057] like Figure 2 As shown, the positive polarity events and negative polarity events are counted for the sub-event stream corresponding to the first frequency level (ie, T), and the stacked event frames of the sub-event stream corresponding to the first frequency level are obtained. Since the number of sub-event streams corresponding to the first frequency level (ie, T) is one, namely, itself, the stacked event frames of the sub-event stream corresponding to the first frequency level are taken as the stacked event frames of the first frequency level.

[0058] Similarly, at the second frequency level, two sub-event streams corresponding to the second frequency level are divided, and each sub-event stream corresponding to the second frequency level is (T / 2, W, H). A stacked event frame is generated according to each sub-event stream corresponding to the second frequency level (T / 2, W, H). Figure 2 As shown, for each sub-event stream corresponding to the second frequency level (i.e., T / 2), the positive polarity events positive and the negative polarity events negative are counted for the sub-event stream corresponding to the second frequency level (i.e., T / 2), and the stacked event frames of the sub-event stream corresponding to the second frequency level are obtained. After performing such operations on each sub-event stream corresponding to the second frequency level (i.e., T / 2), the stacked event frames of the sub-event stream corresponding to the second frequency level (i.e., T / 2) are obtained. Since the number of sub-event streams corresponding to the second frequency level (i.e., T / 2) is two, the stacked event frames of the two sub-event streams corresponding to the second frequency level are both used as the stacked event frames of the second frequency level.

[0059] Similarly, at the third frequency level, two sub-event streams corresponding to the third frequency level are divided, and each sub-event stream corresponding to the third frequency level is (T / 4, W, H). A stacked event frame is generated based on each sub-event stream corresponding to the third frequency level (T / 4, W, H). Figure 2 As shown, for each sub-event stream corresponding to the third frequency level (i.e., T / 4), the positive polarity events positive and the negative polarity events negative are counted for the sub-event stream corresponding to the third frequency level (i.e., T / 2), and the stacked event frames of the sub-event stream corresponding to the third frequency level are obtained. After performing such operations on each sub-event stream corresponding to the third frequency level (i.e., T / 4), the stacked event frames of the sub-event stream corresponding to the third frequency level (i.e., T / 4) are obtained. Since the number of sub-event streams corresponding to the third frequency level (i.e., T / 4) is four, the stacked event frames of the four sub-event streams corresponding to the second frequency level are all taken as the stacked event frames of the third frequency level. Finally, the stacked event frames of each frequency level are obtained.

[0060] After the preprocessing process, the stacked event frames of each frequency level are spliced ​​to obtain the target event frame. Figure 2 For example, after the current event stream passes through the preprocessing process, the stacked event frames of each frequency level are obtained. The stacked event frames of each frequency level are spliced, that is, the stacked event frames of the first frequency level, the stacked event frames of the second frequency level, and the stacked event frames of the third frequency level are spliced ​​by channel to obtain the target event frame (14, H, W), that is, MFT Tensor (MFT multi-frequency tensor).

[0061] The method process for obtaining the target event frame in this embodiment is the MFT (Multi Frequency Tensor) multi-frequency event feature representation method. The data format of the event camera is different from the traditional RGB camera data format. In order to better apply the deep learning algorithm to this data format, the traditional method generally adopts a fixed time window stacking method, such as the processing of visualization frames, but this method will cause the trained model to be able to only adapt to a certain event frequency. When the application scenario adopts different event frame frequencies, it often leads to a sharp decrease in recognition accuracy, and the MFT multi-frequency event feature representation method allows the deep learning model to adapt to event frames of different frequencies during the reasoning process after training. In this way, the deep learning model and the detection method of this embodiment can be widely used in different event cameras and application scenarios, improve the generalization ability of the deep learning model, improve the precision and accuracy of the model's pedestrian detection, and enhance the effect of pedestrian detection. In addition, the MFT (Multi Frequency Tensor) multi-frequency event feature representation method also has the characteristics of low latency, low power consumption and high robustness.

[0062] Then, execute step S103, and process the target event frame in sequence through a preset number of blocks in the deep learning model to obtain pedestrian detection box information of the current event stream, wherein each block of the deep learning model includes a local spatial feature extraction layer, a global spatial feature extraction layer, and a temporal feature fusion layer.

[0063] Specifically, the deep learning model of this embodiment uses the HFT (Historical Frame Transformer) deep learning model to extract image information, and can combine historical information, which improves the accuracy and robustness of recognition compared to the YOLO model directly applied to a single frame. The preset number of blocks of the deep learning model of this embodiment can be set according to actual needs. Figure 3 The structure of a deep learning model is shown in Figure 1. Figure 3As shown, the number of blocks (i.e., HFT Block) of the deep learning model is four. They are connected in sequence, specifically, the target event frame MFTTensor ti is input into the first block HFT Block1, the output of HFT Block1 is connected to the input of the second block HFT Block2, the output of HFT Block2 is connected to the input of the third block HFT Block3, the output of HFT Block3 is connected to the input of the fourth block HFT Block4, and the output of HFT Block4 outputs the pedestrian detection box information through the detection module Detection Head, i.e., the pedestrian detection box information Detections ti of the target event frame MFT Tensor ti. The previous target event frame MFTTensor ti-1 obtains the pedestrian detection box information Detections ti-1 through the deep learning model.

[0064] The feature processing operation of each block of the deep learning model is: in each block, a. the input event frame is downsampled through the convolution layer to obtain the sampled event frame. b. the sampled event frame is subjected to local spatial feature extraction processing through the local spatial feature extraction layer to obtain the first event frame. After the first event frame passes through the first fully connected layer, the first target event frame is obtained. c. the first target event frame is subjected to global spatial feature extraction processing through the global spatial feature extraction layer to obtain the second event frame. After the second event frame passes through the second fully connected layer, the second target event frame is obtained. d. the second target event frame is subjected to temporal feature fusion processing through the temporal feature fusion layer to obtain the third event frame. The third event frame passes through the third fully connected layer to obtain the third target event frame. Among them, the input event frame of the first block is the target event frame, and the input event frames of the blocks except the first block are the third target event frames of the previous block.

[0065] After the target event frame passes through each block in turn to perform the above-mentioned feature processing operation, the third target event frame of the last block is obtained, and the third target event frame of the last block is passed through the detection module to obtain pedestrian detection frame information, wherein the pedestrian detection frame information includes the pedestrian detection frame on the visualization event frame, the position information of the pedestrian detection frame, the pedestrian detection frame confidence, etc.

[0066] Each block of the deep learning model includes a local spatial feature extraction layer, a global spatial feature extraction layer, and a temporal feature fusion layer. In this embodiment, the local spatial feature extraction layer is a self-attention mechanism layer Block-SA (Self-Attention Block), the global spatial feature extraction layer is a dilated attention mechanism layer Grid-SA (Grid Self-Attention Block), and the temporal feature fusion layer is a historical frame self-attention layer HF-SA (Historical Frame self attention) for computer vision tasks. Figure 4 As shown in the figure, each block (i.e., HFT Block) includes a Conv convolution layer, a self-attention mechanism layer Block-SA, a dilated attention mechanism layer Grid-SA, a historically associated frame self-attention layer HF-SA for computer vision tasks, and after Block-SA, Grid-SA, and HF-SA, each is connected to the next layer through a fully connected layer MLP (Fully Connected Layer). For example, after the self-attention mechanism layer Block-SA, the dilated attention mechanism layer Grid-SA is connected through a fully connected layer MLP (i.e., the first fully connected layer, the first MLP). After the dilated attention mechanism layer Grid-SA, the historically associated frame self-attention layer HF-SA for computer vision tasks is connected through a fully connected layer MLP (i.e., the second fully connected layer, the second MLP). After the historically associated frame self-attention layer HF-SA for computer vision tasks, the result is output through a fully connected layer MLP (i.e., the third fully connected layer, the third MLP). The function of the fully connected layer MLP is to map the input features to a new feature space, thereby extracting higher-level abstract features. This mapping capability enables the fully connected layer to capture complex patterns in the input data. The fully connected layer connects all neurons in the previous layer to all neurons in the next layer, which ensures the full propagation of information in the network. It has the characteristics of high expressiveness, easy implementation, and high flexibility, thereby improving the robustness of the model.

[0067] by Figure 3 The deep learning model shown is an example that details the feature processing of each block (HFT Block). Figure 4 As shown, in each block, in step a, the input event frame is downsampled through the convolution layer (i.e., Conv convolution layer) to obtain the sampled event frame [H 采样, W 采样 ,C 采样 ]. Among them, H 采样 Represents the height of the event frame after sampling, W 采样 Indicates the width of the event frame after sampling, C 采样Indicates the number of channels of the sampled event frame. The setting of this convolution layer can be set according to actual needs. An example setting of this convolution layer is that the convolution kernel of this convolution layer is set to {7,3,3,3}, and the stride is set to {4, 2, 2, 2}. Then, the spatial resolutions of the blocks (HFT Block) of these four deep learning models are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the spatial resolution of the target event frame respectively. Figure 3 As shown, the spatial resolution of HFT Block1 is 1 / 4 of the spatial resolution of the target event frame, that is, the spatial resolution is H / 4 and W / 4. The spatial resolution of HFT Block2 is 1 / 8 of the spatial resolution of the target event frame, that is, the spatial resolution is H / 8 and W / 8. The spatial resolution of HFT Block3 is 1 / 16 of the spatial resolution of the target event frame, that is, the spatial resolution is H / 16 and W / 16. The spatial resolution of HFT Block4 is 1 / 32 of the spatial resolution of the target event frame, that is, the spatial resolution is H / 32 and W / 32.

[0068] Step b: Perform local spatial feature extraction on the sampled event frame through the local spatial feature extraction layer, that is, through the self-attention mechanism layer Block-SA to obtain the first event frame. The first event frame passes through the first fully connected layer to obtain the first target event frame.

[0069] Specifically, the local spatial feature extraction process is to use the first window P*P to convert the sampled event frame [H 采样 , W 采样 , C 采样 ] to obtain the first window sample [H 采样 / P*W 采样 / P,P*P,C 采样 ]. And for the first window sample [H 采样 / P*W 采样 / P,P*P,C 采样 ] is processed by the self-attention mechanism, that is, the first event frame is obtained by the self-attention mechanism layer Block-SA. The first event frame is represented as [H 采样 / P*W 采样 / P,P*P,C 采样 ], only the data they carry is different. The first event frame is passed through the first fully connected layer to obtain the processed first event frame. The processed first event frames are spliced ​​to obtain the first target event frame [H1 目标 , W1 目标 , C1 目标 ]. The first target event frame [H1 目标 , W1 目标 , C1 目标] format is similar to the sampled event frame [H 采样 , W 采样 , C 采样 ] is consistent with the format. H1 目标 Indicates the height of the first target event frame, W1 目标 Indicates the width of the first target event frame, C1 目标 Indicates the channel number of the first target event frame.

[0070] The basic structure of the self-attention mechanism layer Block-SA includes:

[0071] Block-SA Query (Q): Block-SA query matrix, representing the information you want to obtain.

[0072] Block-SA Key (K): Block-SA key matrix, representing where information can be obtained.

[0073] Block-SA Value (V): Block-SA value matrix, representing the actual information obtained.

[0074] Attention Score (Attention Weights): The weight calculated based on the similarity between Block-SA's Query and Block-SA's Key.

[0075] The working principle of the self-attention mechanism layer Block-SA:

[0076] (1) Input features: Input features are usually feature maps extracted from previous network layers (such as convolutional layers or fully connected layers).

[0077] (2) Self-attention calculation: Calculate the Block-SA query, Block-SA key, and Block-SA value matrices, which are obtained by linearly transforming the input features. Calculate the Block-SA attention score between the Block-SA query and the Block-SA key, usually using matrix multiplication operations. Normalize the Block-SA attention score, usually using the softmax function. Multiply the normalized Block-SA attention score with the Block-SA value matrix to obtain the weighted value matrix of the Block-SA.

[0078] Advantages of the self-attention mechanism layer Block-SA:

[0079] (1) Reduce the amount of computation: By dividing the input feature map into multiple sub-regions, Block-SA can reduce the amount of computation because each sub-region only performs self-attention calculations with the features in its own region.

[0080] (2) Capturing locality: The self-attention mechanism within each sub-region can capture local information.

[0081] (3) Improve model efficiency: Compared with the full-graph self-attention mechanism, Block-SA improves the model’s operating efficiency by reducing unnecessary calculations.

[0082] (4) Strong adaptability: The size and number of sub-regions can be flexibly adjusted to adapt to different input sizes and task requirements.

[0083] The self-attention mechanism layer Block-SA refers to a neural network module or layer that includes a self-attention mechanism, which is mainly used to capture long-distance dependencies between input features. The self-attention mechanism allows the model to focus on different parts of the input sequence and perform weighted processing based on the contextual relationship. Through the self-attention mechanism, the model can pay more attention to important parts when processing sequence data or image features and ignore irrelevant information. In this way, the detection method of this embodiment extracts local spatial features of input data through the self-attention mechanism layer Block-SA to provide precision and accuracy of the detection method, enhance the effect of pedestrian detection, and ensure robustness.

[0084] Step c: Perform global spatial feature extraction on the first target event frame through the global spatial feature extraction layer, that is, through the expanded attention mechanism layer Grid-SA, to obtain a second event frame. The second event frame passes through the second fully connected layer to obtain a second target event frame.

[0085] Specifically, the global spatial feature extraction process is to use the second window G*G to extract the first target event frame [H1 目标 , W1 目标 , C1 目标 ] to obtain the second window sample [H1 目标 / G*W1 目标 / G,G*G,C1 目标 ]. And the second window sample is processed by the dilated attention mechanism, that is, the second event frame is obtained by the dilated attention mechanism layer Grid-SA. The second event frame is represented as [H1 目标 / G*W1 目标 / G,G*G,C1 目标 ], only the data they carry is different. After the second event frame passes through the second fully connected layer, the processed second event frame is obtained. Then the processed second event frames are spliced ​​to obtain the second target event frame [H2目标 , W2 目标 , C2 目标 ]. The size of the second window G*G is larger than the size of the first window P*P. The second target event frame [H2 目标 , W2 目标 , C2 目标 ] is in the same format as the first target event frame [H1 目标 , W1 目标 , C1 目标 ] is consistent with the format. H2 目标 Indicates the height of the second target event frame, W2 目标 Indicates the width of the second target event frame, C2 目标 Indicates the channel number of the second target event frame.

[0086] The working principle of the expanded attention mechanism layer Grid-SA is:

[0087] (1) Grid division: The input feature map is divided into multiple non-overlapping sub-regions (or grids), each of which can be regarded as a local context.

[0088] (2) The basic structure of the expanded attention mechanism layer Grid-SA:

[0089] Grid-SA Query (Q): Grid-SA query matrix. The features in each sub-region are used to calculate the Grid-SA query matrix.

[0090] Grid-SA Key (K): The key matrix of Grid-SA. The features in each sub-region are used to calculate the Key matrix of Grid-SA.

[0091] Grid-SA Value (V): Grid-SA value matrix. The features in each sub-region are used to calculate the Grid-SA Value matrix.

[0092] The specific calculation process is:

[0093] Calculate the Grid-SA attention score between the Grid-SA query and the Grid-SA key, usually using matrix multiplication. Normalize the Grid-SA attention score, usually using the softmax function. Multiply the normalized Grid-SA attention score with the Grid-SA value matrix to obtain the weighted Grid-SA value matrix.

[0094] Advantages of the expanded attention block Grid-SA:

[0095] (1) Reduce the amount of computation: By dividing the input feature map into multiple sub-regions, Grid-SA can reduce the amount of computation because each sub-region only performs self-attention calculations with the features in its own region.

[0096] (2) Capturing local and global information: The self-attention mechanism in each sub-region can capture local information. In addition, since the size of the second window G*G is larger than the size of the first window P*P, Grid-SA can extract global features and associate features of a larger region within the range of G.

[0097] (3) Improve model efficiency: Compared with the full-graph self-attention mechanism, Grid-SA improves the model’s operating efficiency by reducing unnecessary calculations.

[0098] (4) Strong adaptability: The size and number of sub-regions can be flexibly adjusted to adapt to different input sizes and task requirements.

[0099] Therefore, the expanded attention mechanism layer Grid-SA is an effective self-attention mechanism that captures local and global information by dividing the input feature map into multiple sub-regions while reducing the amount of computation. This mechanism is particularly suitable for feature extraction and context modeling in computer vision tasks. By adjusting the size and number of sub-regions, Grid-SA can adapt to different input sizes and task requirements. The detection method of this embodiment extracts the global spatial features of the input data through the expanded attention mechanism layer Grid-SA to provide the precision and accuracy of the detection method, enhance the effect of pedestrian detection, and ensure robustness.

[0100] Therefore, each block of the deep learning model used in the detection method of this embodiment can extract local and global spatial features at the same time through the alternation of the self-attention mechanism layer Block-SA and the expanded attention block Grid-SA, greatly improving the precision and accuracy of the model and the detection method of this embodiment, improving the efficiency of pedestrian detection, and enhancing the effect of pedestrian detection. It can also make the model and the detection method of this embodiment have the advantages of low latency, low power consumption and high robustness of pedestrian detection.

[0101] Step d: Through the temporal feature fusion layer, that is, through the historical association frame self-attention layer HF-SA for computer vision tasks, the second target event frame is subjected to temporal feature fusion processing to obtain a third event frame. The third event frame is passed through the third fully connected layer to obtain a third target event frame.

[0102] Specifically, the history-related frame self-attention layer HF-SA is designed to more effectively capture local and global contextual information in the image while reducing the amount of computation and improving model efficiency. The HF-SA structure is often used to strengthen feature representation, especially in tasks such as image classification, object detection, and semantic segmentation.

[0103] Working principle of HF-SA structure:

[0104] (1) Hierarchical division: The input feature map is divided into multiple levels, each level represents feature information of different scales.

[0105] (2) Calculation of HF-SA:

[0106] HF-SA Query (Q): The query matrix of HF-SA. The features in each level are used to calculate the Query matrix.

[0107] Key (K) of HF-SA: The key matrix of HF-SA. The features in each level are used to calculate the Key matrix.

[0108] Value (V) of HF-SA: The value matrix of HF-SA. The features in each level are used to calculate the Value matrix.

[0109] The specific calculation process is:

[0110] Calculate the HF-SA attention score between the HF-SA query and the HF-SA key, usually using matrix multiplication. Normalize the HF-SA attention score, usually using the softmax function. Multiply the normalized HF-SA attention score with the HF-SA value matrix to obtain the HF-SA weighted value matrix.

[0111] Therefore, the temporal feature fusion processing process is as follows: Figure 5 As shown, the second target event frame is processed by the Q matrix of the historical association frame self-attention layer HF-SA to obtain the Q matrix result of the second target event frame. The historical event frame is processed by the K matrix of the historical association frame self-attention layer HF-SA to obtain the K matrix result of the historical event frame, and the historical event frame is processed by the V matrix of the historical association frame self-attention layer HF-SA to obtain the V matrix result of the historical event frame. The historical event frame is an event frame generated by the previous event stream of the current event stream.

[0112] According to the Q matrix result of the historical association frame self-attention layer HF-SA and the K matrix result of the historical association frame self-attention layer HF-SA, a QKT operation is performed to obtain the first result. According to the first result and the V matrix result of the historical association frame self-attention layer HF-SA, a softmax function is used to perform softmax (QKT / )V, and get the third event frame. Represents the dimension of the K matrix. The third event frame passes through the third fully connected layer to obtain the third target event frame. In addition, the calculation principles of the self-attention mechanism layer Block-SA, the dilated attention mechanism layer Grid-SA and the historical association frame self-attention layer HF-SA can all be calculated through this formula To express.

[0113] It should be noted that in the feature processing process of the target event frame MFT Tensor ti, the historical event frame used in the historical association frame self-attention layer HF-SA of each block is the event frame generated by the corresponding block in the feature processing process of the historical event frame. Figure 3 As shown, MFT Tensor ti-1 is the historical event frame, and the historical event frame used in the historical association frame self-attention layer HF-SA of HFT Block1 is the event frame generated by HFT Block1 in the feature processing process of MFT Tensor ti-1, and so on for other blocks of the deep learning model.

[0114] In the detection method of this embodiment and in each block of the model, the temporal feature extraction utilizes the HF-SA structure to utilize the historical event frames. The historical association frame self-attention layer HF-SA has the following advantages:

[0115] Multi-scale information: Through hierarchical division, HF-SA can simultaneously capture local and global information in the image, as well as features of different scales.

[0116] Reduce the amount of computation: The amount of computation can be reduced by dividing the layers, because each layer only performs self-attention calculations with the features within its own layer.

[0117] Hierarchical fusion: Feature fusion at different levels can capture richer contextual information.

[0118] Strong adaptability: The number of layers and the feature size of each layer can be flexibly adjusted to adapt to different input sizes and task requirements.

[0119] Therefore, compared with other pedestrian detection for videos, which often use artificially designed post-processing methods to combine the detection results of the previous and next time periods, the HF-SA structure is used for automatic learning through the network, so that the detection method and model of this embodiment have better accuracy and robustness. In addition, each HFT Block of the model uses the HF-SA structure, rather than the last HFT Block using the HF-SA structure, so that each HFT Block can be adjusted to produce better results.

[0120] The detection method and HFT deep learning model structure of this embodiment can more effectively extract spatial features and temporal features, giving full play to the advantages of deep learning and event cameras. It can also effectively solve the problems of low recognition accuracy and large differences in recognition results between previous and next frames caused by reasons such as the sparsity of event camera data. Compared with similar method models, the detection method and model of this embodiment have strong anti-motion blur and anti-illumination change capabilities, higher recognition result accuracy, high robustness, and can adapt to event inputs of different frequencies. In this way, the precision and accuracy of the model and the detection method of this embodiment are greatly improved, the efficiency of pedestrian detection is improved, and the pedestrian detection effect is enhanced. It can also make the model and the detection method of this embodiment have the advantages of low latency, low power consumption and high robustness for pedestrian detection.

[0121] After the target event frame passes through each block in turn to perform the above-mentioned feature processing operation, the third target event frame of the last block is obtained, and the third target event frame of the last block is used as the pedestrian detection frame information. Finally, step S104 is executed to send the pedestrian detection frame information and the visual event frame of the current event stream to the display screen and / or memory connected to the event camera, so that the display screen displays the pedestrian detection frame information on the visual event frame in real time, avoiding accidents caused by misjudgment or missed detection, and improving safety. At the same time, the pedestrian detection frame information and the visual event frame of the current event stream can be stored in the memory for subsequent viewing.

[0122] Due to the special nature of event cameras, event cameras have the following problems:

[0123] Data sparsity: An event camera is a special camera that only records motion and changes in an image, while static scenes do not generate events. Therefore, the amount of data generated by an event camera is much sparser than that of a traditional RGB camera. This makes the data features at different times vary greatly, and events have no color information, which increases the difficulty of feature extraction in the pedestrian detection process.

[0124] The spatiotemporal nature of features: Event cameras record moving objects and changes in scenes, so both spatial and temporal dimensions need to be considered simultaneously. In deep learning models, how to effectively utilize information in these two dimensions is a challenge.

[0125] Model generalization ability: Due to the sparsity and imbalance of event camera data, trained deep learning models may encounter generalization problems when facing new and unseen scenes and data. How to improve the generalization ability of the model is another challenge.

[0126] The above problems have been overcome by the detection method and HFT deep learning model structure of this embodiment. The detection method and model of this embodiment take advantage of the unique advantages of event cameras and use the MFT (Multi Frequency Tensor) multi-frequency event feature representation method, so that after the model training is completed, it can adapt to event frames of different frequencies during the reasoning process, and then be widely used in different event cameras and application scenarios, improve the generalization ability of the model, improve the precision and accuracy of the model's pedestrian detection, and enhance the effect of pedestrian detection.

[0127] The HFT (Historical Frame Transformer) deep learning model is also used to implement a low-latency, low-power, and highly robust pedestrian detection method. The detection method and model of this embodiment can more effectively extract spatial and temporal features, giving full play to the advantages of deep learning and event cameras. It effectively solves the problems of low recognition accuracy and large differences in recognition results between previous and next frames caused by reasons such as the sparsity of event camera data. The detection method and model of this embodiment have strong anti-motion blur and anti-illumination change capabilities, higher recognition result accuracy, high robustness, and can adapt to event inputs of different frequencies. In this way, the precision and accuracy of the model and the detection method of this embodiment are greatly improved, the efficiency of pedestrian detection is improved, and the pedestrian detection effect is enhanced. It can also make the model and the detection method of this embodiment have the advantages of low latency, low power consumption, and high robustness for pedestrian detection.

[0128] Embodiment 2

[0129] Based on the same inventive concept, the second embodiment of the present invention further provides a vehicle-mounted pedestrian detection system based on an event camera, which is used to perform any method step in the pedestrian detection method based on an event camera described in the first embodiment, such as Figure 6 As shown, the system comprises:

[0130] The event camera module 201 is used to obtain the current event stream within a preset time through the event camera after the event camera is powered on;

[0131] An event stream preprocessing module 202 is used to preprocess the current event stream to obtain a stacked event frame of each frequency level in a plurality of frequency levels, and to splice the stacked event frames of the plurality of frequency levels to obtain a target event frame;

[0132] A deep learning algorithm module 203 is used to process the target event frame in sequence through a preset number of blocks in a deep learning model to obtain pedestrian detection frame information of the current event stream, wherein each block of the deep learning model includes a local spatial feature extraction layer, a global spatial feature extraction layer and a temporal feature fusion layer;

[0133] The detection result output module 204 is used to send the pedestrian detection frame information and the visual event frame of the current event stream to a display screen and / or a memory connected to the event camera.

[0134] Since the vehicle-mounted pedestrian detection system based on an event camera introduced in this embodiment is a system used to implement the pedestrian detection method based on an event camera in Example 1 of this application, based on the pedestrian detection method based on an event camera introduced in Example 1 of this application, the technical personnel in this field can understand the specific implementation method and various variations of the vehicle-mounted pedestrian detection system based on an event camera in this embodiment, so how the vehicle-mounted pedestrian detection system based on an event camera implements the method in Example 1 of this application will not be described in detail here. As long as the technical personnel in this field implement the system adopted by the pedestrian detection method based on an event camera in Example 1 of this application, it belongs to the scope of protection of this application.

[0135] Embodiment 3

[0136] Based on the same inventive concept, the third embodiment of the present invention also provides a chip, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any one of the above-mentioned event camera-based pedestrian detection methods are implemented.

[0137] Embodiment 4

[0138] Based on the same inventive concept, the fourth embodiment of the present invention further provides an electronic device, including the chip described in the third embodiment. The electronic device includes but is not limited to: an electronic device applied to the fields of autonomous driving, AR (Augmented Reality), VR (Virtual Reality), etc.

[0139] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0140] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0144] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A pedestrian detection method based on an event camera, characterized in that: include: After the event camera is powered on, the current event stream within a preset time length is obtained through the event camera; Preprocessing the current event stream to obtain stacked event frames of each frequency level in multiple frequency levels, and splicing the stacked event frames of the multiple frequency levels to obtain a target event frame; The target event frame is sequentially processed by a preset number of blocks in the deep learning model to obtain pedestrian detection frame information of the current event stream, wherein each block of the deep learning model includes a local spatial feature extraction layer, a global spatial feature extraction layer and a temporal feature fusion layer; the local spatial feature extraction layer is used to extract the local spatial features of the target event frame through the self-attention mechanism layer Block-SA, the global spatial feature extraction layer is used to extract the global spatial features of the target event frame through the expanded attention mechanism layer Grid-SA, and the temporal feature fusion layer is used to extract the temporal features of the target event frame through the historical association frame self-attention layer HF-SA; The pedestrian detection frame information and the visualized event frame of the current event stream are sent to a display screen and / or a memory connected to the event camera.

2. The method according to claim 1, characterized in that The step of sequentially subjecting the target event frame to feature processing of a preset number of blocks in the deep learning model to obtain pedestrian detection frame information of the current event stream includes: In each of the blocks, the input event frame is downsampled through a convolutional layer to obtain a sampled event frame; Performing local spatial feature extraction processing on the sampled event frame through the local spatial feature extraction layer to obtain a first event frame; After the first event frame passes through the first fully connected layer, a first target event frame is obtained; Performing global spatial feature extraction processing on the first target event frame through the global spatial feature extraction layer to obtain a second event frame; After the second event frame passes through the second fully connected layer, a second target event frame is obtained; Performing time feature fusion processing on the second target event frame through the time feature fusion layer to obtain a third event frame; The third event frame passes through a third fully connected layer to obtain a third target event frame; The input event frame of the first block is the target event frame, and the input event frames of the blocks other than the first block are the third target event frames of the previous block; After the target event frame passes through each of the blocks in turn to perform the above-mentioned feature processing operation, the third target event frame of the last block is obtained, and the third target event frame of the last block is passed through the detection module to obtain the pedestrian detection frame information.

3. The method according to claim 2, characterized in that The step of performing local spatial feature extraction processing on the sampled event frame through the local spatial feature extraction layer to obtain a first event frame includes: The sampled event frame is divided through a first window to obtain first window samples, and the first window samples are processed by a self-attention mechanism to obtain the first event frame.

4. The method according to claim 3, characterized in that After the first event frame passes through the first fully connected layer, a first target event frame is obtained, including: After the first event frame passes through the first fully connected layer, a processed first event frame is obtained, and then the processed first event frames are spliced ​​to obtain the first target event frame.

5. The method according to claim 3, characterized in that The step of performing global spatial feature extraction processing on the first target event frame through the global spatial feature extraction layer to obtain a second event frame includes: Dividing the first target event frame by a second window to obtain second window samples, and performing an expanded attention mechanism on the second window samples to obtain a second event frame, wherein the size of the second window is larger than the size of the first window; The second event frame passes through the second fully connected layer to obtain a second target event frame, including: After the second event frame passes through the second fully connected layer, a processed second event frame is obtained, and then the processed second event frames are spliced ​​to obtain the second target event frame.

6. The method according to claim 2, characterized in that The step of performing time feature fusion processing on the second target event frame through the time feature fusion layer to obtain a third event frame includes: Performing Q matrix processing of the history association frame self-attention layer HF-SA on the second target event frame to obtain a Q matrix result of the second target event frame; Performing K matrix processing of the historical association frame self-attention layer HF-SA on the historical event frame to obtain the K matrix result of the historical event frame, and performing V matrix processing of the historical association frame self-attention layer HF-SA on the historical event frame to obtain the V matrix result of the historical event frame; Obtaining a first result according to the Q matrix result of the historical association frame self-attention layer HF-SA and the K matrix result of the historical association frame self-attention layer HF-SA; The third event frame is obtained according to the first result and the V matrix result of the historical association frame self-attention layer HF-SA.

7. The method according to claim 1, characterized in that The preprocessing of the current event stream to obtain a stacked event frame of each frequency level in a plurality of frequency levels includes: At each frequency level, dividing the current event stream into corresponding sub-event streams according to the frequency level; In each of the corresponding sub-event streams, positive polarity events and negative polarity events are counted, and according to the positive polarity events and the negative polarity events, a stacked event frame of the corresponding sub-event is obtained; After performing statistical operations on each of the corresponding sub-events, a stacked event frame of each of the corresponding sub-events is obtained; According to the stacked event frames of each corresponding sub-event, the stacked event frames of the frequency level are obtained, and then the stacked event frames of each frequency level are obtained.

8. A vehicle-mounted pedestrian detection system based on an event camera, characterized in that: The system is used to perform the pedestrian detection method based on an event camera according to any one of claims 1 to 7, the system comprising: An event camera module, used to obtain a current event stream within a preset time period through the event camera after the event camera is powered on; An event stream preprocessing module, used to preprocess the current event stream to obtain a stacked event frame of each frequency level in a plurality of frequency levels, and to splice the stacked event frames of the plurality of frequency levels to obtain a target event frame; A deep learning algorithm module, used for sequentially subjecting the target event frame to feature processing of a preset number of blocks in a deep learning model to obtain pedestrian detection frame information of the current event stream, wherein each of the blocks in the deep learning model includes a local spatial feature extraction layer, a global spatial feature extraction layer, and a temporal feature fusion layer; The detection result output module is used to send the pedestrian detection frame information and the visual event frame of the current event stream to a display screen and / or a memory connected to the event camera.

9. A chip comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the pedestrian detection method based on an event camera as described in any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: The chip as claimed in claim 9.

Images