An event camera classification and detection method based on time-space-polarity domain decoupling

Abstract

This invention discloses an event camera classification detection method based on time-space-polarity domain decoupling. The method comprises the following steps: 1. data collection and preprocessing based on the event camera; 2. event representation conversion based on an event embedding module; 3. group feature extraction based on time-space-polarity domain decoupling; 4. classification detection based on the group features, network training, and obtaining a trained classification detection model. This method can extract time-space-polarity domain decoupling features from event streams captured by event cameras, thereby achieving high-performance classification detection.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular to an event camera classification detection method based on time-space-polarity domain decoupling. Background Art

[0002] Deep learning-based feature extraction plays a key role in the field of computer vision, extracting useful information from frames captured by visual cameras and playing a role in downstream tasks such as target recognition and detection. Although a variety of accurate and efficient feature extraction methods are currently available, they perform well on high-quality frame images. However, in real-world application scenarios, some extreme challenges still exist, which pose a severe test for traditional camera technology. For example, the image may be overexposed due to drastic changes in lighting, which may lead to loss of details in dim environments, and motion blur when shooting high-speed moving targets. These problems may all lead to target recognition failure or loss of detection.

[0003] To address these challenges, a new type of sensor inspired by biological vision has been introduced: the dynamic vision sensor (DVS), also known as an event camera. Unlike traditional frame-based cameras, DVS operates by asynchronously acquiring pixel-level "event" data, rather than capturing image frames on a fixed clock. When the illumination change at a pixel exceeds a threshold, an event is generated, recording the timestamp, x-coordinate, y-coordinate, and polarity of the illumination change. This event data can be viewed as a sparse spatiotemporal point cloud with high temporal resolution (over 10,000 frames / second), large dynamic range (typically over 120 dB), and low power consumption (less than 10 mW).

[0004] Currently, a range of feature extraction methods for processing event data exist. However, most of them are still based on traditional frame image designs, that is, first downsampling the event data into image representations and then extracting spatial information from the images. This approach overemphasizes spatial domain features and fails to fully consider the unique properties of event data, such as time and polarity. These special properties of event data are too tightly coupled with spatial features such as texture and edges, thus affecting the performance of downstream tasks such as classification and detection.

[0005] Therefore, there is an urgent need for a new method to better utilize the unique advantages of event data and enhance the performance of feature extraction under extreme conditions to further improve computer vision applications for these complex scenarios. Summary of the Invention

[0006] In order to address the shortcomings of the above-mentioned prior art, the present invention proposes an event camera classification detection method based on time-space-polarity domain decoupling, in order to extract the time-space-polarity domain decoupling features from the event stream collected by the event camera, thereby achieving high-performance classification detection.

[0007] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0008] The present invention provides an event camera classification detection method based on time-space-polarity domain decoupling, which comprises the following steps:

[0009] Step 1: Data collection and preprocessing based on event cameras:

[0010] An event camera is used to collect asynchronous event streams and RGB images, and the event streams are divided into single event stream samples according to timestamps, thereby obtaining M event stream samples, where the i-th event stream sample is recorded as E i ,and Among them, x i,j ,y i，j , t i，j ,p i,j Represents the i-th event stream sample E i The abscissa, ordinate, timestamp and polarity of the j-th event in N i Represents the i-th event stream sample E i The total number of events contained in , let the spatial range of a single event stream sample be H×W, where H and W are the maximum values of height and width respectively;

[0011] According to the i-th event stream sample E i Find the RGB image with the closest time stamp t0 of the first event in the image, mark the information of the object in the RGB image and form a normalized label l i , thus forming the event camera training dataset

[0012] Step 2: Build an event embedding module to convert asynchronous event streams into event representations:

[0013] Step 2.1: Take the i-th event stream sample E i Timestamp in Discretize it into a sequence of integers in the range [0, K-1] in, is the rounding down operation, K is the number of time intervals after discretization, t end For N i The timestamp of the event;

[0014] Step 2.2: Based on the i-th event stream sample E i The horizontal axis in vertical axis Get the i-th event stream sample E i The block position of And the i-th event stream sample E i The position of each pixel in the token block Where mod is the remainder operation, and P is the side length of the token block;

[0015] Step 2.3: Use formula (1) to get the i-th event stream sample E i One-dimensional array a i :

[0016]

[0017] In formula (1), p i Represents the i-th event stream sample E i Polarity;

[0018] Step 2.4, for a i After frequency statistics, matrix splicing and matrix reshaping operations are performed to obtain the i-th event representation

[0019] Step 2.5: Use the grouped convolutional layer and the MLP layer to represent R from the i-th event i Extract features from the ith event token containing G groups of time-space-polarity features Among them, token i,k Indicates T i The middle dimension is the kth token of G·C, where G is the number of groups divided according to time and polarity, and C is the number of channels in each group, that is, the feature dimension of each group;

[0020] Step 2.6: Batch select B event stream samples from the event camera training dataset X Follow steps 2.1 to 2.5 to convert XE into event tokens.

[0021] Step 3: Construct a hierarchical group feature extraction network and use it to extract group features of event tokens XT based on time-space-polarity domain decoupling to obtain the output Among them, f l，b represents the b-th output of the l-th level;

[0022] Step 4: Build classification module and detection module:

[0023] Step 4.1: The classification module extracts the output of the Lth level in the hierarchical grouping feature extraction network. After the global average pooling operation, it is input into the fully connected layer for processing to obtain the normalized prediction labels of a batch of event stream samples XE Among them, p b，c Represents the bth event stream sample E among the B event stream samples XE selected in a batch b The corresponding predicted label is the probability of the cth category, Cls is the total number of categories;

[0024] Step 4.2: The detection module will output F out Send it to the YOLOX detection layer for processing and output the target detection results Among them, P′ b Represents the bth event stream sample E b The target detection result, P′ b ∈R N×(Cls+5) , N is the number of detection boxes in the target detection result, and each detection box contains Cls+5 prediction values, which are the rate of the target in the detection box being of various categories, the horizontal coordinate of the center point of the detection box, the vertical coordinate of the center point of the detection box, the width of the detection box, the length of the detection box and the confidence level;

[0025] Detection results are improved by confidence screening and non-maximum suppression Filter to get the final test results Among them, pd b Represents the bth event stream sample E b The prediction result of pd b ∈R N-×C l s+5 , N - is the number of remaining detection frames after screening;

[0026] Step 5: Network training:

[0027] Step 5.1: For classification tasks, use formula (2) to construct the cross entropy loss L CE :

[0028]

[0029] In formula (2), p b is the bth event stream sample E b The probability that the corresponding predicted label is correctly classified;

[0030] Step 5.2: For the detection task, use formula (3) to construct the total loss function L:

[0031] L=a·L Conf +b·L Cls+c·L Reg (3)

[0032] In formula (3), L Reg Represents the regression loss, and is obtained from formula (4), L Conf represents the confidence loss and is expressed by p b Get pd b The confidence level in L is used to calculate Cls Denotes the category loss and is expressed by p b Get pd b The probability of the target in the detection box being correctly classified is used to calculate, and a, b, and c are the weights of the three losses;

[0033] L Reg =-log(IoU(PD box ,GT)) (4)

[0034] In formula (3), PD box is the horizontal coordinate of the center point of the detection box in the detection result PD, the vertical coordinate of the center point of the detection box, the width of the detection box, and the length of the detection box; GT is the horizontal coordinate of the center point of the corresponding detection box in the normalized label, the vertical coordinate of the center point of the detection box, the width of the detection box, and the length of the detection box; IoU(·) refers to the calculation of the intersection over union ratio of two sets of boxes;

[0035] Step 5.3: Use the gradient descent method to train the event embedding module, hierarchical grouping feature extraction network, classification module and detection module, and calculate the loss function to update the network parameters until the loss converges, thereby obtaining a trained classification detection model for classifying and detecting event stream samples.

[0036] The event camera classification detection method based on time-space-polarity domain decoupling described in the present invention is also characterized in that the first layer in the hierarchical grouping feature extraction network of step three includes a dual-residual event self-attention module, and each of the remaining layers includes a grouping feature aggregation module and a dual-residual event self-attention module;

[0037] The dual residual event self-attention module is composed of a window operation layer, a spatial self-attention layer, a grouping self-attention layer, a merging operation layer, and a feature fusion layer in sequence;

[0038] Step 3.1.1: When l=1, the window operation layer in the lth level performs windowing on the event token XT to obtain the event token after windowing at the lth level. Among them, S is the side length of the window, T l，w represents the event token in the w-th window at the l-th level, and token l，w，m The event token T of the wth window at the lth levell，w The mth token in ;

[0039] Step 3.1.2: When l = 1, the spatial self-attention layer in the lth level pays attention to the token l，w，m The relative spatial position within the window is encoded to obtain the mth relative position bias bp of each window at the lth level l，m ∈[-S+1, S-1], thus obtaining the total relative position bias of the lth level

[0040] Bias the total relative position by Bp l and the event token T in the w-th window l，w After adding, self-attention operation is performed to obtain the spatial features of the lth level Among them, WFS l，w The event token T of the wth window at the lth level l，w The extracted spatial features, and wfs l，w，m The event token T of the wth window at the lth level l，w The mth spatial feature extracted;

[0041] Step 3.1.3, when l = 1, the grouped self-attention layer in the lth level will take the spatial feature FS of the lth level l Transpose to obtain the transposed spatial features Among them, WFS T l，w represents the spatial features of the w-th window after transposition at the l-th level, and wfs T l，w，n Represents the nth spatial feature after the transposition of the wth window at the lth level;

[0042] The grouped self-attention layer in the lth level is wfs T l，w，n The relative group and channel relationships within the window are encoded to obtain the nth relative group bias bg for each window at the lth level l，n ∈[-G+1, G-1], thus obtaining the total relative group bias of the lth level

[0043] Bias the total relative position by Bg l The spatial feature WFS after transposing the w-th window T l，w After adding, self-attention operation and transposition operation are performed to obtain the grouping features of the lth level Among them, WFG l，w Represents the spatial feature WFS after the transposition of the w-th window at the l-th levelT l，w The extracted grouping features, and Among them, wfg l，w，n Represents the spatial feature WFS after the transposition of the w-th window at the l-th level T l，w The nth grouping feature extracted;

[0044] Step 3.1.4, when l=1, the merging operation layer in the lth level combines the spatial features FS l and grouping feature FG l Merge them separately to get the space tokens at level l and grouping tokens Among them, ST l，b represents the bth spatial token at the lth level, and st l，b，k represents the kth token in the bth spatial token at the lth level, GT b represents the bth grouping token at the lth level, and gt l，b，k represents the kth token in the bth grouped token at the lth level;

[0045] Step 3.1.5, when l=1, the feature fusion layer in the lth level combines the spatial token XST of the lth level through double residual connection, MLP layer, and regularization layer l and grouping token XGT l Fusion is performed to obtain the time-space-polarity domain token of the lth level Among them, SGT l，b represents the bth time-space-polarity domain token at the lth level and is recorded as the bth output f at the lth level l，b ,and sgt l，b，k represents the kth token in the bth time-space-polarity domain token at the lth level;

[0046] Step 3.1.6: When l=2, the grouping feature aggregation module in the lth level combines the time-space-polarity domain token XSGT of the l-1th level l-1 After processing by overlapping convolutional layers and regularization layers, we get the aggregated tokens at the lth level. Among them, AT l，b represents the bth aggregate token at the lth level, and at l，b，k represents the kth token in the bth aggregate token at the lth level, at l，b，k The dimension is at l，b，k The number of groups is

[0047] Aggregate token XAT at level l l Input the double residual event self-attention module of the lth level, and follow the process of steps 3.1.1 to 3.1.5 to obtain the aggregated time-space-polarity domain token of the lth level Among them, VT l，b represents the bth aggregated time-space-polarity domain token at the lth level and is recorded as the bth output f at the lth level l，b ;

[0048] Step 3.1.7, when l∈[3,L], the aggregated time-space-polarity domain token XVT of the l-1th layer l-1 After being input into the group feature aggregation module of the lth level for processing, it passes through the double residual event self-attention module of the lth level to obtain the aggregated time-space-polarity domain token of the lth level. Among them, VT l，b represents the bth aggregated time-space-polarity domain token at the lth level and is recorded as the bth output f at the lth level l，b ; and constitute the output of the hierarchical grouping feature extraction network in, f l，b The number of groups is

[0049] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute the event camera classification detection method, and the processor is configured to execute the program stored in the memory.

[0050] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the steps of the event camera classification detection method when the computer program is executed by a processor.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. This invention utilizes an event camera feature extraction method based on time-space-polarity domain decoupling, significantly improving feature extraction performance under extreme conditions compared to existing technologies. Traditional frame image feature extraction methods are prone to failure in complex scenarios such as large lighting variations, dim environments, and high-speed moving targets. The method provided by this invention effectively addresses these problems by leveraging the unique data source of event cameras. Therefore, this invention possesses significant technical advantages in feature extraction and can address various real-world challenges.

[0053] 2. Through the event embedding module, the present invention can fully preserve the information of event data and achieve efficient information decoupling. This feature helps improve the stability and robustness of features, thereby providing more reliable performance in downstream tasks such as object recognition and detection.

[0054] 3. This paper introduces a dual-residual event self-attention module to effectively extract features in the temporal-spatial-polarity domain. Compared with traditional spatial self-attention methods, this module better captures the temporal and polarity information of event data, further improving the expressiveness of features.

[0055] 4. The present invention's grouped feature aggregation module facilitates efficient and reliable spatiotemporal information aggregation across the entire network, further enhancing performance. This module's design fully considers the unique properties of event camera data, ensuring that features remain efficient in a variety of complex scenarios.

[0056] 5. The classification detection network provided by the present invention relies on an event embedding module, a hierarchical grouping feature extraction network, a classification module, and a detection module. It has a low parameter count, computational complexity, and fast operation speed, making it easy to deploy and apply to various event camera applications. This makes the method of the present invention more practical and economical in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a structural diagram of the event embedding module of the present invention;

[0058] Figure 2 This is a diagram of the event embedding module and hierarchical grouping feature extraction network structure of the present invention;

[0059] Figure 3 This is the structure diagram of the dual residual event self-attention module of the present invention;

[0060] Figure 4 This is a structural diagram of the grouping feature aggregation module of the present invention. DETAILED DESCRIPTION

[0061] In this embodiment, an event camera classification and detection method based on time-space-polarity domain decoupling fully utilizes the unique properties of event camera data, especially the time-space-polarity domain information, to overcome the limitations of traditional feature extraction methods in complex scenarios, significantly improve performance while maintaining low resource consumption, and bring significant technological progress and application prospects to the field of computer vision. Specifically, it includes the following steps:

[0062] Step 1: Data collection and preprocessing based on event cameras:

[0063] The event camera is used to collect asynchronous event streams and RGB images, and the event streams are divided into single event stream samples according to the timestamps, thereby obtaining M event stream samples, where the i-th event stream sample is recorded as E i ,and Among them, x i,j ,y i，j , t i，j , p i,j Represents the i-th event stream sample E i The abscissa, ordinate, timestamp and polarity of the j-th event in N i Represents the i-th event stream sample E i The total number of events contained in is , and the spatial extent of a single event stream sample is assumed to be H × W, where H and W are the maximum height and width, respectively. Unlike traditional frame cameras, event cameras acquire event data asynchronously, meaning that events are triggered and recorded only when pixel-level illumination changes occur. This event data is stored in files in formats such as .npy or .h5py, and features high temporal resolution and low power consumption.

[0064] According to the i-th event stream sample E i Find the RGB image with the closest time stamp t0 of the first event in the image, mark the information of the object in the RGB image and form a normalized label l i , thus forming the event camera training dataset

[0065] Step 2: Figure 1 As shown, an event embedding module is built to convert asynchronous event streams into event representations:

[0066] Step 2.1: Take the i-th event stream sample E i Timestamp in Discretize it into a sequence of integers in the range [0, K-1] in, is the rounding down operation, K is the number of time intervals after discretization, t end For N i The timestamp of the event;

[0067] Step 2.2: Based on the i-th event stream sample E i The horizontal axis in vertical axis Get the i-th event stream sample E i The block position of And the i-th event stream sample E i The position of each pixel in the token block Where mod is the remainder operation, and P is the side length of the token block;

[0068] Step 2.3: Use formula (1) to get the i-th event stream sample E i One-dimensional array a i :

[0069]

[0070] In formula (1), p i Represents the i-th event stream sample E i Polarity;

[0071] Step 2.4, for a i After frequency statistics, matrix splicing and matrix reshaping operations are performed to obtain the i-th event representation This step helps synchronize asynchronous event data, making it easier to use for feature extraction;

[0072] Step 2.5: Use the grouped convolutional layer and the MLP layer to represent R from the i-th event i Extract features from the ith event token containing G groups of time-space-polarity features Among them, token i,k Indicates T i The middle dimension is the kth token of G·C, where G is the number of groups divided according to time and polarity, and C is the number of channels in each group, that is, the feature dimension of each group;

[0073] Step 2.6: Batch select B event stream samples from the event camera training dataset X Follow steps 2.1 to 2.5 to convert XE into event tokens. These event tokens will preserve the event time-space-polarity information, and the entire conversion process is very efficient.

[0074] Step 3: Group feature extraction based on time-space-polarity domain decoupling:

[0075] Step 3.1, such as Figure 2 As shown in the figure, a hierarchical grouping feature extraction network is constructed after the event embedding module, which includes L layers. The first layer contains a dual-residual event self-attention module, and each of the remaining layers contains a grouping feature aggregation module and a dual-residual event self-attention module. The design and structure of these modules are to better utilize the unique properties of event data, especially the time-space-polarity domain information, so as to improve the performance of feature extraction.

[0076] like Figure 3 As shown in Figure 2, the dual residual event self-attention module consists of a window operation layer, a spatial self-attention layer, a grouped self-attention layer, a merging operation layer, and a feature fusion layer.

[0077] Step 3.1.1: When l=1, the window operation layer in the lth level performs windowing on the event token XT to obtain the event token after windowing at the lth level. Among them, S is the side length of the window, T l，w represents the event token in the w-th window at the l-th level, and token l，w，m The event token T of the wth window at the lth level l，w The mth token in ;

[0078] Step 3.1.2: When l = 1, the spatial self-attention layer in the lth level pays attention to the token l，w，m The relative spatial position within the window is encoded to obtain the mth relative position bias bp of each window at the lth level l，m ∈[-S+1, S-1], thus obtaining the total relative position bias of the lth level

[0079] Bias the total relative position by Bp l and the event token T in the w-th window l，w After adding, self-attention operation is performed to obtain the spatial features of the lth level Among them, WFS l，w The event token T of the wth window at the lth level l，w The extracted spatial features, and wfs l，w，m The event token T of the wth window at the lth level l，w The mth spatial feature extracted; in the spatial self-attention layer, we emphasize the spatial features of the event data, so that we can better capture the location and shape information of the object.

[0080] Step 3.1.3, we further proposed an efficient grouped self-attention to make up for the lack of spatial self-attention in the feature extraction ability of the time-polarity domain. When l = 1, the grouped self-attention layer in the lth level will take the spatial feature FS of the lth level as the input. l Transpose to obtain the transposed spatial features Among them, WFS T l，w represents the spatial features of the w-th window after transposition at the l-th level, and wfs T l，w，n Represents the nth spatial feature after the transposition of the wth window at the lth level;

[0081] The grouped self-attention layer in the lth level is wfs T l，w，nThe relative group and channel relationships within the window are encoded to obtain the nth relative group bias bg for each window at the lth level l，n ∈[-G+1, G-1], thus obtaining the total relative group bias of the lth level

[0082] Bias the total relative position by Bg l The spatial feature WFS after transposing the w-th window T l，w After adding, self-attention operation and transposition operation are performed to obtain the grouping features of the lth level Among them, WFG l，w Represents the spatial feature WFS after the transposition of the w-th window at the l-th level T l，w The extracted grouping features, and Among them, wfg l，w，n Represents the spatial feature WFS after the transposition of the w-th window at the l-th level T l，w The nth grouping feature extracted;

[0083] Step 3.1.4, when l=1, the merging operation layer in the lth level combines the spatial features FS l and grouping feature FG l Merge them separately to get the space tokens at level l and grouping tokens Among them, ST l，b represents the bth spatial token at the lth level, and st l，b，k represents the kth token in the bth spatial token at the lth level, GT b represents the bth grouping token at the lth level, and gt l，b，k represents the kth token in the bth grouped token at the lth level;

[0084] Step 3.1.5, when l=1, the feature fusion layer in the lth level combines the spatial token XST of the lth level through double residual connection, MLP layer, and regularization layer l and grouping token XGT l Fusion is performed to obtain the time-space-polarity domain token of the lth level At the same time, it ensures that the characteristics of the time-space-polarity domain will not be over-coupled; among them, SGT l，b represents the bth time-space-polarity domain token at the lth level and is recorded as the bth output f at the lth level l，b ,and sgt l，b，krepresents the kth token in the bth time-space-polarity domain token at the lth level

[0085] Step 3.1.6, such as Figure 4 As shown, when l=2, the grouping feature aggregation module in the lth level combines the time-space-polarity domain token XSGT of the l-1th level l-1 After processing by overlapping convolutional layers and regularization layers, we get the aggregated tokens at the lth level. Among them, AT l，b represents the bth aggregate token at the lth level, and at l，b，k represents the kth token in the bth aggregate token at the lth level, at l，b，k The dimension is at l，b，k The number of groups is The main function of the grouping feature aggregation module is to achieve efficient aggregation of global information.

[0086] Aggregate token XAT at level l l Input the double residual event self-attention module of the lth level, and follow the process of steps 3.1.1 to 3.1.5 to obtain the aggregated time-space-polarity domain token of the lth level Among them, VT l，b represents the bth aggregated time-space-polarity domain token at the lth level and is recorded as the bth output f at the lth level l，b ;

[0087] Step 3.1.7, when l∈[3,L], the aggregated time-space-polarity domain token XVT of the l-1th layer l-1 After being input into the group feature aggregation module of the lth level for processing, it passes through the double residual event self-attention module of the lth level to obtain the aggregated time-space-polarity domain token of the lth level. Among them, VT l,b represents the bth aggregated time-space-polarity domain token at the lth level and is recorded as the bth output f at the lth level l，b ; and constitute the output of the hierarchical grouping feature extraction network Among them, f l，b Represents the bth event stream sample E b T b The output of the first layer in a hierarchical grouped feature extraction network; and f l，b The number of groups is

[0088] Step 4: Build classification module and detection module:

[0089] Step 4.1: The classification module extracts the output of the Lth level in the hierarchical grouping feature extraction network After the global average pooling operation, it is input into the fully connected layer for processing to obtain the normalized prediction labels of a batch of event stream samples XE Among them, p b,c Represents the bth event stream sample E among the B event stream samples XE selected in a batch b The corresponding predicted label is the probability of the cth category, Cls is the total number of categories;

[0090] Step 4.2, the detection module will output F out Send it to the YOLOX detection layer for processing and output the target detection results Among them, P′ b Represents the bth event stream sample E b The target detection result, P′ b ∈R N×(Cls+5) , N is the number of detection boxes in the target detection result, and each detection box contains Cls+5 prediction values, which are the rate of the target in the detection box being of various categories, the horizontal coordinate of the center point of the detection box, the vertical coordinate of the center point of the detection box, the width of the detection box, the length of the detection box and the confidence level;

[0091] Detection results are improved by confidence screening and non-maximum suppression Filter to get the final test results Among them, pd b Represents the bth event stream sample E b The prediction results; N - is the number of remaining detection frames after screening;

[0092] Step 5: Network training:

[0093] Step 5.1: For classification tasks, use formula (2) to construct the cross entropy loss L CE :

[0094]

[0095] In formula (2), p b is the bth event stream sample E b The probability that the corresponding predicted label is correctly classified;

[0096] Step 5.2: For the detection task, use formula (3) to construct the total loss function L:

[0097] L=a·L Conf +b·L Cls +c·L Reg (3)

[0098] In formula (3), L Reg Represents the regression loss, and is obtained from formula (4), L Conf represents the confidence loss and is expressed by p b Get pd b The confidence level in L is used to calculate Cls Denotes the category loss and is expressed by p b Get pd b The probability of the target in the detection box being correctly classified is used to calculate, and a, b, and c are the weights of the three losses;

[0099] L Reg =-log(IoU(PD box ,CT)) (4)

[0100] In formula (3), PD box is the horizontal coordinate of the center point of the detection box in the detection result PD, the vertical coordinate of the center point of the detection box, the width of the detection box, and the length of the detection box; GT is the horizontal coordinate of the center point of the corresponding detection box in the normalized label, the vertical coordinate of the center point of the detection box, the width of the detection box, and the length of the detection box; IoU(·) refers to the calculation of the intersection over union ratio of two sets of boxes;

[0101] Step 5.3: Use the gradient descent method to train the event embedding module, hierarchical grouping feature extraction network, classification module, and detection module, and calculate the loss function to update the network parameters until the loss converges, thereby obtaining a trained classification detection model for classifying and detecting event stream samples.

[0102] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0103] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0104] In order to verify the effectiveness of the proposed scheme, a complete test was carried out on four event camera-based classification datasets: Cifar10-DVS, N-MNIST, N-CARS and DVS128Gesture datasets, and two event camera-based object detection datasets: Gen1 and 1Mpx datasets.

[0105] The Cifar10-DVS dataset contains 10,000 simulated event samples from the Cifar10 dataset. It covers 10 different object classes and has a spatial resolution of 128×128. The N-MNIST dataset contains 70,000 simulated event streams of 10 handwritten digits at a resolution of 34×34. The N-CARS dataset contains 12,336 vehicle samples and 11,693 non-vehicle samples at various spatial resolutions. The DVS128Gesture dataset contains 1,342 real-time event streams captured by an event camera at a spatial resolution of 128×128. It covers 11 different action classes, including one random action class. The Gen1 dataset was collected at a resolution of 304×240 and annotated at a frequency of 20Hz. The Gen1 dataset contains two object classes and took a total of 39 hours to collect, with 22.63 hours used for training, 6.59 hours for validation, and 10.10 hours for testing. The 1Mpx dataset was acquired at a resolution of 1280×720 and annotated at a frequency of 60Hz. Covering three categories, the 1Mpx dataset is the largest of all open-source event camera-based datasets, and also features the most complex detection scenarios. The 1Mpx dataset contains 14.65 hours of recorded samples, of which 11.19 hours were used for training, 2.21 hours for validation, and 2.25 hours for testing.

[0106] This method achieved classification accuracy rates of 84.8%, 99.7%, and 96.7% on the Cifar10-DVS, N-MNIST, and N-CARS datasets for short-term event classification, respectively, and 97.9% on the DVS128Gesture dataset for long-term event classification, surpassing the state-of-the-art solutions. Furthermore, it maintained a low parameter count and computational complexity. On the Gen1 and 1Mpx object detection datasets, this method also outperformed all state-of-the-art solutions while maintaining a relatively fast runtime.

Claims

1. An event camera classification detection method based on time-space-polarity domain decoupling, characterized in that: The steps include: Step 1: Data collection and preprocessing based on event cameras: An event camera is used to collect asynchronous event streams and RGB images, and the event streams are divided into single event stream samples according to timestamps, thereby obtaining M event stream samples, where the i-th event stream sample is recorded as E i ,and Among them, x i,j ,y i,j ,t i,j ,p i,j Represents the i-th event stream sample E i The abscissa, ordinate, timestamp and polarity of the j-th event in N i Represents the i-th event stream sample E i The total number of events contained in , let the spatial range of a single event stream sample be H×W, where H and W are the maximum values of height and width respectively; According to the i-th event stream sample E i Find the RGB image with the closest time stamp t0 of the first event in the image, mark the information of the object in the RGB image and form a normalized label l i , thus forming the event camera training dataset Step 2: Build an event embedding module to convert asynchronous event streams into event representations: Step 2.1: Take the i-th event stream sample E i Timestamp in Discretize it into a sequence of integers in the range [0, K-1] in, is the rounding down operation, K is the number of time intervals after discretization, t end For N i The timestamp of the event; Step 2.2: Based on the i-th event stream sample E i The horizontal axis in vertical axis Get the i-th event stream sample E i The block position of And the i-th event stream sample E i The position of each pixel in the token block Where mod is the remainder operation, and P is the side length of the token block; Step 2.3: Use formula (1) to get the i-th event stream sample E i One-dimensional array a i : In formula (1), p i Represents the i-th event stream sample E i Polarity; Step 2.4, for a i After frequency statistics, matrix splicing and matrix reshaping operations are performed to obtain the i-th event representation Step 2.5: Use the grouped convolutional layer and the MLP layer to represent R from the i-th event i Extract features from the ith event token containing G groups of time-space-polarity features Among them, token i,k Indicates T i The middle dimension is the kth token of G·G, where G is the number of groups divided according to time and polarity, and C is the number of channels in each group, that is, the feature dimension of each group; Step 2.6: Batch select B event stream samples from the event camera training dataset X Follow steps 2.1 to 2.5 to convert XE into event tokens. Step 3: Construct a hierarchical group feature extraction network and use it to extract group features of event tokens XT based on time-space-polarity domain decoupling to obtain the output Among them, f l,b represents the b-th output of the l-th level; Step 4: Build classification module and detection module: Step 4.1: The classification module extracts the output of the Lth level in the hierarchical grouping feature extraction network. After the global average pooling operation, it is input into the fully connected layer for processing to obtain the normalized prediction labels of a batch of event stream samples XE Among them, p b,c Represents the bth event stream sample E among the B event stream samples XE selected in a batch b The corresponding predicted label is the probability of the cth category, Cls is the total number of categories; Step 4.2: The detection module will output F out Send it to the YOLOX detection layer for processing and output the target detection results Among them, P' b Represents the b-th event stream sample F b The target detection result, P' b ∈R N×(Cls+5) , N is the number of detection boxes in the target detection result, and each detection box contains Cls+5 prediction values, which are the rate of the target in the detection box being of various categories, the horizontal coordinate of the center point of the detection box, the vertical coordinate of the center point of the detection box, the width of the detection box, the length of the detection box and the confidence level; Detection results are improved by confidence screening and non-maximum suppression Filter to get the final test results Among them, pd b Represents the bth event stream sample E b The prediction results; N - is the number of remaining detection frames after screening; Step 5: Network training: Step 5.1: For classification tasks, use formula (2) to construct the cross entropy loss L CE : In formula (2), p b is the bth event stream sample E b The probability that the corresponding predicted label is correctly classified; Step 5.2: For the detection task, use formula (3) to construct the total loss function L: L=a·L Conf +b·L Cls +c·L Reg (3) In formula (3), L Reg Represents the regression loss, and is obtained from formula (4), L Conf represents the confidence loss and is expressed by p b Get pd b The confidence level in L is used to calculate Cls Denotes the category loss and is expressed by p b Get pd b The probability of the target in the detection box being correctly classified is used to calculate, and a, b, and c are the weights of the three losses; L Reg =-log(IoU(PD box ,GT)) (4) In formula (3), PD box is the horizontal coordinate of the center point of the detection box in the detection result PD, the vertical coordinate of the center point of the detection box, the width of the detection box, and the length of the detection box; GT is the horizontal coordinate of the center point of the corresponding detection box in the normalized label, the vertical coordinate of the center point of the detection box, the width of the detection box, and the length of the detection box; IoU(·) refers to the calculation of the intersection over union ratio of two sets of boxes; Step 5.3: Use the gradient descent method to train the event embedding module, hierarchical grouping feature extraction network, classification module and detection module, and calculate the loss function to update the network parameters until the loss converges, thereby obtaining a trained classification detection model for classifying and detecting event stream samples.

2. The event camera classification detection method based on time-space-polarity domain decoupling according to claim 1 is characterized in that: The first layer of the hierarchical grouping feature extraction network in step 3 includes a dual-residual event self-attention module, and each of the remaining layers includes a grouping feature aggregation module and a dual-residual event self-attention module; The dual residual event self-attention module is composed of a window operation layer, a spatial self-attention layer, a grouping self-attention layer, a merging operation layer, and a feature fusion layer in sequence; Step 3.1.1: When l=1, the window operation layer in the lth level performs windowing on the event token XT to obtain the event token after windowing at the lth level. Among them, S is the side length of the window, T l,w represents the event token in the w-th window at the l-th level, and token l,w,m The event token T of the wth window at the lth level l,w The mth token in ; Step 3.1.2: When l = 1, the spatial self-attention layer in the lth level pays attention to the token l,w,m The relative spatial position within the window is encoded to obtain the mth relative position bias bp of each window at the lth level l,m ∈[-S+1,S-1], thus obtaining the total relative position bias of the lth level Bias the total relative position by Bp l and the event token T in the w-th window l,w After adding, self-attention operation is performed to obtain the spatial features of the lth level Among them, WFS l,w The event token T of the wth window at the lth level l,w The extracted spatial features, and wfs l,w,m The event token T of the wth window at the lth level l,w The mth spatial feature extracted; Step 3.1.3, when l = 1, the grouped self-attention layer in the lth level will take the spatial feature FS of the lth level l Transpose to obtain the transposed spatial features Among them, WFS T l,w represents the spatial features of the w-th window after transposition at the l-th level, and wfs T l,w,n Represents the nth spatial feature after the transposition of the wth window at the lth level; The grouped self-attention layer in the lth level is wfs T l,w,n The relative group and channel relationships within the window are encoded to obtain the nth relative group bias bg for each window at the lth level l,n ∈[-G+1,G-1], thus obtaining the total relative group bias of the lth level Bias the total relative position by Bg l The spatial feature WFS after transposing the w-th window T l,w After adding, self-attention operation and transposition operation are performed to obtain the grouping features of the lth level Among them, WFG l,w Represents the spatial feature WFS after the transposition of the w-th window at the l-th level T l,w The extracted grouping features, and Among them, wfg l,w,n Represents the spatial feature WFS after the transposition of the w-th window at the l-th level T l,w The nth grouping feature extracted; Step 3.1.4, when l=1, the merging operation layer in the lth level combines the spatial features FS l and grouping feature FG l Merge them separately to get the space tokens at level l and grouping tokens Among them, ST l,b represents the bth spatial token at the lth level, and st l,b,k represents the kth token in the bth spatial token at the lth level, GT b represents the bth grouping token at the lth level, and gt l,b,k represents the kth token in the bth grouped token at the lth level; Step 3.1.5, when l = 1, the feature fusion layer in the lth level combines the spatial token XST of the lth level through double residual connection, MLP layer, and regularization layer l and grouping token XGT l Fusion is performed to obtain the time-space-polarity domain token of the lth level Among them, SGT l,b represents the bth time-space-polarity domain token at the lth level and is recorded as the bth output f at the lth level l,b ,and sgt l,b,k represents the kth token in the bth time-space-polarity domain token at the lth level; Step 3.1.6: When l=2, the grouping feature aggregation module in the lth level combines the time-space-polarity domain token XSGT of the l-1th level l-1 After processing by overlapping convolutional layers and regularization layers, we get the aggregated tokens at the lth level. Among them, AT l,b represents the bth aggregate token at the lth level, and at l,b,k represents the kth token in the bth aggregate token at the lth level, at l,b,k The dimension is at l,b,k The number of groups is Aggregate token XAT at level l l Input the double residual event self-attention module of the lth level, and follow the process of steps 3.1.1 to 3.1.5 to obtain the aggregated time-space-polarity domain token of the lth level Among them, VT l,b represents the bth aggregated time-space-polarity domain token at the lth level and is recorded as the bth output f at the lth level l,b ; Step 3.1.7, when l∈[3,L], the aggregated time-space-polarity domain token XVT of the l-1th layer l-1 After being input into the group feature aggregation module of the lth level for processing, it passes through the double residual event self-attention module of the lth level to obtain the aggregated time-space-polarity domain token of the lth level. Among them, VT l,b represents the bth aggregated time-space-polarity domain token at the lth level and is recorded as the bth output f at the lth level l,b ; and constitute the output of the hierarchical grouping feature extraction network in, f l,b The number of groups is 3. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the event camera classification detection method according to claim 1 or 2, and the processor is configured to execute the program stored in the memory.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the event camera classification detection method according to claim 1 or 2 are executed.

Images