Adaptive multi-scale aggregated data augmentation method and system for event-based cameras

Abstract

The application discloses an adaptive multi-scale aggregated data enhancement method and system for an event camera, adjusts the scale of an aggregated integral adaptively according to data conditions, forms a multi-scale time sequence integral strategy, thereby realizes simulation of different motion modes, generates different samples in a diversified motion scene from a single time sequence event data sample, and further exposes different motion features to a model for learning; in addition, the training result of a test model in a series of alternative scales guides the scale of formal training on the test model, and realizes scale adaptation for different distributed data sets. The method makes the network more robust to various motion modes, and can also improve the generalization ability of the sensor to different noises. Test results show that the method has greater improvement in performance compared with various other enhancement methods.

Description

Technical Field

[0001] This invention belongs to the technical field of visual learning and data augmentation of event data, and particularly relates to event camera data augmentation technology. It mainly relates to an adaptive multi-scale aggregation data augmentation method and system for event cameras. Background Technology

[0002] Recently, thanks to advancements in Dynamic Visual Sensors (DVS) and neuromorphic computing technologies, event-based learning has become a research hotspot. In event learning, existing research primarily focuses on network architecture design and task-specific network construction. Representative works include techniques such as Group Event Transformer (GET), Spikepoint, video deraining, and motion deblurring. However, the limited quantity of event data and the significant differences between different datasets constrain network performance, thus affecting the model's generalization ability. To overcome these challenges and utilize event data more effectively, researchers have proposed various learning strategies, such as unsupervised learning, self-supervised learning, pre-training, and transfer learning. Nevertheless, these strategies have several limitations; for example, they often rely on hard-to-obtain paired RGB data, and many strategies are not applicable to other tasks.

[0003] Data augmentation techniques play a crucial role in improving model generalization ability. Existing research has confirmed that data augmentation is a practical technique applicable to a variety of tasks. However, current research results in event data augmentation are relatively limited. Typical attempts include methods such as NDA, EventDrop, and EventMix. NDA attempts to directly apply traditional RGB data augmentation techniques, such as CutMix and flipping, to event data, but its results are not ideal due to insufficient consideration of the unique sparsity and spatiotemporal characteristics of event data. EventDrop increases the diversity of the original dataset by designing three random dropout strategies, while EventMix applies three-dimensional variations of Mixup and CutMix to event data. Although these methods consider spatial and temporal dimensions, their augmentation performance is still unsatisfactory and lacks stability. This is because they rely too heavily on randomness and prior assumptions, ignoring the uneven and diverse distribution of different event data. Furthermore, these methods only make preliminary use of the spatiotemporal information of events and fail to fully utilize the rich spatiotemporal relationships within the event stream. Therefore, they exhibit insufficient robustness when dealing with complex real-world scenarios. In particular, each method only uses a single time scale for temporal integration, which ignores the rich temporal information in the event stream data and greatly hinders the application of event cameras. Summary of the Invention

[0004] This invention addresses the significant challenges and design shortcomings of existing event information learning methods, providing an adaptive multi-scale aggregation data augmentation method and system for event cameras. By adjusting the scale of the aggregation integral, a multi-scale temporal integral is formed; different motion modes are simulated to generate samples from diverse motion scenarios from single event-based samples; and the training results of the test model at a series of alternative scales guide the final scale for formal training on the test model, achieving scale adaptation for datasets with different distributions. This invention makes the network more robust to various motion modes and also improves its generalization ability across sensor noise ranges.

[0005] To achieve the above objectives, the technical solution adopted by this invention is: an adaptive multi-scale temporal aggregation data augmentation method for event camera data, which adaptively selects and adjusts the integration scale when performing image aggregation integration on event data according to the distribution and spatiotemporal characteristics of the target data, and at the same time selects different temporal scales of different lengths for model training, learns data features, and improves the accuracy on downstream tasks.

[0006] As an improvement to the present invention, an adaptive multi-scale aggregation data augmentation method for event cameras includes the following steps:

[0007] S1, Obtain the dataset: Select the corresponding dataset according to the given task type; the task type includes, but is not limited to, action recognition and image classification.

[0008] S2, Select the network to be tested: The network is a Convolutional Spike Neural Network (CSNN) defined in ResNet-34 and Spikingjelly;

[0009] S3, Event Data Integration into Framed Data: This involves converting the event stream into frame-like data. Specifically, the event stream is pre-arranged according to timestamp order, and time integration is performed within the target time region.

[0010] E i =(x i y i p i , t i )

[0011]

[0012]

[0013]

[0014]

[0015] Where N is the total number of events in the event stream, (x i y i ) is event E i The generated coordinates, t i It is event E i The timestamp generated by p i It is event E i polarity; i start and i end For the start and end timestamps of the event frame; (x k y k ) is event E k The generated coordinates, t k It is event E k The timestamp generated by p k It is event E k The polarity of ; I is the indicator function; after integration, each event stream is converted into T event frames, j is the sequence number of the converted frame F, and the j-th frame is F. j Each event frame can be viewed as a 2-channel image with [W, H] resolution;

[0016] S4, Adaptive Integral Scale Search: Based on the distribution characteristics of the target dataset, a small parameter model with the same configuration as the target network structure in step S2 is used to search for the integral scale. The set of scales that performs best in the final task is identified as the candidate time scales for the final multi-scale aggregation.

[0017] S5, Multi-scale integral sample generation and temporal aggregation: Based on the integral candidate group scale confirmed in step S4, aggregate and integrate in the event stream to generate image frames of the corresponding scale, generate image samples of different scales, and complete data augmentation.

[0018] S6, Train and validate the augmentation effect: Train the augmented dataset and analyze and validate its actual performance in downstream tasks.

[0019] As an improvement of the present invention, step S4, adaptive scale search, mainly includes reducing the depth of the feature extractor according to the structure of the test model, constructing a small model, and then obtaining the performance of the small model on this dataset by searching and training the small model in the candidate time scales. This performance is used as an estimate of the performance of the original test model. The best performing candidate scales that are smaller than the cardinality scale and larger than the cardinality scale are selected as the short-term time scale and long-term time scale used for final training.

[0020] As another improvement of the present invention, step S5, multi-scale integral sample generation, mainly includes event frame aggregation using three different scales: the short-term time scale and the long-term time scale searched in S4, and the baseline scale. The specific aggregation method is as described in S3. Finally, the samples of different scales are shuffled and combined to form the training set after data augmentation, while the test set uses the cardinality scale for time aggregation.

[0021] To achieve the above objectives, the present invention also adopts the following technical solution: an adaptive multi-scale aggregation data augmentation system for event cameras, comprising a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the methods described above.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) This invention enhances the generalization ability of the model from the data side, and can be used in different datasets and different downstream tasks to achieve general enhancement.

[0024] (2) This method is inspired by our observation that motion speed determines the completeness of motion cues and the clarity of target boundaries within an event frame. For long-term timescale integration, more motion information is revealed, including motion trajectory and speed. For short-term timescale integration, more information about the object itself is revealed (e.g., contour and shape). Furthermore, frames with different integration scales contain varying degrees of noise, which is beneficial for feature extraction because it enhances diversity. The AMSTI technique enables event-based models to learn additional motion cues and spatial features.

[0025] (3) In view of the complex motion of objects in real-world scenarios, the enhancement method of the present invention simulates the motion patterns of objects at different motion speeds, thereby enhancing the robustness of the model to different motion speeds of objects in complex scenarios. Attached Figure Description

[0026] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0027] Figure 2 This is a schematic diagram illustrating the enhanced results of the present invention on the sample data;

[0028] Figure 3 This is a visual diagram illustrating the enhanced effects of the present invention and existing methods. Detailed Implementation

[0029] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0030] Example 1

[0031] Adaptive multi-scale temporal aggregation data augmentation for event cameras, such as Figure 1 As shown, it includes the following steps:

[0032] Step S1, Obtain the dataset: Select the corresponding dataset according to the given task (such as action recognition, image classification, etc.).

[0033] Step S2, Select the network to be tested: In this embodiment, ResNet-34 and the Convolutional Spike Neural Network (CSNN) defined in Spikingjelly are used as the test networks;

[0034] Step S3: Integrate event data into frame data:

[0035] Existing Artificial Neural Networks (ANNs) primarily process RGB frames and cannot directly handle sparse event streams. For SNNs, directly inputting frames without pre-encoding has become a widely adopted strategy in deep spiking neural networks. Therefore, to apply existing powerful ANN and SNN models to event vision and extract discriminative spatial cues, the mainstream solution is to convert event streams into frame-like data.

[0036] E i =(x i y i p i , t i )

[0037] Where N is the total number of events in the event stream, (x i y i ) is event E i The generated coordinates, t i It is a timestamp indicating when the event occurred, p i It is polar;

[0038] The event stream is pre-arranged in chronological order according to timestamps. For the integrated integral, the event stream is evenly divided into T slices. Let F... j This represents the frame generated from the j-th event slice.

[0039] First, define i start and i end For the start and end timestamps of the event frame:

[0040]

[0041]

[0042] Then, perform time integration within the target time region:

[0043]

[0044]

[0045] Where I is the indicator function and N is the total number of events in the event stream.

[0046] Where N is the total number of events in the event stream, (x i y i ) is event E i The generated coordinates, t i It is a timestamp indicating when the event occurred, p i It is polarity; i start and i end Here, k represents the start and end timestamps of the event frame, and k is the timestamp sequence number between the two. k y k ) is event E k The generated coordinates, t k It is a timestamp indicating when the event occurred, p k It is polarity, both are events E. k The attributes; I is an indicator function, representing different values ​​when different conditions are met; after integration, each event stream is converted into T event frames, j is the sequence number of the converted frame F, and the j-th frame is F. j Each event frame can be viewed as a 2-channel image with [W, H] resolution;

[0047] Step S4, Adaptive Integral Scale Search:

[0048] Experiments revealed that significant performance enhancement can be achieved simply by setting the long-term and short-term time scales to 2x and 0.5x, respectively, without requiring careful tuning. However, this invention also proposes an alternative adaptive dominance scale search method. This method primarily involves reducing the depth of the feature extractor, constructing a small model, and then obtaining the model's performance on the dataset by searching and training it across candidate time scales. This performance is used as an estimate of the original test model's performance. The best-performing candidate scales (smaller than cardinality and larger than cardinality) are selected as the final short-term and long-term time scales used for training. Event frames are aggregated at three different scales (these two scales and the baseline scale) and combined in a shuffled order to form the training set, while the test set uses the cardinality scale for temporal aggregation.

[0049] Step S5: Multi-scale sample generation and temporal aggregation

[0050] This embodiment employs multi-scale time integration, enabling the neural network to learn different types of patterns. It also increases the diversity of motion speeds to mitigate the negative impacts of varying movement speeds, making the network more robust. In implementation, a speed-aware strategy is applied: simultaneously selecting the cardinality scale and the short-term and long-term time scales obtained in S4, aggregating event frames as shown in step S3 at all three scales, and then mixing and concatenating them in a shuffled order to form the training set. The test set uses the cardinality scale for time aggregation. This enhancement strategy is highly versatile and can be applied to all datasets.

[0051] Step S6: Train and verify the enhancement effect: Train the enhanced dataset and analyze and verify its actual performance in downstream tasks. Figure 2 The visualizations show the enhanced data. The top image shows the visualization results at the cardinal scale, while the bottom images show the integration results at the short-term and long-term time scales, respectively. It can be seen that the long-term integration reveals more motion information, including trajectory and speed. The short-term integration reveals more information about the object itself, such as its outline and shape.

[0052] Figure 3 The diagram illustrates the enhancement method of the present invention, the "baseline method", and the effect of the existing enhancement method EventDrop. Figure 3 The visualization results show that the AMSTI technique in this invention's enhancement method enables the event-based model to learn additional motion cues and spatial features (two images on the right), thus allowing the network to better extract effective features. Other methods, such as EventDrop, do not significantly improve data diversity; instead, they lose the core features of the data (second image from the left).

[0053] This invention establishes an adaptive multi-temporal-scale temporal integration strategy to simulate different motion patterns. It generates diverse samples from single-temporal-event data samples, allowing different motion features to be exposed to the model for learning. Furthermore, by testing the training results of a small model on a series of alternative scales, it guides the final training scale on the test model, achieving scale adaptation for datasets with different distributions. This method makes the network more robust to various motion patterns and improves the sensor's generalization ability to different noise levels. Test results show that this method significantly improves performance compared to other existing enhancement methods.

[0054] Test case

[0055] Performance testing: To verify the effectiveness of the pre-trained parameters of this invention, this test case uses two different test models to perform performance tests on two tasks: action recognition and image classification, and compares them with several advanced augmentation methods.

[0056] The enhancement methods used for comparison are as follows:

[0057] Baseline: No data augmentation methods are used;

[0058] Event Drop: Current state-of-the-art augmentation methods include three random dropout strategies designed to improve the diversity of the original dataset;

[0059] NDA: Current advanced augmentation methods apply data augmentation methods (such as CutMix, flip, etc.) in the RGB modality to event data;

[0060] In the test, we used the accuracy rate (%) of the task as the evaluation metric. The higher the difference from the baseline model, the better the enhancement effect.

[0061] The comparison results of the method of this invention with two other existing methods on different task datasets and different network models are shown in the table below:

[0062]

[0063] As shown in the table above, the augmentation method of this invention significantly outperforms the baseline method without augmentation and demonstrates performance exceeding that of existing state-of-the-art augmentation methods. The table compares this invention with other state-of-the-art event augmentation methods, covering two backbone architectures and using the same hyperparameters for training. This invention consistently achieves significant performance improvements, demonstrating its effectiveness in enriching data diversity and reducing overfitting.

[0064] Compared to other advanced data augmentation methods, this invention significantly enhances the diversity of sample motion velocities. The AMSTI technique enables event-based models to learn additional motion cues and spatial features. This provides the model with the ability to generalize across different scenarios, including those involving objects moving at varying speeds.

[0065] Experimental results on artificial neural networks (ANNs) and spiking neural networks (SNNs) demonstrate that our proposed method comprehensively and efficiently enhances spatiotemporal diversity. These improvements significantly enhance the accuracy and generalization ability of the model.

[0066] Furthermore, the experimental results demonstrate that the data augmentation effect of the multi-scale temporal aggregation strategy proposed in this invention does not originate from a single dominant scale, but rather from a multi-scale set of multiple time scales. Specific experimental results are shown in the table below:

[0067]

[0068] We argue that poor performance at short-term scales is due to a lack of object information at long-term scales, making it difficult for the model to accurately learn motion semantics. Therefore, our MSTI method combines short-term and long-term time scales to complementarily utilize multi-scale temporal cues, resulting in significant improvements.

[0069] In summary, this invention proposes a multi-scale temporal aggregation data augmentation method for event camera data. Since motion velocity determines the completeness of motion cues and the clarity of target boundaries after event data is aggregated into event frames, adjusting the aggregation scale enriches the diversity of motion velocities, forming a multi-scale temporal integral. By applying the multi-scale augmentation strategy, we effectively simulate different motion modes, enabling us to generate samples from diverse motion scenes using a single event-based sample. For different datasets, an adaptive scale determination method is proposed. The training results of a small test model at a series of alternative scales guide the final training scale on the test model, achieving scale adaptation for different distributed datasets. As the integration scale changes, features such as edges, textures, and motion also change, thereby enhancing diversity and making the network more robust to various motion modes. Furthermore, the adaptive multi-scale temporal aggregation data augmentation technique can also improve the generalization ability across sensor noise spans, as frames generated from different integration scales have diverse noise levels.

[0070] An adaptive multi-scale temporal integration (AMTI) technique is employed to enhance the diversity of motion velocities. Inspired by our observations of motion velocity variations, this method determines the completeness of motion cues and the clarity of object boundaries within an event frame. Longer timescale integration reveals more motion information, including trajectory and speed. Shorter timescale integration reveals more information about the object itself (e.g., contour and shape). Furthermore, frames with different integration scales contain varying levels of noise, which is beneficial for feature extraction due to the increased diversity. AMTI enables event-based models to learn additional motion cues and spatial features, thereby improving the model's generalization ability across different scenarios, including those involving objects moving at varying speeds.

[0071] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. An adaptive multi-scale aggregation data augmentation method for event cameras, characterized by: The integration scale for image aggregation integration of event data is adaptively selected and adjusted based on the target data distribution and spatiotemporal characteristics. Simultaneously, different temporal scales of varying lengths are selected for model training to learn data features and improve accuracy on downstream tasks. This includes at least the following steps: S1, Obtain the target dataset: Select the corresponding dataset according to the given task type; the task type includes, but is not limited to, action recognition, image classification, and semantic segmentation; S2, Select the target network to be tested: The target network adopts the convolutional neural network ResNet-34 and the spiking neural network CSNN defined in spikingjelly. The CSNN consists of 5 convolutional layers and 3 fully connected layers. S3, Event Data Integration into Image Frame Data: This involves converting the event stream into image frame data, specifically by pre-arranging the event stream in chronological order by timestamps and performing time integration within the target time region. ； in, It is the total number of events in the event stream. These are the coordinates where the event occurred. It is an event The timestamp that generates the time, It is an event polarity; These are the start and end timestamps of the event frame; These are the coordinates where the event occurred. It is an event The timestamp that generates the time, It is an event polarity; It is an indicator function; each event stream is converted to... one event frame It is the converted frame. The serial number, the Each frame is Each event frame is considered to have Two-channel image with high resolution; S4, Adaptive Integral Scale Search: Based on the distribution characteristics of the target dataset, the model with the smaller number of parameters under the same configuration as the target network structure in step S2 is used to search for the integral scale, and the set of scales that performs best in the final task is identified as the candidate time scale for the final multi-scale aggregation. S5, Multi-scale integral sample generation and temporal aggregation: Based on the integral candidate group scale confirmed in step S4, aggregate and integrate in the event stream to generate image frames of the corresponding scale, generate image samples of different scales, and complete data augmentation. S6, Train and validate the augmentation effect: Train the augmented dataset and analyze and validate its actual performance in downstream tasks.

2. The adaptive multi-scale aggregation data augmentation method for event cameras as described in claim 1, characterized in that: In step S4, adaptive scale search, the depth of the feature extractor is reduced according to the structure of the test model to construct a small model. The performance of the small model on this dataset is obtained by searching and training the small model in the alternative time scales. This performance is used as an estimate of the performance of the original test model. The best performing alternative time scales that are smaller than the cardinality scale and those that are larger than the cardinality scale are selected as the short-term and long-term time scales used for final training.

3. The adaptive multi-scale aggregation data augmentation method for event cameras as described in claim 2, characterized in that: In step S5, multi-scale integral sample generation, the short-term time scale, long-term time scale, and cardinality scale searched in step S4 are used to aggregate event frames. The samples of different scales are shuffled and combined to form the training set after data augmentation. The test set is aggregated using the cardinality scale.

4. An adaptive multi-scale temporal aggregation data augmentation system for event camera data, comprising a computer program, characterized in that: When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-3 above.

Images