Event camera-oriented adaptive multi-scale aggregated data enhancement method and system

By using an adaptive multi-scale aggregation data augmentation method, the integral scale of event data is adjusted to generate multi-scale temporal integrals, which solves the problem of insufficient model generalization ability in existing methods and achieves robust recognition and feature extraction of object motion in complex scenes.

CN118982731AActive Publication Date: 2024-11-19SOUTHEAST UNIV
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202411148854.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-21
Publication Date
2024-11-19
Estimated Expiration
2044-08-21

AI Technical Summary

Technical Problem

Existing event data augmentation methods suffer from insufficient model generalization ability in event camera applications, especially poor robustness in complex real-world scenarios, and fail to fully utilize the spatiotemporal relationships and diversity of event data.

Method used

An adaptive multi-scale aggregation data augmentation method is adopted. By adjusting the scale of the aggregation integral, multi-scale time series integrals are generated to simulate different motion modes. Data augmentation is performed by combining short-term and long-term time scales to improve the robustness of the model to different motion modes and the generalization ability of sensor noise.

Benefits of technology

It enhances the model's generalization ability across different datasets and downstream tasks, improves robustness to object motion speed in complex scenes, and enhances the diversity and accuracy of feature extraction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982731A_ABST
    Figure CN118982731A_ABST
Patent Text Reader

Abstract

The invention discloses a self-adaptive multi-scale aggregated data enhancement method and system for an event camera, and the method achieves the self-adaptive adjustment of the scale of an aggregated integral according to the data condition, forms a multi-scale time sequence integral strategy, achieves the simulation of different motion modes, and achieves the self-adaptive multi-scale aggregated data enhancement. Different samples in diversified motion scenes are generated from a single time sequence event data sample, so that different motion features can be exposed to a model for learning; in addition, a final formal training scale on the test model is guided through a training result of the test model in a series of alternative scales, and scale self-adaption to different distributed data sets is achieved. According to the method, the network is more stable to various motion modes, and the generalization ability of the sensor to different noises can be improved. Test results show that compared with various existing enhancement methods, the method has the advantage that the performance is greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of visual learning and data enhancement of event data, and in particular to event camera data enhancement technology, and mainly relates to an adaptive multi-scale aggregation data enhancement method and system for event cameras. Background Art

[0002] Recently, thanks to the progress of dynamic vision sensors (DVS) and neuromorphic computing technology, the field of learning based on event data has become a research hotspot. In the field of event learning, existing research mainly focuses on the design of network architecture and the construction of networks for specific tasks. Among them, representative works include Group Event Transformer (GET), Spikepoint, video deraining and motion deblurring. However, the limited amount of event data and the significant differences between different datasets have restricted the performance of the network, which in turn affects the generalization ability of the model. In order to overcome these challenges and make more effective use of event data, researchers have proposed a variety of learning strategies, such as unsupervised learning, self-supervised learning, pre-training and transfer learning. Nevertheless, these strategies have many limitations. For example, they often rely on paired RGB data that is difficult to obtain, and many strategies cannot be applied to other tasks.

[0003] Data augmentation technology plays a vital role in improving the generalization ability of models. Existing studies have confirmed that data augmentation is a practical technology applicable to a variety of tasks. However, in terms of event data augmentation, current research results are relatively limited. Typical attempts include methods such as NDA, EventDrop, and EventMix. NDA attempts to apply traditional RGB data augmentation techniques, such as CutMix and Flip, directly to event data, but the effect is not ideal because it does not fully consider the unique sparsity and spatiotemporal characteristics of event data. EventDrop increases the diversity of the original dataset by designing three random dropout strategies, while EventMix applies three-dimensional variants of Mixup and CutMix to event data. Although these methods take into account spatial and temporal dimensions, their enhancement performance is still unsatisfactory and lacks stability. This is because they rely too much on randomness and prior assumptions and ignore the uneven and diverse distribution of different event data. In addition, these methods only make preliminary use of the spatiotemporal information of events and fail to fully utilize the rich spatiotemporal relationships within the event stream. Therefore, they show insufficient robustness when dealing with complex real-world scenarios. In particular, each method only uses a single time scale for temporal integration, which ignores the rich temporal information in the event stream data and greatly hinders the application of event cameras. Summary of the invention

[0004] The present invention is aimed at the major challenges in the existing event information learning field and the design deficiencies in event enhancement methods, and provides an adaptive multi-scale aggregation data enhancement method and system for event cameras. By adjusting the scale of the aggregation integral, a multi-scale temporal integral is formed; different motion modes are simulated to generate samples in a variety of motion scenes from a single event-based sample; the training results of the test model in a series of alternative scales guide the scale of the final formal training on the test model, and achieve scale adaptation for different distribution data sets. The method of the present invention makes the network more robust to various motion modes and can also improve the generalization ability of the sensor noise span.

[0005] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: an adaptive multi-scale time series aggregation data enhancement method for event camera data, which adaptively selects and adjusts the integration scale when performing image aggregation integration on event data according to the target data distribution and spatiotemporal characteristics, and simultaneously selects time series scales of different lengths for model training, learns data features, and improves the accuracy of downstream tasks.

[0006] As an improvement of the present invention, an adaptive multi-scale aggregation data enhancement method for event cameras includes the following steps:

[0007] S1, obtaining a data set: selecting a corresponding data set according to a given task type; the task type is not limited to action recognition and image classification;

[0008] S2, select a network to be tested: the network uses ResNet-34 and the Convolution Spike Neural Network (CSNN) defined in spikingjelly;

[0009] S3, event data integration into frame data: convert the event stream into frame-like data, specifically: pre-arrange the event stream in the order of timestamps, and perform time integration in the target time region:

[0010] E i =(x i ,y i , p i , t i )

[0011]

[0012]

[0013]

[0014]

[0015] Where N is the total number of events in the event stream, (x i ,y i ) is event E i The resulting coordinates, t i It is event E i Timestamp of the time when it was generated, p i It is event E i The polarity of start and i end are the start and end timestamps of the event frame; (x k ,y k ) is event E k The resulting coordinates, t k It is event E k Timestamp of the time when it was generated, p k It is event E k The polarity of; I is the indicator function; after integration, each event stream is converted into T event frames, j is the sequence number of the frame F after conversion, and the jth frame is F j , each event frame can be regarded as a 2-channel image with [W, H] resolution;

[0016] S4, adaptive integral scale search: Based on the distribution characteristics of the target data set, a small parameter model with the same configuration as the target network structure in step S2 is used to search for integral scales, and a set of scales with the best performance on the final task is identified as the candidate time scale for the final multi-scale aggregation;

[0017] S5, multi-scale integral sample generation and time aggregation: according to the integral candidate group scale confirmed in step S4, aggregate the integral in the event stream, generate picture frames of corresponding scales, generate image samples of different scales, and complete data enhancement;

[0018] S6, train and verify the enhancement effect: train the enhanced dataset and analyze and verify its actual performance in downstream tasks.

[0019] As an improvement of the present invention, the adaptive scale search in step S4 mainly includes reducing the depth of the feature extractor according to the structure of the test model, building a small model, and then obtaining the performance of the small model on this data set by allowing the small model to search and train in alternative time scales, and using this performance as an estimate of the performance of the original test model, and selecting the best performing alternative scales smaller than the cardinality scale and the best performing alternative scales larger than the cardinality scale as the short-term time scale and long-term time scale used in the final training.

[0020] As another improvement of the present invention, the multi-scale integral sample generation in step S5 mainly includes using the short-term time scale and the long-term time scale searched in S4 and the base scale to aggregate event frames at three different scales, and the specific aggregation method is as described in S3. Finally, the samples of different scales are combined in a shuffled order as a training set after data enhancement, and the test set uses the base scale for time aggregation.

[0021] In order to achieve the above-mentioned purpose, the present invention also adopts a technical solution: an adaptive multi-scale aggregation data enhancement system for event cameras, including a computer program, which implements the steps of any of the above-mentioned methods when executed by a processor.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) The present invention enhances the generalization ability of the model from the data side and can be used in different data sets and different downstream tasks to achieve universal enhancement.

[0024] (2) This method is inspired by our observation that motion speed determines the completeness of motion cues and the clarity of object boundaries within the event frame. For integration over long time scales, more motion information is revealed, including motion trajectory, moving speed, etc. For integration over short time scales, more information about the object itself (e.g., contour, shape) is revealed. In addition, frames with different integration scales contain different degrees of noise, which is beneficial for feature extraction because diversity is improved. The AMSTI technique enables event-based models to learn additional motion cues and spatial features.

[0025] (3) In view of the complex motion of objects in real scenes, the enhancement method of the present invention simulates the motion patterns of objects at different motion speeds, thereby enhancing the robustness of the model to the different motion speeds of objects in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is a flow chart of the steps of the method of the present invention;

[0027] Figure 2 is a schematic diagram of the enhancement result of the sample data according to the present invention;

[0028] Figure 3 It is a visual schematic diagram of the enhancement effect of the present invention and the existing method. DETAILED DESCRIPTION

[0029] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0030] Example 1

[0031] Adaptive multi-scale temporal aggregation data enhancement for event cameras, such as Figure 1 As shown, the following steps are included:

[0032] Step S1, obtaining a data set: selecting a corresponding data set according to a given task (such as action recognition, image classification, etc.).

[0033] Step S2, selecting a network to be tested: In this embodiment, ResNet-34 and the Convolution Spike Neural Network (CSNN) defined in spikingjelly are used as the network to be tested;

[0034] Step S3, integrating the event data into frame data:

[0035] Existing artificial neural networks (ANNs) mainly process RGB frames and cannot directly process sparse event streams. For SNNs, directly inputting frames without a pre-encoding process has become a widely adopted strategy in deep spiking neural networks. Therefore, in order to apply existing powerful ANN and SNN models to event vision and extract discriminative spatial cues, the mainstream solution is to convert event streams into frame-like data.

[0036] E i =(x i ,y i , p i , t i )

[0037] Where N is the total number of events in the event stream, (x i ,y i ) is event E i The resulting coordinates, t i is the timestamp indicating the time when the event occurred, p i It is polarity;

[0038] Arrange the event stream in the order of timestamps in advance, and divide the event stream into T slices evenly for integration. Let F j represents the frame generated from the j-th event slice.

[0039] First, define i start and i end The start and end timestamps of the event frame:

[0040]

[0041]

[0042] Then, perform a time integration over the target time region:

[0043]

[0044]

[0045] Where I is the indicator function and N is the total number of events in the event stream.

[0046] Where N is the total number of events in the event stream, (x i ,y i ) is event E i The resulting coordinates, t i is the timestamp indicating the time when the event occurred, p i is polarity; i start and i end are the start and end timestamps of the event frame, k is the timestamp sequence number between the two, (x k ,y k ) is event E k The resulting coordinates, t k is the timestamp indicating the time when the event occurred, p k It is polarity, it is event E k ; I is the indicator function, which takes different values ​​when different conditions are met; after integration, each event stream is converted into T event frames, j is the sequence number of the converted frame F, and the jth frame is F j , each event frame can be regarded as a 2-channel image with [W, H] resolution;

[0047] Step S4, adaptive integral scale search:

[0048] In the experiment, it was found that by simply setting the enhanced long-term time scale and short-term time scale to 2 times and 0.5 times the scale, good enhancement performance can be achieved without careful adjustment. However, the present invention also proposes another adaptive dominant scale search method, which mainly includes reducing the depth of the feature extractor, building a small model, and then obtaining the performance of the small model on this data set by allowing the small model to search and train in the alternative time scale, using this performance as an estimate of the performance of the original test model, and selecting the best performing alternative scales smaller than the cardinality scale and the best performing alternative scales larger than the cardinality scale as the short-term time scale and long-term time scale used in the final training. These two time scales and the benchmark scale total three different scales for event frame aggregation, and the order is disrupted and combined as a training set, while the test set uses the cardinality scale for time aggregation.

[0049] Step S5, multi-scale sample generation and temporal aggregation

[0050] This embodiment designs a multi-scale time integration to allow the neural network to learn different types of patterns. The diversity of movement speeds can also be increased to eliminate the negative effects of different movement speeds and make the network more robust. In the implementation, a speed-aware strategy is applied: the cardinality scale and the short-term time scale and long-term time scale obtained in S4 are selected at the same time, and the three different scales are all subjected to the event frame aggregation shown in step S3, and the order is shuffled and mixed as the training set, and the test set uses the cardinality scale for time aggregation. This enhancement strategy is very general and can be applied to all data sets.

[0051] Step S6, training and verifying the enhancement effect: training the enhanced data set and analyzing and verifying its actual performance in downstream tasks. Figure 2 The visualization diagram of the enhanced data is shown. The upper part is the visualization result of the cardinality scale, and the lower part is the visualization diagram of the integration results of the short-term time scale and the long-term time scale. It can be seen that for the integration of the long-term time scale, it reveals more motion information, including motion trajectory, moving speed, etc. For the integration of the short-term time scale, it reveals more information about the object itself, such as the outline and shape of the object.

[0052] Figure 3 A schematic diagram showing the effects of the enhanced method of the present invention, the “baseline method” and the existing enhanced method EventDrop is shown. Figure 3 From the visualization analysis, we can see that the AMSTI technology of the enhanced method of the present invention enables the event-based model to learn additional motion clues and spatial features (two pictures on the right), so that the network can better extract effective features. Other methods such as EventDrop do not significantly improve the diversity of data, but lose the core features of the data (2 pictures on the left).

[0053] The present invention forms an adaptive multi-time-scale temporal integration strategy, thereby simulating different motion modes, generating different samples in diversified motion scenes from a single temporal event data sample, and thus exposing different motion features to the model for learning; in addition, by testing the training results of a small model in a series of alternative scales, the scale of the final formal training on the test model is guided, and scale adaptation for different distributed data sets is achieved. The method of the present invention makes the network more robust to various motion modes, and can also improve the generalization ability of the sensor to different noises. Test results show that the method of the present invention has a significant improvement in performance compared to various other existing enhancement methods.

[0054] Test Case

[0055] Performance test: To verify the effectiveness of the pre-training parameters of the present invention, this test example uses two different test models to perform performance tests on two tasks, action recognition and image classification, and compares them with multiple advanced enhancement methods.

[0056] The enhancements used for comparison are as follows:

[0057] Baseline: No data augmentation method is used;

[0058] Event Drop: Based on the current advanced enhancement methods, three random dropout strategies are designed to improve the diversity of the original dataset;

[0059] NDA: Currently available advanced enhancement methods apply data enhancement methods in RGB modality (such as CutMix, Flip, etc.) to event data;

[0060] In the test, we use the task accuracy (%) as the evaluation indicator. The higher the difference with the baseline model, the better the enhancement effect.

[0061] The comparison results between the method of the present invention and other two existing methods on different task data sets and different network models are shown in the following table:

[0062]

[0063] As can be seen from the table above, the enhancement method of the present invention far exceeds the effect of the baseline method without enhancement, and shows performance that exceeds the existing advanced enhancement methods. The table compares the present invention with other state-of-the-art event enhancement methods, covering two backbone architectures and using the same hyperparameters for training. The present invention achieves significant performance improvements, demonstrating its effectiveness in enriching data diversity and reducing overfitting.

[0064] Compared with other advanced data enhancement methods, the present invention greatly enhances the diversity of sample motion speeds. AMSTI technology enables event-based models to learn additional motion cues and spatial features. This provides the model with the ability to generalize in different scenarios, including objects moving at different speeds.

[0065] Experimental results on artificial neural networks (ANN) and spiking neural networks (SNN) show that our proposed method comprehensively and efficiently enhances spatiotemporal diversity. These improvements significantly improve the accuracy and generalization ability of the model.

[0066] In addition, the experimental results also prove that the data enhancement effect of the multi-scale time series aggregation strategy proposed in the present invention does not come from a single dominant scale, but from a multi-scale set of multiple time scales. The specific experimental results are shown in the following table:

[0067]

[0068] We believe that the poor performance on short-term scales is due to the lack of object information on long-term scales, which makes it difficult for the model to accurately learn motion semantics. Therefore, our MSTI method combines short-term and long-term time scales to exploit multi-scale temporal cues in a complementary manner, resulting in significant improvements.

[0069] In summary, the present invention proposes a multi-scale temporal aggregation data enhancement method for event camera data. Since the motion speed determines the integrity of motion clues and the clarity of target boundaries after the event data is integrated and aggregated into event frames, by adjusting the scale of the aggregated integral, we can enrich the diversity of motion speeds and form a multi-scale temporal integral. By applying a multi-scale enhancement strategy, we actually simulate different motion modes, which enables us to generate samples in diverse motion scenes using a single event-based sample. For different data sets, a method for adaptively determining the scale size is proposed. The training results of a small test model in a series of alternative scales guide us to finally perform formal training on the test model to achieve scale adaptation for different distribution data sets. As the integration scale changes, features such as edges, textures, and motion also change, thereby enhancing diversity and making the network more robust to various motion modes. In addition, the adaptive multi-scale temporal aggregation data enhancement technology can also improve the generalization ability of the sensor noise span, because the generated frames from different integration scales have diverse noise levels.

[0070] Adaptive Multi-Scale Time Integration Technique to Enhance the Diversity of Motion Speed. This method is inspired by our observation of the variation of motion speed, which determines the completeness of motion cues and the clarity of object boundaries within the event frame. For the integration of long-term time scales, more motion information is revealed, including motion trajectory, movement speed, etc. For the integration of short-term time scales, more information about the object itself (e.g., contour, shape) is revealed. In addition, frames with different integration scales contain different degrees of noise, which is beneficial for feature extraction because the diversity is improved. The AMSTI technique enables the event-based model to learn additional motion cues and spatial features, thereby improving the generalization ability of the model in different scenarios that include objects moving at different speeds.

[0071] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. Adaptive multi-scale aggregation data enhancement method for event cameras, Features: According to the target data distribution and spatiotemporal characteristics, the integration scale for image aggregation integration of event data is adaptively selected and adjusted. At the same time, different time series scales are selected for model training to learn data features and improve the accuracy of downstream tasks.

2. The adaptive multi-scale aggregation data enhancement method for event cameras according to claim 1, Features: At least the following steps are included: S1, obtaining a target data set: selecting a corresponding data set according to a given task type; the task type is not limited to action recognition, image classification, and semantic segmentation; S2, select the target network to be tested: the target network adopts the convolutional neural network ResNet-34 and the spiking neural network (CSNN) defined in spikingjelly, and the CSNN consists of 5 convolutional layers and 3 fully connected layers; S3, event data integration into image frame data: convert the event stream into data similar to image frames, specifically: pre-arrange the event stream in the order of timestamps, and perform time integration in the target time region: E; =((*, yuan, p, t) Where N is the total number of events in the event stream, (x i ,y i ) is event E i The resulting coordinates, t i It is event E i Timestamp of the time when it was generated, p i It is event E i The polarity of start and i end are the start and end timestamps of the event frame; (x k ,y k ) is event E k The resulting coordinates, t k It is event E k Timestamp of the time when it was generated, p k It is event E k The polarity of; I is the indicator function; each event stream is converted into T event frames, j is the sequence number of the frame F after conversion, and the jth frame is F j , each event frame can be regarded as a 2-channel image with [W, H] resolution; S4, adaptive integral scale search: Based on the distribution characteristics of the target data set, a small parameter model with the same configuration as the target network structure in step S2 is used to search for integral scales, and a set of scales with the best performance on the final task is identified as the candidate time scale for the final multi-scale aggregation; S5, multi-scale integral sample generation and time aggregation: according to the integral candidate group scale confirmed in step S4, aggregate the integral in the event stream, generate picture frames of corresponding scales, generate image samples of different scales, and complete data enhancement; S6, train and verify the enhancement effect: train the enhanced dataset and analyze and verify its actual performance in downstream tasks.

3. The adaptive multi-scale aggregation data enhancement method for event cameras according to claim 2, Features: In the step S4 adaptive scale search, the depth of the feature extractor is reduced according to the structure of the test model, and a small model is constructed. The performance of the small model on this data set is obtained by searching and training the small model in the alternative time scale. This performance is used as an estimate of the performance of the original test model, and the best performing alternative scales smaller than the cardinality scale and the best performing alternative scales larger than the cardinality scale are selected as the short-term time scale and the long-term time scale used in the final training.

4. The adaptive multi-scale aggregation data enhancement method for event cameras according to claim 3, Features: In the multi-scale integral sample generation of step S5, the short-term time scale, long-term time scale and reference scale searched in step S4 are used to aggregate event frames, and samples of different scales are combined in a shuffled order as a training set after data enhancement, and the test set is time aggregated using the cardinality scale.

5. Adaptive multi-scale temporal aggregation data augmentation system for event camera data, including computer programs, Features: When the computer program is executed by a processor, the steps of any of the above methods are implemented.

Citation Information

Patent Citations

  • Remote sensing image cloud detection method based on feature multi-scale perception and adaptive aggregation

    CN114494821A

  • Fog penetrating target detection method and device based on multi-sensor fusion

    CN114694011A

  • Automatic learning enhancement method and system for asynchronous event data

    CN114723010A

  • Event camera data simulation method and device based on standardized flow

    CN116309356A

  • Multi-scale target detection method based on semi-supervision and sample adaptive data enhancement

    CN117830624A