An efficient target detection method based on event stream sparse characteristics

Abstract

The application provides a kind of high-efficiency target detection method based on event stream sparse characteristics, comprising: constructing event-based agile network (EvKeenDet), introducing event-based spatiotemporal target copy-paste strategy (Event GT Copy-Paste);Put the autonomous driving simulation dataset into the event-based agile network, realize event-based target detection.The event-based agile network constructed by the application expands and corrodes the shallow aggregation in long short-term memory network and enhances feature representation, effectively processes stream sparse event data, and generates complete targets, while solving the problem of shallow information aggregation and representation ambiguity in event camera-based target detection, and introducing event-based spatiotemporal target copy-paste strategy to improve the robustness of the network to stream sparse data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular, and especially relates to an efficient target detection method based on event stream sparse characteristics. BACKGROUND

[0002] Target detection based on frame cameras has some limitations, such as poor performance in high-speed motion, overexposure, and low light challenge scenes. In addition, although target detection based on event cameras has been proposed, there are still robustness and efficiency problems caused by stream sparse data.

[0003] When the luminance of a pixel changes by a threshold C on a logarithmic scale, an event is generated. A positive change in luminance will generate a positive event, and vice versa. Therefore, event cameras have the characteristics of asynchrony, high temporal resolution, and high dynamic range. Event data can be considered as residual data, which can be analogized as the difference between two scene image grayscale frames with time error in microseconds. This reveals the fundamental difference between event data and frame data.

[0004] The current methods RED / DMANet / RVT generally use the previous general frame-based model, and the mAP@0.5 performance indicators on the 1Megapixel Automotive Detection Datastet real event data set are 39.7% / 44.4% / 47.4%, respectively. The existing method processes stream sparse event data by compressing event data into pseudo images and using them like frame data, ignoring the stream sparse characteristics of event data, resulting in missing target outlines and waste of network calculation. SUMMARY

[0005] In view of the above-mentioned deficiencies of the traditional frame-based target detection method in processing stream sparse events, the robustness and efficiency problems caused by stream sparse data, an efficient target detection method based on event stream sparse characteristics is provided. The present application mainly uses the constructed event-based agile network to solve the problems of shallow information aggregation and fuzzy representation in event camera-based target detection, and introduces the Event GT Paste data enhancement mechanism to improve the robustness of the network to stream sparse data.

[0006] The technical means adopted by the present application are as follows:

[0007] An efficient target detection method based on event stream sparse characteristics, comprising:

[0008] Constructing an event-based agile network;

[0009] Based on the constructed event-based agile network, an event-based spatio-temporal target copy-paste strategy Event GT Copy-Paste is introduced;

[0010] The autonomous driving simulation dataset is put into the event-based agile network to realize event-based target detection.

[0011] Further, the core module of the constructed event-based agile network is an inflation-erosion long short-term memory module, which aggregates the information in the pseudo image composed of event data in a short time by introducing inflation-erosion operation, including an inflator module and an eroder module, wherein:

[0012] The inflator module uses a large receptive field of cavity local attention mechanism to quickly aggregate the time series information, and realizes the fast and efficient inflation of the neighborhood information.

[0013] The eroder module combines enhanced semantic information and generates adaptive erosion weights through relative scale and absolute scale information matching to eliminate feature blur and ghosting.

[0014] Further, the event-based spatio-temporal target copy-paste strategy is to increase the learning samples of the network by introducing high-quality target data, and to improve the robustness and prediction ability of the network, including selection of time sequence target events and pasting of time sequence target events, wherein:

[0015] The selection of time sequence target events is used to select high-quality target to be pasted.

[0016] The pasting of time sequence target events is used to paste the enhanced target data to be pasted to new data.

[0017] Further, the selection of time sequence target events specifically includes the following criteria:

[0018] Criterion one, only the target that meets the IOU matching relationship of the target of any two consecutive frames on the continuous event frames within the training sequence can be added to the candidate area;

[0019] Criterion two, the matched target belongs to the same category;

[0020] Criterion three, the number of events of the target in the first pseudo image is screened to ensure that the early events of the introduced data are not too sparse.

[0021] Further, the pasting of time sequence target events includes pasting in random motion direction, pasting in random size and pasting in random position, wherein:

[0022] The pasting of the random motion direction is used for performing motion direction and orientation flipping of the pasted target according to a probability p, with a vertical axis of a center of a target GT bounding box as a symmetric axis.

[0023] The pasting of the random size is used for obtaining a resize scale of the pasted target under a uniform probability, with a center of a target GT bounding box as an origin;

[0024] The pasting of the random position is used for allowing the target to appear at any reasonable spatial position, including not colliding with any existing target in a motion process and not being truncated by a pseudo image edge.

[0025] Compared with the prior art, the present application has the following advantages:

[0026] 1. The efficient target detection method based on event stream sparse characteristics provided by the present application effectively processes stream sparse event data and generates complete target representation through shallow aggregation and enhanced feature representation in the dilated erosion long short-term memory network module of the event-based agile network.

[0027] 2. The efficient target detection method based on event stream sparse characteristics provided by the present application provides an efficient method to process event camera data and improve the performance of the target detection task, and proposes a novel event-based data enhancement method. Compared with the existing event-based target detection method, the event-based agile network (EvKeenDet) has better efficiency and robustness in processing stream sparse event data.

[0028] Based on the above reasons, the present application can be widely promoted in the field of data processing. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0030] Figure 1 It is the overall framework diagram of the present application.

[0031] Figure 2 It is the structure diagram of the out-of-order convolution module provided by the embodiment of the present application.

[0032] Figure 3 It is the architecture diagram of the dilated erosion long short-term memory network provided by the embodiment of the present application.

[0033] Figure 4 The structure schematic diagram of the expander and the corrosion device provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0034] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0035] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. The description of the at least one exemplary embodiment is actually only illustrative, but not as any limitation to the present application and its application or use. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0036] It should be noted that the terms used herein are only for describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form, unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a presence of the features, steps, operations, devices, components and / or their combinations.

[0037] Unless specifically stated otherwise, the relative arrangement of the components and steps, numerical expressions, and numerical values set forth in the embodiments are not meant to limit the scope of the present application. At the same time, it should be clear that the sizes of the various parts shown in the drawings are not drawn in accordance with the actual proportion relationship. The technology, methods and devices known to those skilled in the relevant art can not be discussed in detail, but should be considered as part of the authorized specification under appropriate circumstances. In all examples shown and discussed herein, any specific value should be interpreted as merely exemplary, and not as a limitation. Therefore, other examples of exemplary embodiments can have different values. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0038] In the description of the present application, it should be understood that the orientation words such as "front, back, upper, lower, left, right", "transverse, vertical, perpendicular, horizontal" and "top, bottom" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate and imply that the devices or elements referred to must have a particular orientation or be constructed and operated in a particular orientation, therefore cannot be understood as a limitation on the scope of protection of the present application: the orientation words "inner, outer" refer to the inner and outer relative to the contour of each component itself.

[0039] For the convenience of description, spatial relative terms such as "over", "above", "upper surface", "upper" and the like can be used herein to describe the spatial positional relationship of one device or feature with other devices or features as shown in the drawings. It should be understood that the spatial relative terms are intended to include different orientations in use or operation in addition to the orientation of the device described in the drawings. For example, if the device in the drawing is inverted, the device described as "above" or "over" other devices or structures will be positioned "below" or "under" the other devices or structures. Thus, the exemplary term "above" can include both "above" and "below" orientations. The device can also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein are interpreted accordingly.

[0040] In addition, it should be noted that the use of the words "first", "second" and the like to define parts only facilitates the differentiation of the corresponding parts, and the above words have no special meaning unless otherwise stated, and therefore cannot be understood as a limitation on the scope of protection of the present application.

[0041] The present application provides a high-efficiency target detection method based on event stream sparse characteristics, comprising:

[0042] S1, construct an event-based agile network; in this embodiment, in order to solve the problem of shallow ConvLSTM, an expanded and etched long short-term memory network is designed. It gradually calculates the space-time profile from the event data, aggregates information in short intervals, and further aggregates based on aggregated data in long intervals. The expanded and etched long short-term memory network is composed of two important sub-modules, namely Expander (Expander) and Etcher. The first sub-module Expander (Expander) is designed based on the expansion local attention mechanism, which can expand the acceptance domain, quickly aggregate short-term time information and neighborhood information. The etcher is to eliminate long space-time redundant information to remove the ambiguity and noise in the profile complete representation we propose. It can adaptively match the information at the current time with the information at the historical time, and then fuse the information.

[0043] In specific implementation, as a preferred embodiment of the present application, the core module of the event-based agile network constructed in step S1 is an expanded and etched long short-term memory module. By introducing expansion-etching operation, information in the pseudo image composed of event data in a short time is aggregated, including Expander module and Etcher module. The Expander module uses a large receptive field of hollow local attention mechanism to quickly aggregate time series information, and realizes fast and efficient expansion of neighborhood information. The Etcher module combines enhanced semantic information and generates adaptive etching weight through relative scale and absolute scale information matching to eliminate feature ambiguity and ghosting.

[0044] S2, based on the constructed event-based agile network, introduce event-based space-time target copy-paste strategy (Event GT Copy-Paste);

[0045] In specific implementation, as a preferred embodiment of the present application, the event-based space-time target copy-paste strategy (Event GT Copy-Paste) in step S2 is to increase the learning samples of the network by introducing high-quality target data, and improve the robustness and prediction ability of the network, including selection of time series target events (Event GT Copy) and pasting of time series target events (Event GT Paste), wherein:

[0046] The selection of time series target events (Event GT Copy) is used to select high-quality target to be pasted; the Event GT Copy specifically includes the following criteria:

[0047] Criterion 1: Only the object in the consecutive frames of the training sequence can be added to the candidate region, and the object in any two consecutive frames must satisfy the IOU matching relationship (for example, IOU>0.6);

[0048] Criterion 2: The matched object belongs to the same category;

[0049] Criterion 3: The number of events of the object in the first pseudo image is screened to ensure that the introduced data is not too sparse (for example, it is required that a>500).

[0050] In the embodiment, in order to enhance the training set with high-quality event data, a filtering strategy is designed to select the sequential object events to be pasted to the existing data. First, only the object sequence with rich events in the GT region in the first time window is concerned. This helps to avoid selecting sequences with excessively sparse events and ensures the quality of the copy-paste data. Second, only the sequences with IOU (intersection over union) between the consecutive time windows greater than the threshold are retained. This operation can help to select a series of high-quality events corresponding to the same object. Once the filtering process is completed, a set of high-quality object event sequences for each category can be obtained, which are saved in the corresponding in-memory library for copy-paste data enhancement.

[0051] The paste of the time-series target event (Event GT Paste) is used for data enhancement, including paste with random direction (RandomDirection), paste with random size (RandomResize), and paste with random position (RandomMove), wherein: the paste with random direction (RandomDirection) is used to flip the motion direction and orientation of the target to be pasted according to the vertical axis of the center of the target GT bounding box as the symmetry axis with a probability p (p=0.5); the paste with random size (RandomResize) is used to obtain the resize scale of the target to be pasted with a uniform probability with the center of the target GT bounding box as the origin; and the paste with random position (RandomMove) is used to allow the target to appear at any reasonable spatial position, including not colliding with any existing target during the motion and not being truncated by the pseudo image edge.

[0052] S3, put the autonomous driving simulation dataset into the event-based agile network, and realize event-based target detection. In the embodiment, in order to follow the comparative setting established by the previous research RED / DMANet, the EvKeenDet model of the application and other event-based models are evaluated on the EventCARLA dataset using the mAP@0.5 index. All models use a unified input representation, Voxel Grid voxel grid event representation. Among them, because the frame-based method cannot fully process the time sequence characteristics of event data, the performance of methods such as RetinaNet (parameter quantity 20.10M) / YOLOX-S (parameter quantity 8.94M) / YOLOX-M (parameter quantity 8.94M) is poor, which is 25.3% / 43.2% / 46.1% respectively. By observing YOLO-S and YOLO-M, it is found that even if the parameters are significantly increased, the performance improvement is very small, which shows that these non-time sequence model architectures cannot adapt to event data, and seriously limit the performance. The event-based target detection method RED (27.58M) introduces multi-scale time sequence aggregation detection on the basis of the SSD method, and obtains a performance of 46.6%. DMANet (28.17M) is inspired by the poor memory of RED, and introduces long-short dual spatio-temporal aggregation characteristics on the basis of RetineNet, and obtains a performance of 59.5%. The recent high-performance model RVT-B obtains a performance result of 72.9% on EventCARLA. While these models all ignore the streaming sparsity characteristics of events, on the contrary, the EvKeenDet model provided by the application considers the streaming sparsity characteristics of events, and obtains a performance of 77.1% mAP on EventCARLA. After using the data enhancement EventGT Copy-Paste proposed in the application which conforms to the characteristics of event data, the performance reaches 79.6% mAP@0.5Similarly, similar performance comparisons are also made on the real data 1 Megapixel Automotive Detection Datastet. The performance indicators of RetinaNet-18 / YOLO-S / YOLO-M on this real data set are 26.4% / 26.4% / 28.2%, which reflects the disadvantage of the non-timed frame-based network in event data. While the network architecture based on the event camera RED(27.58M) / DMANet(28.17M) / RVT-B(18.49M) has a performance of 39.7% / 44.4% / 47.4% on this real data set. The EvKeenDet(2.60M) model without applying the Event GT Copy-Paste of the application achieves a performance indicator of 51.1%, which is still much higher than these event-based target detection in the case of a substantial reduction in the number of parameters. After applying the data enhancement Event GT Copy-Paste method proposed in the application for training the stream sparse data, a higher performance of 53.5%mAP@0.5 is achieved.

[0053] As shown in Figure 1 The basic disordered convolution module of the encoder and the decoder of the target detection network in the application is composed of a convolution layer, a batch normalization layer (BN layer) and a ReLU activation function. The disordered convolution module serves as the backbone of the network and performs a feature extraction task. Different times of down-sampling are performed by setting the step of the internal convolution module. Before entering the target detection network, the stream event is first voxelized into a dense tensor Voxel Grid. Then, the information is aggregated in space and time by the dilated erosion long short-term memory network. After the network features are perceived in space and time, they enter the backbone network with multiple disordered convolution modules, which continuously perform spatial down-sampling. After obtaining the smaller spatial resolution features of the last three layers, a bottom-up aggregation process is performed to propagate semantic information to high-resolution features. Specifically, the higher low-resolution features with higher semantic information are continuously up-sampled by two times bilinear interpolation, and are aggregated with the high-resolution features of the next level. After the bottom-up aggregation process is completed, a top-down aggregation process is performed again, which aims to transfer the position information at high resolution to the low-resolution image. Specifically, the higher high-resolution features with higher position information are continuously down-sampled by two times bilinear interpolation, and are aggregated with the low-resolution features of the next level. The final processed multi-scale features are delivered to the multi-scale detection head for detection to obtain the final detection result.

[0054] In specific implementation, as a preferred embodiment of the application, as shown in Figure 2As shown in the figure, it is a structural diagram of the disordered convolution module. The module is used as a feature extraction module in the backbone network. The module uses 1x1 grouped convolution to perform less computationally intensive information aggregation calculation on the original channel order. After shuffling the feature channels, a 3x3 depth separable convolution is performed on each channel unit. Then the new channel order of the features is re-grouped and 1x1 grouped convolution is performed. Then it is added to the original input to form a residual structure. Considering that the convolution type module should have the function of two times down-sampling for scale transformation. When there is a requirement for two times feature down-sampling, the step of 3x3 depth separable convolution in it is modified to 2, and a 3x3 average pooling two times down-sampling is also added to the skip connection of the input to ensure the uniformity of the size. In general, the disordered convolution module obtains a larger range of perception with less computational load, thereby greatly improving the computational efficiency and further improving the efficient performance of the target detector.

[0055] In specific implementation, as a preferred embodiment of the present application, as shown in the figure, Figure 3 As shown in the figure, it is a structural diagram of the dilated erosion long short-term memory network. The network is used as an event spatio-temporal feature aggregation layer. The module uses a dilator with a hollow local attention to perform spatial linear efficient calculation of information, and then combines a feature map with certain semantic information with a corrosion module with a deblurring prior to obtain an adaptive weight. The adaptive weight will corrode the historical memory to form a more flexible adaptive memory. The current input feature will be connected with the corroded historical hidden state as the current input total feature. The input total feature will pass through multiple S-shaped growth extrusion functions to compress the value to [0, 1] as a soft valve to further control the proportion of the memory gate / update gate / output gate. The memory gate controls how much of the previous cell state should be retained, the update gate controls how much of the current memory should flow into the cell state, and the output gate controls how the current memory state should control the proportion of the output.

[0056] In specific implementation, as a preferred embodiment of the present application, as shown in the figure, Figure 4The expansion device and the corrosion device are shown in the structural schematic diagram. The expansion device is used as the input end of the expansion corrosion long short-term memory network, and is used to quickly improve the semantic information of the features. The expansion device uses a hollow adjacent attention mechanism to calculate the local space-time information of the input features connected with the previous expansion features, thereby quickly, accurately and efficiently improving the quality of local information and providing an input with sufficient semantic information for the network. The original intention of the corrosion device is to remove the blurred area. It calculates the similarity of the Euclidean distance absolute scale and the cosine distance relative scale of the features obtained by respectively performing learnable linear calculation on the expansion features at the current time and the expansion features at the previous time, normalizes the two types of similarity to the [0, 1] interval, and finally uses two learnable parameters to weight the absolute scale similarity and the relative scale similarity to form the final memory weight to corrode the memory.

[0057] Embodiment

[0058] In this embodiment, the EvKeenDet model used by the present application is modified based on NanoDet-Plus, and a small amount of additional training parameters for label assignment are used. The AdamW optimizer is used with an initial learning rate of 0.001, and the learning rate is adjusted during the training process using cosine annealing. In order to restore the event-based target detection paradigm, the present application introduces a variant of ConvLSTM at a deeper layer of the network, which is lightweighted by a Ghost module, and designs the framework of the present application according to this modification.

[0059] Following the configuration in RED / DMANet, the synthesis time of the single-frame pseudo image is set to 50 milliseconds. According to the setting of DMANet, the resolution of the pseudo image space is set to 512x 512. During the training process, the time step is fixed at 10, and for too small Ground Truth (GT) data, i.e. data with a diagonal length less than 60 pixels or a side length less than 20 pixels, it is ignored.

[0060] To ensure the robustness of the network, those Ground Truth objects (and all events within their ranges) that do not produce any events and have no intersection with other objects in the initial event time step are selected to be removed from the training dataset. In the event GT copy-paste method of the present application, the IoU matching threshold between consecutive frames is set to a = 0.6. In addition, the present application applies a filtering condition to the target objects, requiring them to have at least b > 500 events in the initial frame. Furthermore, the present application implements an upper limit l = 2000 for the maximum Class Memory Bank storage of each class. For the class sampling strategy of the present application, a positive linear correlation is established between the sampling probability of each class and its effective number in the training set. Specifically, the slope is set to 3.0, and the intercept is set to 500,000. The RandomResize operation of the present application is configured to scale the range [0.9, 1.3]. In addition, the present application deactivates the enhancement strategy after the 4th epoch, allowing the network to learn from the original data distribution. And all experiments are carried out on an NVIDI A100 Tensor Core GPU.

[0061] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An efficient target detection method based on event stream sparse characteristics, characterized in that, The application relates to an event-based agile network, a core module of which is an inflation and erosion long short-term memory network used as an aggregation layer of event spatio-temporal features, comprising an inflator module and an eroder module, the inflator module being used as an input end of the inflation and erosion long short-term memory network to improve semantic information of features; the inflator module uses a hollow adjacent attention mechanism to calculate local spatio-temporal information of the input features connected with previous inflation features, thereby improving local information quality and providing sufficient semantic information for the inflation and erosion long short-term memory network; the eroder module is used to remove fuzzy areas, calculate similarities of Euclidean distance absolute scales and cosine distance relative scales of features obtained by respectively performing learnable linear calculation on the inflation features of the current moment and the inflation features of the previous moment, normalize the similarities of the Euclidean distance absolute scales and the cosine distance relative scales to the [0, 1] interval, and finally use two learnable parameters to weight the absolute scale similarity and the relative scale similarity to form a final memory weight to erode the memory. Based on the constructed event-based agile network, an event-based spatio-temporal target copy-paste strategy Event GTCopy-Paste is introduced to increase the learning samples of the network and improve the robustness and prediction ability of the network by introducing high-quality target data, comprising selection of a time sequence target event and pasting of the time sequence target event, wherein: The selection of the time sequence target event is used to filter high-quality target to be pasted, comprising the following criteria: Criterion one: only the target satisfying an IOU matching relationship on the continuous event frames in the training sequence and any two continuous frames can be added to the candidate area; Criterion two: the matched target belongs to the same category; The pasting of the time sequence target event is used to paste the target data to be pasted after data enhancement to new data; Criterion three, number of events for the target in the first pseudo image Screening is performed to ensure that the early events of the introduced data are not too sparse; The event-based agile network is used for automatic driving simulation data set, and event-based target detection is realized. The pasting of the time sequence target event comprises random motion direction pasting, random size pasting and random position pasting, wherein:

2. The method of claim 1, wherein, The random size pasting is used to obtain a resize scale of the target to be pasted with the center of the target GT bounding box as an origin under uniform probability; The pasting of the random motion direction is used for taking the vertical axis of the center of the target GT bounding box as the axis of symmetry, and performing the motion direction and orientation flip of the target to be pasted according to the probability The pasting of the random motion direction is used for taking the vertical axis of the center of the target GT bounding box as the axis of symmetry, and performing the motion direction and orientation flip of the target to be pasted according to the probability The random position pasting is used to allow the target to appear at any reasonable spatial position, including not colliding with any existing target in the motion process and not being truncated by a pseudo-image edge. ​

Images