A target detection method based on motion and event information decoupling

Abstract

The application discloses a target detection method based on motion and event information decoupling, aiming at the problem of coupling of appearance information and motion information in an event stream, proposes an event representation method for decoupling motion and appearance, and designs a double-flow detection network and an event motion guided attention module based on decoupled input, uses motion features to enhance appearance features, and thus greatly improves target detection performance.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a target detection method based on the decoupling of motion and event information. Background Technology

[0002] An event camera is a novel biomimetic vision sensor that encodes visual information in a pixel-level event stream. Unlike traditional cameras with a fixed frame rate, each pixel in an event camera responds asynchronously and independently to changes in light intensity. The main advantages of event cameras are their high temporal resolution (microseconds), high dynamic range (>120dB), low redundancy, and low power consumption. Because events are generated asynchronously only as light intensity changes, event cameras are naturally well-suited for object detection, especially in challenging visual scenarios such as high-speed and low-light conditions.

[0003] An event camera creates an event for each instantaneous action, recording the object's position at that instant. Specifically, an event can be described by four parameters: (x, y, t, p), where x and y represent spatial coordinates, t represents a timestamp, and p is the polarity, indicating whether the light intensity at that pixel increases or decreases. An event stream is a sparse and discrete set of points in spacetime, consisting of a series of event points. The key difference between event streams and still images is that event streams record the trajectory of each pixel on an object continuously, while still images capture only the appearance of an object at a specific moment. The differences between event cameras and still images are as follows: Figure 1 As shown.

[0004] For object detection tasks, static image-based object detectors typically rely on appearance information such as shape, texture, and pose to locate and identify object types. For event stream data, additional motion information from the event stream can enhance appearance-based object detection. On one hand, motion information provides an important clue to temporal continuity, as objects often exhibit continuous motion, while noise displays random and irregular variations. On the other hand, motion information provides important clues to spatial consistency, as adjacent image patches with similar displacements are likely to belong to the same foreground object or background region.

[0005] Due to the asynchronous and sparsity of event streams, existing frame-based object detection methods often require converting event data into dense, structured data (i.e., dense tensors). Some existing methods attempt to retain more temporal information while mapping event data to dense tensor representations. For example, Time Surface encodes temporal information as pixel intensity values, and Event Volume encodes temporary information as channel dimensions. Although event streams also contain motion information about the target, adding new information for object detection compared to static images, the coupling of motion and event information makes it difficult for the network to learn effective features for object detection in an end-to-end manner. This may actually lead to a decrease in object detection performance. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a target detection method based on the decoupling of motion and event information, which addresses the shortcomings of existing technologies and solves the problem that the coupling of motion and event information in existing event data makes it difficult for the network to learn effective features.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a target detection method based on decoupling motion and event information, comprising the following steps:

[0008] S1. Input the original events into the optical flow estimation network to obtain the motion tensor. Align the original events to the same reference time according to the motion tensor to obtain the appearance tensor.

[0009] S2. Use the motion tensor to generate motion features, use the appearance tensor to extract appearance features, and use the motion features and appearance features as input to the event motion-guided attention module to obtain three motion-related appearance features at different scales.

[0010] The event motion-guided attention module includes four attention units, wherein the output of the i-th attention unit is... Represented as: ψ e and ψ c These are the spatial attention coefficient and the channel attention coefficient, respectively. h(·) and h'(·) are both convolution operations. GAP(·) represents the global average pooling in the spatial dimension, and ⊙ represents element-wise multiplication. and These are the motion features and appearance features input to the i-th attention unit, respectively;

[0011] The output of the first attention unit serves as the input to the appearance branch network; the outputs of the second and third attention units serve as the inputs to the appearance branch network and the feature pyramid network; and the output of the fourth attention unit serves as the input to the feature pyramid network. The appearance branch network is used to extract appearance features.

[0012] S3. The motion-related appearance features at three different scales are used as inputs to the feature pyramid network, and the feature maps of the three sizes output by the feature pyramid network are used as inputs to the three detection heads to obtain the detection results.

[0013] Each of the detection heads includes a classification subnetwork and a detection subnetwork; the classification subnetwork includes M cascaded convolutional layers, and the detection subnetwork includes N cascaded convolutional layers.

[0014] The output of the first attention unit in this invention is used as the input of the appearance branch network. That is, the appearance features are enhanced by using motion features, the information of the event flow is decoupled, and the network is guided to learn information such as texture and contour that are more important for the target detection task. This solves the problem that the coupling of motion information and event information in existing event data makes it difficult for the network to learn effective features, and improves the target detection performance.

[0015] The event motion-guided attention module of this invention exhibits superior performance because it allows the network to focus on regions of interest and critical channels for object detection. Furthermore, by combining residual units with spatial and channel attention, the original features supplement regions that might be erroneously suppressed, preventing occlusion of key regions and enhancing the robustness of the detection method.

[0016] In step S1, the appearance tensor The expression is:

[0017]

[0018] Here, the value of each pixel (x,y) represents the number of event points falling within that pixel, δ(·) is the Dirac function, and (x′) k ,y′ k )=(x k ,y k )+(t k -t ref )v(x k ,y k ), x k ,y k Let v(x) be the coordinates of the event point at time k. k ,y k )=(v x ,v y ) represents a pixel (x) k ,yk Optical flow at point v x ,v y t represents the velocity of the pixel in the x and y directions, respectively. k t represents the time before event alignment. ref Reference time for event alignment. For the aligned event data, e′ k The original event point corresponding to the aligned event point is e. k =(x k ,y k ,t k ,p k ), e′ k It is e k The aligned result, p k Let N represent the polarity of the event point, and N be the total number of event points within a given time period.

[0019] This invention performs an alignment operation on events to obtain appearance features. The aligned events have clearer contours and edge features, further improving the accuracy of target detection.

[0020] In step S2, the motion features are extracted using a motion branch network, which includes multiple cascaded convolutional layers.

[0021] Both the appearance branch network and the motion branch network have 4 convolutional layers. Each branch network's four convolutional layers use 3*3 convolutions with a stride of 2 and the ReLU activation function. The number of output channels in the four cascaded convolutional layers increases sequentially.

[0022] In step S2, the appearance branch network includes multiple cascaded convolutional layers; the output of the first attention unit serves as the input to the second convolutional layer of the appearance branch network.

[0023] The three feature maps output by the feature pyramid network have the same number of channels.

[0024] The output channels of the detection sub-network are 4*K, where K is the number of anchor frames.

[0025] The three detection heads share the same weight.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention addresses the problem of coupling appearance information and motion information in event streams, and proposes an event representation method that decouples motion and appearance. Based on the decoupled input, the present invention designs a dual-stream detection network and an event motion-guided attention module, and uses motion features to enhance appearance features, thereby greatly improving target detection performance. Attached Figure Description

[0027] Figure 1 A comparison between traditional cameras and event cameras;

[0028] Figure 2 This is a schematic diagram of the network structure in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram of the Event Motion Guided Attention Module (EMGA) according to an embodiment of the present invention.

[0030] Figure 4 Visualization results of event representation; (a) direct projection, (b) projection after time alignment;

[0031] Figure 5 This is a schematic diagram of the feature pyramid network and detection head structure according to an embodiment of the present invention;

[0032] Figure 6 Visualizing the detection results; (a) and (b) show that the model of the present invention can work well in urban scenes with complex backgrounds, and (c) shows that fast-moving targets are often motion-blurred, making detection difficult. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] In this embodiment of the invention, the terms "first," "second," and other similar words are not intended to imply any order, quantity, or importance, but are merely used to distinguish different elements. In this embodiment of the invention, the terms "a," "an," and other similar words are not intended to indicate that only one of the things exists, but rather that the description refers only to one of the things, which may have one or more. In this embodiment of the invention, the terms "comprising," "including," and other similar words are intended to indicate a logical relationship, not a spatial relationship. For example, "A includes B" is intended to indicate that logically B belongs to A, not that spatially B is located inside A. Furthermore, the meaning of the terms "comprising," "including," and other similar words should be considered open-ended, not closed. For example, "A includes B" is intended to indicate that B belongs to A, but B does not necessarily constitute all of A; A may also include other elements such as C, D, and E.

[0035] The network proposed in this embodiment of the invention includes an event flow decoupling module, an appearance and motion information fusion module, and a result output module. The event flow decoupling module comprises an optical flow estimation subnetwork and an event alignment subnetwork. The appearance and motion information fusion module consists of a motion branch and an appearance branch, and their features are fused in a motion-guided attention module (EMGA). The result output module consists of a feature pyramid and a detector head, employing the same network structure as the RetinaNet model. The network structure is as follows: Figure 2 As shown.

[0036] The specific network process is as follows:

[0037] (1) Input the original event stream into the optical flow estimation network Ev-FlowNet to obtain the motion tensor T m (Also called optical flow graph), which aligns events to the same reference time t based on estimated motion. ref This yields an appearance tensor T with clear outlines and edges. a This separates appearance information from motion information.

[0038] An optical flow graph represents the velocity and direction of motion of each pixel in an image. At time t0, point A is located at (x1, y1), and at time t... k If we find point A again at time t, and its position is (x2, y2), then the motion of point A is: (u(x), v(y)) = [(x2, y2) - (x1, y1)] / (t) k -t0).

[0039] Given a set of event points This invention assumes that the optical flow is locally uniform over a short period of time. Based on the translation approximation, the event data is geometrically transformed using the optical flow v to obtain aligned event data. Each event point follows the motion curve from time t k Aligned to time t ref This process can be described as: (x′) k ,y′ k )=(x k ,y k )+(t k -t ref )v(x k ,y k Then, the event Aggregated into the appearance tensor:

[0040] Here, the value of each pixel (x, y) represents the number of event points falling within it, and δ(·) is the Dirac function. A comparison between directly projected original events and aligned projected original events is shown below. Figure 4As shown, the aligned event image has clearer contours and edge features.

[0041] (2) The motion tensor obtained in step (1) is fed into the motion branch to generate motion features. Then, the appearance tensor is fed into the Conv-0 convolutional layer of the appearance branch to extract appearance features. Then it was combined with motion characteristics Together, they are sent to the EMGA module. In the EMGA module, motion features serve as a guide, emphasizing important locations and channels within the appearance features. Next, we will focus on the motion appearance features. The input to the Conv-1 convolutional layer used for the appearance branch is used to generate appearance features. This process is repeated four times, extracting visual features from low to high scale to obtain motion-related appearance features at four different scales. In this embodiment of the invention, the length and width of the feature maps at the four scales are halved sequentially, while the number of channels is doubled sequentially. The length*width*number of channels for the four motion-related appearance features are 160*90*64, 80*45*128, 40*22*256, and 20*11*512, respectively.

[0042] In this embodiment of the invention, the four convolutional layers of the appearance branch and motion branch all use 3*3 convolution with a stride of 2, the activation function is the ReLU function, and the number of output channels are 64, 128, 256, and 512 respectively.

[0043] EMGA module such as Figure 3 As shown, the Event Motion Guided Attention (EMGA) module is a residual model with lightweight spatial and channel attention, which uses motion features... and appearance features The input is used as the basis. This module first converts motion features into spatial attention to emphasize important locations and elements in the appearance features, then applies channel attention to enhance some latent properties of the motion-focused appearance features, and finally uses the input features as supplementary input. Similar to the residual unit, there are two paths: one is the identity path, and the other is formed by spatial and channel attention. EMGA can be described as:

[0044]

[0045]

[0046]

[0047]

[0048] Where ⊙ denotes element-wise multiplication, h(·) and are both 1×1 convolutions with output channels 1 and 2 respectively. C.GAP(·) denotes global average pooling in the spatial dimension, and ψ e and ψ c These are the spatial attention coefficients and the channel attention coefficients, respectively. In summary, this module exhibits excellent performance because it allows the network to focus on the region of interest and the important channels used for object detection. Furthermore, by combining residual units with spatial and channel attention, the original features supplement regions that might be incorrectly suppressed, preventing some critical regions from being occluded. This strategy significantly enhances the algorithm's robustness.

[0049] In this embodiment of the invention, the event motion-guided attention module includes four attention units with identical structures. The output of the first attention unit enters the appearance branch, the outputs of the second and third attention units enter the appearance branch and the Feature Pyramid Network (FPN), and the output of the fourth attention unit does not enter the appearance branch but is only used as the input of the FPN.

[0050] (3) Motion-related appearance features at three different scales are sent to the feature pyramid and the detection head to obtain detection results.

[0051] The pyramid of features in the embodiments of the present invention The input is a feature map, and the output consists of three feature maps of different sizes, each with 256 channels. The larger feature map is used to detect small objects, and the smaller feature map is used to detect large objects. Specifically, the higher-level features of the two feature layers are upsampled by a factor of 2 (generally using interpolation, i.e., inserting new elements between pixels using a suitable interpolation algorithm, effectively doubling the size of the feature map). The lower-level features are then modified by a 1×1 convolution, and the corresponding elements of the upsampled and 1×1 convolution results are summed.

[0052] The detection head consists of two branches: classification and location detection. Each branch has four convolutional layers, but the parameters between these two branches are not shared. The output channels of the classification sub-network are A*K, where A is the number of classes; the output channels of the detection sub-network are 4*K, where K is the number of anchor boxes. Although the weights of the classification and regression branches of the two sub-networks are not shared, the weights of the detection head for the three output feature maps are shared.

[0053] This invention was tested on the GEN1 car detection dataset (GEN1 dataset) and the Prophesee 1Mpx car detection dataset (1Mpx detection dataset). More specifically, the Gen1 dataset consists of 39 hours of event data (304×240 resolution) recorded by ATIS cameras, containing manually annotated bounding boxes for 226,719 cars and 27,658 pedestrians. The 1Mpx detection dataset provides an event stream at a resolution of 1280×720, containing over 25 million labeled bounding boxes for cars, pedestrians, and two-wheeled vehicles, annotated at a frequency of 60Hz, and provides 11.19 hours of training data, 2.21 hours of validation data, and 2.25 hours of test data.

[0054] In this embodiment of the invention, the time window of the model is set to 50ms. The optical flow estimation network and the object detection network are trained separately. The optical flow estimation network is implemented as EV-FlowNet, an unsupervised optical flow estimation method. In this embodiment of the invention, all layers of the model are randomly initialized, and the Adam optimizer is used to schedule 100 iterations with a linearly decaying learning rate. The initial learning rate is set to 0.0001 on the GEN1 dataset and 0.00005 on the 1Mpx detection dataset, with a batch size of 1 for both datasets.

[0055] The appearance branch is initialized using a ResNet model pre-trained on ImageNet. For the Gen1 dataset, the initial learning rate is set to 0.0001 and the batch size to 32. For the 1Mpx detection dataset, this embodiment sets the initial learning rate to 0.00002 and the batch size to 16. The learning rate decays linearly until it reaches 1 / 10 of the initial value.

[0056] To prevent overfitting of the network, this embodiment of the invention uses random horizontal flipping, scaling up, and scaling down. This embodiment selects the optimal training model on the validation dataset and evaluates it on the test dataset. For both datasets, mAP (mean accuracy) and runtime are the main metrics considered in this embodiment.

[0057] As shown in Table 1, the method proposed in this embodiment of the invention is compared with six existing event target detection models. It can be seen that, compared with the second-best performing method, the MADNet of this embodiment achieves improvements of approximately 6.4% and 5.1% on the Gen1 Automotive Dataset and 1Mpx Detection Dataset, respectively. Furthermore, the method of this embodiment has the lowest inference latency.

[0058] Table 1 Comparison with other event target detection algorithms

[0059]

[0060] Some representative visualizations from the 1Mpx dataset, such as Figure 6 As shown. Figure 6 Images (a) and (b) show that the model works well in urban scenes with complex backgrounds. Fast-moving targets are often difficult to detect due to motion blur, such as... Figure 6 As shown in (c), by decoupling motion and appearance information, a clear contour appearance image is obtained. Therefore, MADNet can also achieve good detection performance for fast-moving targets.

[0061] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0062] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A target detection method based on decoupling motion and event information, characterized in that, Includes the following steps: S1. Input the original events into the optical flow estimation network to obtain the motion tensor. Align the original events to the same reference time according to the motion tensor to obtain the appearance tensor. S2. Use the motion tensor to generate motion features, use the appearance tensor to extract appearance features, and use the motion features and appearance features as input to the event motion-guided attention module to obtain three motion-related appearance features at different scales. The event motion-guided attention module includes four attention units, wherein the output of the i-th attention unit is... Represented as: ; , , , and These are the spatial attention coefficient and the channel attention coefficient, respectively. and All are convolution operations. The ⊙ symbol represents the global average pooling in the spatial dimension, and ⊙ represents element-wise multiplication. and These are the motion features and appearance features input to the i-th attention unit, respectively; The output of the first attention unit serves as the input to the appearance branch network; the outputs of the second and third attention units serve as the inputs to the appearance branch network and the feature pyramid network; and the output of the fourth attention unit serves as the input to the feature pyramid network. The appearance branch network is used to extract appearance features. The motion features are extracted using a motion branch network, which includes multiple cascaded convolutional layers. The appearance branch network and the motion branch network each have 4 convolutional layers. The four convolutional layers of each branch network use 3*3 convolution with a stride of 2 and the activation function is the ReLU function. The number of output channels of the four cascaded convolutional layers increases sequentially. The appearance branch network includes multiple cascaded convolutional layers; the output of the first attention unit serves as the input to the second convolutional layer of the appearance branch network. S3. The motion-related appearance features at three different scales are used as inputs to the feature pyramid network, and the feature maps of the three sizes output by the feature pyramid network are used as inputs to the three detection heads to obtain the detection results. Each of the detection heads includes a classification subnetwork and a detection subnetwork; the classification subnetwork includes M cascaded convolutional layers, and the detection subnetwork includes N cascaded convolutional layers.

2. The target detection method based on decoupling motion and event information according to claim 1, characterized in that, In step S1, the appearance tensor The expression is: ； Each pixel The value represents the number of event points that fall within that pixel. It is the Dirac function. , Let v( be the coordinates of the event point at time k) ) = ( ) represents pixels ( The light flow at that location These represent the velocity of the pixel in the x and y directions, respectively. The time before event alignment. Reference time for event alignment. For the aligned event data, , The corresponding original event point is for the aligned event point. , Let N represent the polarity of the event point, and N be the total number of event points within a given time period.

3. The target detection method based on decoupling motion and event information according to claim 1, characterized in that, The three feature maps output by the feature pyramid network have the same number of channels.

4. The target detection method based on decoupling motion and event information according to claim 1, characterized in that, The output channels of the detection sub-network are 4*K, where K is the number of anchor frames.

5. The target detection method based on decoupling motion and event information according to claim 1, characterized in that, The three detection heads share the same weight.

Images