Target detection algorithm based on RGB and event camera fusion in dynamic scene

Abstract

The invention relates to the technical field related to automatic driving, in particular to a target detection algorithm based on RGB and event camera fusion in a dynamic scene, which comprises the following steps: an event correction module (ECM) based on optical flow estimation; according to the target detection algorithm based on RGB and event camera fusion in the dynamic scene, by fusing the advantages of an RGB image and an event camera, an event correction module (ECM) based on optical flow estimation is provided, event feature representation is trained and optimized in a combined mode, an event feature dynamic up-sampling module (EDUM) is designed, an up-sampling kernel is adjusted in a self-adaptive mode, noise is restrained, and the target detection accuracy is improved. A cross-modal mamba fusion module (CMM) is provided, inter-modal feature interaction and global information extraction are achieved, the feature fusion effect is improved, experiments show that the method is remarkably superior to an existing method on DSEC and PKU-DAVIS-SOD data sets, higher detection precision and robustness are achieved, and the method is suitable for target detection tasks in dynamic scenes such as automatic driving.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a target detection algorithm based on the fusion of RGB and event cameras in dynamic scenes. Background Technology

[0002] In intelligent transportation systems, object detection is a key task for improving road safety, optimizing traffic management, and supporting autonomous driving technologies. Traditional RGB cameras, due to the limited dynamic range of their photosensitive elements and the time delay introduced by exposure, may lose important information in dynamic scenes such as complex lighting and high-speed motion, leading to the failure of perception algorithms. Therefore, it is necessary to introduce biomimetic vision sensors, such as event cameras, which have microsecond-level temporal resolution and high sensitivity to changes in lighting, performing excellently under high-speed motion and complex lighting conditions. Some researchers are leveraging the complementary characteristics of event and image modalities to achieve more robust perception algorithms.

[0003] Due to the unique paradigm of event cameras, researchers have proposed various event representation methods. Among them, dense representation methods are widely used because they can leverage the advantages of existing mature computer vision algorithms. These include image-based methods such as Event Histogram and Event Image, timestamp-based methods such as DiST, and voxel-based methods. However, these methods compress temporal information, which may introduce ambiguity and affect the extraction quality of spatial structure. Some motion compensation methods rely on complex model structures to alleviate the limitations of constant illumination and linear motion assumptions, resulting in complex training and designs that are not directly intended for object detection tasks. Furthermore, current real-world datasets often use spatially separate imaging systems for standard cameras and event cameras. When fusing events and frames, the problem of inconsistent data source resolution needs to be addressed. In terms of RGB and event fusion, many intermediate fusion methods have recently emerged to guide the fusion of the two modalities at the feature level. For example, bidirectional fusion modules are used to model multimodal features in spatial and channel dimensions, or attention mechanisms and feature statistics are used to enhance feature representation at the feature level. However, these methods do not simultaneously consider the global information of the two modalities and the imbalance problem during fusion. Summary of the Invention

[0004] The purpose of this invention is to provide a target detection algorithm based on the fusion of RGB and event cameras in dynamic scenes, so as to solve the problem of the decline in target detection performance of traditional RGB cameras in complex lighting and high-speed motion scenes mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a target detection algorithm based on RGB and event camera fusion in dynamic scenes, comprising the following steps:

[0006] Step 1: Event Correction Module (ECM) Based on Optical Flow Estimation:

[0007] 1) Select the event flow within the time interval between two image frame exposures, predict the motion vectors of the events using an optical flow estimation network, and generate a flow map;

[0008] 2) Based on the flow map, perform a warp operation on the events to make the event pairs correspond to the reference timestamp, and generate a corrected event flow;

[0009] 3) Map the corrected event stream to the nearest pixel using bilinear interpolation, and perform weighted summation based on polarity to generate event feature representations;

[0010] 4) Jointly train the event correction network and the object detection network, optimize the event correction process through an end-to-end backpropagation mechanism, and generate high-quality event feature representations that meet the requirements of object detection;

[0011] Step 2, Event Feature Dynamic Upsampling Module (EDUM):

[0012] 1) Perform global average pooling on the low-resolution event features of the input, and generate a dynamic upsampling kernel through 1x1 convolution;

[0013] 2) Use dynamic upsampling kernels to perform transposed convolution upsampling on low-resolution event features to generate high-resolution event features;

[0014] 3) Extract spatial attention from high-resolution RGB features, and use the smoothness of the RGB image to suppress noise in the upsampled event features to generate enhanced high-resolution event features;

[0015] Step 3, Cross-Modal Mamba Convergence Module (CMM):

[0016] 1) Perform linear transformations on the RGB features and event features respectively and project them into the latent space. Then normalize the features using scaling and offset factors.

[0017] 2) Cross-splicing is performed along the feature dimensions to form hybrid features;

[0018] 3) Input the hybrid features into the Mamba module, and use the SS2D (spatial-spatial two-dimensional scan) mechanism to alternately scan the features of the two modalities. Combine this with depthwise separable convolution (DWConv) and attention mechanism to enhance the modeling of correlations between modalities.

[0019] 4) Decouple the output features of the Mamba module back to RGB and event modalities, generate enhanced modal features, and fuse them with the original features through residual connections;

[0020] Step 4: Multi-scale feature fusion and target detection:

[0021] 1) Input the fused features into the Feature Pyramid Network (FPN+PANet) for multi-scale feature fusion;

[0022] 2) Finally, the target detection results are output through the YOLOX detection head.

[0023] Preferably, the event correction module (ECM) adopts an end-to-end joint training method, which optimizes the optical flow estimation network in reverse through the target detection results, so that the event features are more in line with the target detection requirements.

[0024] Preferably, the Event Feature Dynamic Upsampling Module (EDUM) suppresses noise amplification during the upsampling process by dynamically adjusting the upsampling kernel weights and combining them with the spatial attention mechanism of RGB features.

[0025] Preferably, the cross-modal Mamba fusion module (CMM) alternately scans RGB and event features through the SS2D mechanism, and combines depthwise separable convolution and attention mechanisms to achieve adaptive feature fusion between modalities.

[0026] Preferably, the optical flow estimation network in step one adopts a simple optical flow network structure to avoid introducing complex recurrent structures (such as ConVGRU, LSTM), thereby reducing training complexity and directly serving the target detection task.

[0027] Preferably, the dynamic upsampling module (EDUM) in step two extracts feature distribution information through global average pooling and dynamically adjusts the upsampling kernel parameters using 1x1 convolution to adapt to the differences in pixel distribution in different scenarios.

[0028] Preferably, the cross-modal Mamba fusion module (CMM) in step three considers both the global information of the two modalities and the imbalance between the modalities during the fusion process, and preserves the original feature information through residual connections to enhance modal complementarity.

[0029] Preferably, the Feature Pyramid Network (FPN+PANet) in step four is used to fuse multi-scale features to improve the robustness of object detection at different scales.

[0030] Preferably, the target detection head uses a YOLOX detector to achieve efficient target detection inference.

[0031] Compared with the prior art, the beneficial effects of the present invention are:

[0032] 1. An event correction module (ECM) based on optical flow estimation is adopted. Through joint training, the target detection results are effectively guided to optimize event correction and generate event feature representations that better meet the needs of target detection.

[0033] 2. An Event Feature Upsampling Module (EDUM) is proposed, which dynamically adjusts the upsampling kernel by the spatial distribution of feature information and suppresses event noise by the local smoothness of image modalities, thereby constructing high-resolution event features.

[0034] 3. A cross-modal Mamba fusion module (CMM) is proposed, which simultaneously performs feature interaction and global information extraction between modalities to perceive and select the most beneficial complementary information between the two modalities, thereby achieving adaptive feature fusion. Attached Figure Description

[0035] Figure 1 This is a diagram illustrating the overall architecture of the target detection algorithm based on RGB and event camera fusion in dynamic scenes according to the present invention.

[0036] Figure 2 This is a visualization of the two modal features of the present invention before and after passing through the cross-modal Mamba fusion module (CMM). Detailed Implementation

[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] Please see Figure 1-2 This invention provides a technical solution: a target detection algorithm based on RGB and event camera fusion in dynamic scenes, comprising the following steps:

[0039] Step 1: Event Correction Module (ECM) Based on Optical Flow Estimation:

[0040] 1) Select the event flow within the time interval between two image frame exposures, predict the motion vectors of the events using an optical flow estimation network, and generate a flow map;

[0041] 2) Based on the flow map, perform a warp operation on the events to make the event pairs correspond to the reference timestamp, and generate a corrected event flow;

[0042] 3) Map the corrected event stream to the nearest pixel using bilinear interpolation, and perform weighted summation based on polarity to generate event feature representations;

[0043] 4), Jointly train the event correction network and the target detection network, and optimize the event correction process through an end-to-end backpropagation mechanism to generate high-quality event feature representations that meet the requirements of target detection;

[0044] Further, select the event stream within the time period between the exposures of two image frames where, t0 < t , , detection , ,

[0045] , detection , , , , , ,

[0047] ,

[0050] , flow ,

[0046] ,

[0049] , flow , ,

[0048] , , ,

[0053] , ,

[0052] ,

[0051] < t0 + Δt, Δt is set to 50ms, representing the time window, N represents the number of events, regard the event stream as the motion trajectory record of an object in three-dimensional space, one dimension is time, and the event correction network predicts the horizontal and vertical motion conditions of each pixel point in this time period, that is, the flowmap. Based on this optical flow map, perform a warp operation on the events, that is, apply an optical flow transformation to each event pixel to align it to the reference time t ref , so as to propagate the events to the same timestamp:

[0045]

[0046] The corrected event stream ), Use bilinear interpolation to map it to the nearest pixel point according to the four domain points, and perform weighted summation by polarity:

[0047]

[0048] Jointly train the event correction network based on optical flow estimation and target detection, use the output of the optical flow network as the input of the detection network, and through an integrated backpropagation mechanism, make the target detection result effectively guide the optical flow network, so as to generate high-quality event feature representations that meet the requirements of target detection. The loss function of the end-to-end network is as follows:

[0049] L = λL flow +(1 - λ)L detection .

[0050] where, L flow [[ID=​​​​​​​​​​​

[0054] 3) Extract spatial attention from high-resolution RGB features, and use the smoothness of the RGB image to suppress noise in the upsampled event features to generate enhanced high-resolution event features;

[0055] Furthermore, firstly, the input event features f e ∈R B×C×H×W Global average pooling is used, and sequential interaction is achieved through 1*1 convolutions, assigning the learned weights to the transposed convolution kernel. Transpose and convolve the low-resolution event features to obtain high-resolution event features f. e ′∈R B×C×2H×2W The formula is as follows:

[0056] D = GlobalAPooling(f e ),

[0057]

[0058] f′ e =De-Conv(W f ,f e ).

[0059] GlobalAPooling represents global average pooling;

[0060] Considering that event cameras inevitably generate noise due to their sensitivity to junction leakage current and photocurrent, in order to suppress the amplified noise during the upsampling process, noise is generated from high-resolution RGB features f. R ∈R B×C×2H×2W Spatial attention is extracted, and the smoothness of the RGB image is used to enhance the upsampled event features, ultimately obtaining high-resolution enhanced event features. The formula is as follows:

[0061]

[0062] Where De-Conv represents the deconvolution operation.

[0063] Step 3, Cross-Modal Mamba Convergence Module (CMM):

[0064] 1) Perform linear transformations on the RGB features and event features respectively and project them into the latent space. Then normalize the features using scaling and offset factors.

[0065] 2) Cross-splicing is performed along the feature dimensions to form hybrid features;

[0066] 3) Input the hybrid features into the Mamba module, and use the SS2D (spatial-spatial two-dimensional scan) mechanism to alternately scan the features of the two modalities. Combine this with depthwise separable convolution (DWConv) and attention mechanism to enhance the modeling of correlations between modalities.

[0067] 4) Decouple the output features of the Mamba module back to RGB and event modalities, generate enhanced modal features, and fuse them with the original features through residual connections;

[0068] Furthermore, the two input modal features F e ∈R C×H×W ,F r ∈R C×H×W After linear transformation and projection onto the latent space, the amplitude and offset of the modal features are adjusted by scaling factors to adapt to the feature distribution of different modes, and the feature domains of different modal features are aligned and the distribution is normalized.

[0069] Z e =F e ⊙r e +β e

[0070] Z r =F r ⊙r r +β r

[0071] Where r e / r r and β e / β r These represent the scaling and offset factors, respectively. ⊙ represents the dot product operation. Then, Z... e Z r Cross-stitching along the W dimension yields the feature-level fine-grained hybrid feature Z. f ∈R C×H×2W This splicing operation provides the spatial and structural basis for fusion, enabling the two modal information to interact in the same representation space;

[0072] The mixed features are input into the Mamba module, and the two modal paths are scanned alternately by SS2D. The two modes have a common hidden state, which allows each modality to selectively compress relevant information based on information from other modalities, or selectively extract relevant information from the hidden state. Then, by combining depthwise separable convolution (DWConv) and attention mechanism, the computational efficiency is further improved and the ability to model the correlation between modalities is enhanced.

[0073] The output features of the Mamba module are decoupled back to two modes, generating enhanced modal features respectively. These enhanced features are then fused with the original features through residual connections, thereby preserving the original feature information while effectively enhancing modal complementarity.

[0074] Step 4: Multi-scale feature fusion and target detection:

[0075] 1) Input the fused features into the Feature Pyramid Network (FPN+PANet) for multi-scale feature fusion;

[0076] 2) Finally, the target detection results are output through the YOLOX detection head.

[0077] Furthermore, the Event Correction Module (ECM) adopts an end-to-end joint training approach, using the target detection results to back-optimize the optical flow estimation network, making the event features more consistent with the target detection requirements.

[0078] Furthermore, the Event Feature Dynamic Upsampling Module (EDUM) suppresses noise amplification during the upsampling process by dynamically adjusting the upsampling kernel weights and combining them with the spatial attention mechanism of RGB features.

[0079] Furthermore, the cross-modal Mamba fusion module (CMM) alternately scans RGB and event features through the SS2D mechanism, and combines depthwise separable convolution and attention mechanisms to achieve adaptive feature fusion between modalities.

[0080] Furthermore, the optical flow estimation network in step one adopts a simple optical flow network structure to avoid introducing complex recurrent structures (such as ConVGRU and LSTM), thereby reducing training complexity and directly serving the object detection task.

[0081] Furthermore, the Dynamic Upsampling Module (EDUM) in step two extracts feature distribution information through global average pooling and dynamically adjusts the upsampling kernel parameters using 1x1 convolution to adapt to the differences in pixel distribution in different scenarios.

[0082] Furthermore, the cross-modal Mamba fusion module (CMM) in step three considers both global information of the two modalities and the imbalance between modalities during the fusion process, and preserves the original feature information through residual connections to enhance modal complementarity.

[0083] Furthermore, the Feature Pyramid Network (FPN+PANet) in step four is used to fuse multi-scale features to improve the robustness of object detection at different scales.

[0084] Furthermore, the target detection head employs a YOLOX detector to achieve efficient target detection inference.

[0085] Furthermore, Figure 1 This demonstrates the framework of the invention, where the network takes RGB images and corresponding event streams as input. First, the event stream input is jointly trained based on an end-to-end architecture for optical flow estimation (ECM), as follows: Figure 1 As shown in (a), a high-quality event representation that meets the requirements of object detection is generated. Then, the event representation, along with the RGB image, is input into a five-stage object detection framework, as follows: Figure 1 As shown in (b), features are first extracted from the two modalities using a dual-stream architecture based on CSPDarkNet. Then, an Event Feature Dynamic Upsampling (EDUM) module is used after the third stage of the network. Figure 1 As shown in (c), event features are upsampled based on the spatial distribution of feature information to achieve resolution alignment between event features and RGB features. Then, the cross-modal Mamba fusion module (CMM) is used in the third, fourth, and fifth stages of the detection framework, respectively. Figure 1 As shown in (d), feature interaction between modalities and global information extraction are performed simultaneously to achieve adaptive feature fusion. The features fused in the last three stages are input into the feature pyramid network (FPN+PANet) for multi-scale feature fusion. Finally, the YOLOX detection head outputs the detection results.

[0086] Example 1: The present invention conducts comparative experiments on two autonomous driving datasets, DSEC and PKU-DAVIS-SOD, and uses COCO mAP (mean accuracy) to evaluate the accuracy of object detection.

[0087] On the DSEC and PKU-DAVIS-SOD datasets, the proposed method was compared with state-of-the-art RGB-Event fusion-based object detection methods: FPN-fusion, SODFormer, RENet, EOLO, EvImHandNet, and SFNet. Since EvImHandNet performs hand mesh reconstruction, only the cross-modal Mamba fusion module (CMB) in the proposed method's MCFNet was replaced with the fusion proposed by EvImHandNet for comparison. In addition, a baseline model without sparsity strategy was constructed to evaluate the effectiveness of the proposed method.

[0088] As shown in Table 1, on both datasets, the MCFNet method of this invention outperforms all state-of-the-art (SOTA) methods. In the case of class imbalance in the DSEC dataset, MCFNet outperforms the second-best method by 6.5% and 7.4% in mAP(50:95) and mAP50, respectively. In the case of class balance in the DSEC dataset, MCFNet outperforms the second-best method by 7.8% and 4.3% in mAP(50:95) and mAP50, respectively. The consistent performance improvement on real autonomous driving datasets of different resolutions demonstrates the effectiveness and generalization of the method of this invention.

[0089] Table 2 shows the performance metrics of the model on two datasets when using the same network framework but changing only the event representation method. The methods are divided into model-based motion estimation, unsupervised optical flow estimation, and traditional dense representation. The results show that the event representation generated by the event correction network ECNet based on optical flow estimation adopted in this invention can generate high-quality representation information in various dynamic scenarios.

[0090] Table 1 shows the comparison of detection performance with state-of-the-art methods on the autonomous driving datasets DSEC and PKU-DAVIS-SOD.

[0091]

[0092] Table 2 shows the performance comparison results with state-of-the-art event representation methods on the autonomous driving datasets DSEC and PKU-DAVIS-SOD.

[0093]

[0094] Example 2 Figure 2 The image shows feature maps before and after applying CMM, demonstrating the imbalance between the two modalities in different scenarios. For the event modality, in Figure 2 In the night scene of the first row, it can be observed that compared with the RGB mode, the event mode has a more stable imaging in low-light scenes due to its high dynamic range, and can capture target information more effectively. In the snow scene of the second row, a car triggers sparse events due to its slow relative speed to the camera. Therefore, the event mode does not pay attention to the information of the target. However, after CMM fusion, key information is selectively highlighted through adaptive fusion, which alleviates the modal imbalance problem. This shows that the CMM of the present invention has a strong scene adaptation capability.

[0095] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A target detection algorithm based on RGB and event camera fusion in dynamic scenes, characterized in that, Includes the following steps: Step 1: Event Correction Module (ECM) Based on Optical Flow Estimation: 1) Select the event flow within the time interval between two image frame exposures, predict the motion vectors of the events using an optical flow estimation network, and generate a flow map; 2) Based on the flow map, perform a warp operation on the events to make the event pairs correspond to the reference timestamp, and generate a corrected event flow; 3) Map the corrected event stream to the nearest pixel using bilinear interpolation, and perform weighted summation based on polarity to generate event feature representations; 4) Jointly train the event correction network and the object detection network, optimize the event correction process through an end-to-end backpropagation mechanism, and generate high-quality event feature representations that meet the requirements of object detection; Step 2, Event Feature Dynamic Upsampling Module (EDUM): 1) Perform global average pooling on the low-resolution event features of the input, and generate a dynamic upsampling kernel through 1x1 convolution; 2) Use dynamic upsampling kernels to perform transposed convolution upsampling on low-resolution event features to generate high-resolution event features; 3) Extract spatial attention from high-resolution RGB features, and use the smoothness of the RGB image to suppress noise in the upsampled event features to generate enhanced high-resolution event features; Step 3, Cross-Modal Mamba Convergence Module (CMM): 1) Perform linear transformations on the RGB features and event features respectively and project them into the latent space. Then normalize the features using scaling and offset factors. 2) Cross-splicing is performed along the feature dimensions to form hybrid features; 3) Input the hybrid features into the Mamba module, and use the SS2D (spatial-spatial two-dimensional scan) mechanism to alternately scan the features of the two modalities. Combine this with depthwise separable convolution (DWConv) and attention mechanism to enhance the modeling of correlations between modalities. 4) Decouple the output features of the Mamba module back to RGB and event modalities, generate enhanced modal features, and fuse them with the original features through residual connections; Step 4: Multi-scale feature fusion and target detection: 1) Input the fused features into the Feature Pyramid Network (FPN+PANet) for multi-scale feature fusion; 2) Finally, the target detection results are output through the YOLOX detection head.

2. The target detection algorithm based on RGB and event camera fusion in dynamic scenes according to claim 1, characterized in that, The Event Correction Module (ECM) adopts an end-to-end joint training approach, which optimizes the optical flow estimation network in reverse through the target detection results, so that the event features are more in line with the target detection requirements.

3. The target detection algorithm based on RGB and event camera fusion in dynamic scenes according to claim 1, characterized in that, The Event Feature Dynamic Upsampling Module (EDUM) suppresses noise amplification during the upsampling process by dynamically adjusting the upsampling kernel weights and combining them with the spatial attention mechanism of RGB features.

4. The target detection algorithm based on RGB and event camera fusion in dynamic scenes according to claim 1, characterized in that, The cross-modal Mamba fusion module (CMM) uses the SS2D mechanism to alternately scan RGB and event features, and combines depthwise separable convolution and attention mechanisms to achieve adaptive feature fusion between modalities.

5. The target detection algorithm based on RGB and event camera fusion in dynamic scenes according to claim 1, characterized in that, The optical flow estimation network in step one adopts a simple optical flow network structure to avoid introducing complex recurrent structures (such as ConVGRU and LSTM), thereby reducing training complexity and directly serving the target detection task.

6. The target detection algorithm based on RGB and event camera fusion in dynamic scenes according to claim 1, characterized in that, The Dynamic Upsampling Module (EDUM) in step two extracts feature distribution information through global average pooling and dynamically adjusts the upsampling kernel parameters using 1x1 convolution to adapt to the differences in pixel distribution in different scenarios.

7. The target detection algorithm based on RGB and event camera fusion in dynamic scenes according to claim 1, characterized in that, The cross-modal Mamba fusion module (CMM) in step three considers both global information of the two modalities and the imbalance between modalities during the fusion process. It preserves the original feature information through residual connections and enhances modal complementarity.

8. The target detection algorithm based on RGB and event camera fusion in dynamic scenes according to claim 1, characterized in that, The Feature Pyramid Network (FPN+PANet) in step four is used to fuse multi-scale features to improve the robustness of object detection at different scales.

9. The target detection algorithm based on RGB and event camera fusion in dynamic scenes according to claim 1, characterized in that, The target detection head uses a YOLOX detector to achieve efficient target detection inference.

Images