An event and image fusion perception-oriented target detection method

Abstract

The application relates to the technical field of target detection, and provides a target detection method for event and image fusion perception, which comprises the following steps: constructing a multi-modal target detection model; performing spatio-temporal alignment up-sampling on event data, so that event features and RGB features are aligned at a spatial resolution level; performing RGB feature extraction on RGB data to obtain RGB features; performing event feature extraction on a two-dimensional event frame image to obtain event features; performing multi-modal adaptive feature fusion on the RGB features and the event features to obtain multi-modal adaptive fusion features; performing multi-scale feature fusion on the multi-modal adaptive fusion features; and inputting the multi-modal fusion features after the multi-scale feature fusion into a target detection head module, so that the target detection head module completes a target detection task. The application realizes effective alignment of event features and RGB features at a spatial resolution level and a feature expression level, thereby significantly improving the detection precision and stability of target detection.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a target detection method based on event and image fusion perception. Background Technology

[0002] In recent years, numerous object detection algorithms have attempted to fuse RGB and event data to improve object detection performance in complex paths. However, the heterogeneous characteristics of event pixels and RGB images in their imaging mechanisms lead to significant alignment differences in multimodal data across time and space. This poses a serious challenge to existing methods in multimodal feature extraction and alignment. First, the extremely high frame rate of event data makes it difficult to synchronize the two on the timeline. Common methods include sampling and compressing event data, but both result in feature loss. Furthermore, event pixels have a much smaller resolution than RGB images, leading to a significant resolution difference. Downsampling RGB data to match event data further exacerbates feature loss. During feature fusion, simple addition or concatenation operations are insufficient to perfectly integrate cross-modal data. Summary of the Invention

[0003] This invention primarily addresses the significant performance degradation of existing target detection methods based on a single RGB camera in complex scenarios such as low light, overexposure, and high-speed motion. It also addresses the common technical problems in existing RGB image-event pixel fusion methods, including difficulties in spatiotemporal alignment, resolution mismatch, and insufficient robustness of cross-modal feature fusion. The invention proposes a target detection method oriented towards event and image fusion perception. Through a spatiotemporal alignment upsampling module for event data and a multimodal adaptive feature fusion module, it achieves effective alignment and collaborative modeling of event features and RGB features at the spatial resolution and feature representation levels while maintaining the advantages of high temporal resolution and high dynamic range of event data. This significantly improves the detection accuracy and stability of the target detection model in complex lighting and high-speed dynamic scenarios.

[0004] This invention provides a target detection method based on event and image fusion perception, comprising the following processes: Step 1: Collect RGB data and event stream data of the object to be detected in real time, and construct event frame images using the event stream data; Step 2: Construct a multimodal object detection model and train it. Step 3: The event data spatiotemporal alignment upsampling module performs spatiotemporal alignment upsampling on the event data to align the event features with the RGB features at the spatial resolution level. Step 4: The RGB feature extraction module extracts RGB features from the RGB data to obtain RGB features; the event feature extraction module extracts event features from the two-dimensional event frame image to obtain event features. Step 5: The multimodal adaptive feature fusion module performs multimodal adaptive feature fusion on the RGB features and event features to obtain multimodal adaptive fused features; Step 6, Multi-scale feature fusion module, performs multi-scale feature fusion on multi-modal adaptive fusion features; Step 7: Input the multi-modal fusion features obtained by multi-scale feature fusion into the target detection head module, and the target detection head module completes the target detection task.

[0005] Furthermore, in step 1, the RGB data and event stream data of the object to be detected are collected in real time using an RGB-event integrated device.

[0006] Furthermore, the multimodal target detection model includes: an RGB feature extraction module, an event feature extraction module, an event data spatiotemporal alignment upsampling module, a multimodal adaptive feature fusion module, a multiscale feature fusion module, and a target detection head module.

[0007] Furthermore, the RGB feature extraction module and the event feature extraction module include: an SPPF submodule, a C3K2 submodule, and a bottleneck layer submodule.

[0008] Furthermore, the event data spatiotemporal alignment upsampling module generates RGB attention scores based on RGB data using spatial attention operations. RGB attention score The formula is as follows: (1); in, This represents the sigmoid activation function; This refers to a convolution operation with a kernel size of 7; This is a global average pooling operation; This is a max pooling operation; F is the input. The event data generates event attention scores based on spatial continuity, including: selection A square neighborhood window is used to count the number of valid event pixels within the neighborhood. And based on the maximum neighborhood event density in the event frame As a normalization factor, a spatial continuity attention score is constructed, the expression of which is: (2); in, Indicates Centered Neighborhood window, For valid event indicator functions, This is the lower limit constant for normalization; By adding the RGB attention score and the event attention score to guide the upsampling process of the event features, event features consistent with the spatial resolution of the RGB data are obtained. ; The enhanced event features and RGB data are extracted layer by layer by the event feature extraction network and the RGB feature extraction network, and finally the event modality and RGB modality features at multiple scales are obtained.

[0009] Furthermore, the input to the multimodal adaptive feature fusion module is the output of the RGB feature extraction module and the event feature extraction module, and the output is the multimodal fused feature. ; RGB features and event characteristics First, perform a Concat operation to concatenate along the channel dimension, and then denote the result as the joint feature. : (3); Then combine features After global average pooling and The convolution operation yields an RGB adaptive convolution kernel. and event-adaptive convolution kernels ; RGB adaptive convolution kernel Event features are processed using depthwise separable convolutional operations and residual connections, with event-adaptive convolutional kernels. Perform depthwise separable convolution operations and residual connections on RGB features: (4); in, This indicates a depthwise separable convolution operation. For splicing operations, This is a convolution operation; The final output contains multiple fused features. .

[0010] Furthermore, the multi-scale feature fusion module performs feature fusion for features at different scales; and receives multiple outputs from the multi-modal adaptive feature fusion module. Where i = 1, 2, 3…; Output multi-scale feature fusion features .

[0011] Furthermore, the target detection head module adopts an Anchor-based approach, comprising three convolutional layers. The first two convolutional layers consist of convolution, batch normalization, and ReLU activation functions; the last convolutional layer has no batch normalization or activation function.

[0012] Furthermore, step 3 includes the following steps 301 to 303: Step 301: In the event data spatiotemporal alignment upsampling module, the multimodal mixed attention score is first obtained based on the attention scores of the RGB data and the attention scores of the event data. ; Step 302: The event data spatiotemporal alignment upsampling module performs an upsampling convolution operation on the event data to obtain the upsampled event data. ; Step 303: The event data spatiotemporal alignment upsampling module multiplies the upsampled event data with the multimodal mixed attention score to obtain the weighted event features. .

[0013] Furthermore, step 5 includes the following steps 501 to 502: Step 501: The multimodal adaptive feature fusion module fuses the RGB features with the event features to obtain joint features. ; and based on the joint features, generate RGB adaptive convolution kernels and event adaptive convolution kernels; Step 502: Using RGB adaptive convolution kernels and event adaptive convolution kernels, perform depthwise separable convolution operations and residual connection operations on event features and RGB features respectively.

[0014] Traditional RGB object detection is limited by the physical properties of RGB devices, making effective object detection impossible under extreme conditions such as low light, overexposure, and high-speed motion. This limits object detection performance to some extent. This invention provides an object detection method based on event and image fusion perception. Through a spatiotemporal alignment upsampling module for event data, it effectively aligns event features with RGB features at the spatial resolution level, avoiding information loss caused by traditional downsampling or simple interpolation. A multimodal adaptive feature fusion module enables dynamic feature interaction between different modalities, improving the model's ability to perceive and utilize dominant modality information in different scenarios. This invention constructs a multimodal object detection model, achieving spatiotemporal alignment of RGB and event data and multimodal adaptive feature fusion, thereby realizing heterogeneous feature fusion of multimodal data and achieving accurate detection under poor lighting and high-speed conditions. In complex environments such as low light, overexposure, and high-speed motion, it can significantly improve the accuracy and stability of object detection. This method is applicable to various object detection network structures and has good versatility and scalability. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the implementation of the target detection method based on event and image fusion perception provided by the present invention. Figure 2 This is a schematic diagram of a multimodal target detection model; Figure 3 This is a schematic diagram of the RGB feature extraction module and the event feature extraction module; Figure 4 This is a schematic diagram of the event data spatiotemporal alignment upsampling module; Figure 5 This is a schematic diagram of the multimodal adaptive feature fusion module; Figure 6 This is a schematic diagram of a multi-scale feature fusion module; Figure 7 This is a screenshot of the test results. Detailed Implementation

[0016] To make the technical problems solved by this invention, the technical solutions adopted, and the technical effects achieved clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings, not all of them.

[0017] like Figure 1 As shown in the figure, an embodiment of the present invention provides a target detection method based on event and image fusion perception, comprising the following processes: Step 1: Collect RGB data and event stream data of the object to be detected in real time, and construct event frame images using the event stream data.

[0018] For pedestrian or vehicle traffic, a real-time sensing and detection system using an integrated RGB-event device is employed to collect RGB data and event stream data of the object being detected. This device simultaneously acquires RGB image data and event stream data, and converts the acquired event stream data into two-dimensional event frame images.

[0019] The event stream data is in the format x, y, t, p, where x and y are pixel coordinates; t is the timestamp, indicating the time the event occurred; and p is the event polarity.

[0020] The process of converting event stream data into two-dimensional event frame images includes: classifying and encoding the event stream according to event polarity, mapping events of different polarities to different pixel representations, and then converting the event stream data into two-dimensional event frame images within a preset time window to serve as input features for subsequent neural networks. Specifically, the event polarity is either anode or cathode, with the anode being coded as 1 and the cathode as -1, to map events of different polarities to different pixel representations.

[0021] Step 2: Construct a multimodal target detection model and train the multimodal target detection model.

[0022] like Figure 2 As shown, the multimodal target detection model includes: an RGB feature extraction module, an event feature extraction module, an event data spatiotemporal alignment upsampling module, a multimodal adaptive feature fusion module, a multiscale feature fusion module, and a target detection head module.

[0023] like Figure 3 As shown, the RGB feature extraction module and the event feature extraction module include: SPPF (Spatial Pyramid Pooling Fast Version) submodule, C3K2 submodule, and Bottleneck submodule.

[0024] The bottleneck layer submodule is the most basic feature extraction unit, which first... Convolutional layers perform channel compression on the input features to reduce the channel dimension of the computation. Then, through... The compressed features are then subjected to spatial feature extraction, and finally... Convolution restores the number of channels to the output dimension. Simultaneously, a residual connection structure is introduced to ensure stable gradient propagation.

[0025] The C3K2 submodule divides the input features into two parts: one part is extracted for depth features by the bottleneck layer submodule, and the other part is directly passed through a convolution operation. Finally, the two feature parts are concatenated and then... Convolution is used for fusion.

[0026] The SPPF submodule is designed to concatenate multiple pooling layers, performing max pooling at different scales and then concatenating them before performing convolution operations. This indirectly forms a multi-scale feature representation.

[0027] Figure 3 In this context, Conv represents a combination of convolution, batch normalization, and Silu activation. Concat concatenates two features of the same size according to their batch dimensions. MaxPool performs max pooling on the features.

[0028] The RGB feature extraction module receives RGB data and outputs multiple RGB features at different scales. The event feature extraction module receives upsampled event data output by the event data spatiotemporal alignment upsampling module and outputs event features at multiple different scales. , where i represents the feature scale, which can be 1, 2, 3, ...

[0029] like Figure 4 As shown, the event data spatiotemporal alignment upsampling module generates RGB attention scores based on RGB data using spatial attention operations. RGB attention score The formula is as follows: (1); in, This refers to the sigmoid activation function; This is a convolution operation with a kernel size of 7; This is a global average pooling operation; This is a max pooling operation; F is the input, which takes RGB data.

[0030] The attention score is a weight tensor with values ​​between 0 and 1. Its magnitude is the same as the corresponding feature, and its coordinates are... The numerical representation of the corresponding coordinates in the feature The weighting of the target region significantly improves the localization and classification capabilities of target detection by emphasizing the target region and weakening the background region.

[0031] This invention generates event attention scores based on spatial continuity of event data. The invention employs a neighborhood event density statistical method as the spatial continuity statistical approach. Event pixels in the target region are densely distributed spatially. By calculating the number of valid events within the neighborhood of each pixel and normalizing the result, an attention score is generated. The more events within the neighborhood, the higher the attention weight.

[0032] The event data generates event attention scores based on spatial continuity, including: selection Square neighborhood window k For example, if we choose 3, we count the number of valid event pixels in the neighborhood. And based on the maximum neighborhood event density in the event frame The normalization factor is used to construct the spatial continuity attention score, and its expression is as follows: (2); in Indicates Centered Neighborhood window, For valid event indicator functions, This is the lower limit constant for normalization.

[0033] By adding the RGB attention score and the event attention score to guide the upsampling process of the event features, event features consistent with the spatial resolution of the RGB data are obtained. .

[0034] The enhanced event features and RGB data are extracted layer by layer by the event feature extraction network and the RGB feature extraction network, and finally the event modality and RGB modality features at multiple scales are obtained.

[0035] like Figure 5 As shown, the multimodal adaptive feature fusion module aims to achieve dynamic interaction between RGB features and event features. This module dynamically generates adaptive convolutional kernels from one modality of features and applies them to another, achieving adaptive modulation of cross-modal features and thus enhancing the expressive power of the dominant modality in different scenarios. The features after cross-modal adaptive fusion are input into a multi-scale feature fusion network to generate feature maps at different scales, and finally, the target detection head outputs the target's location and category information. The specific operation is as follows: The multimodal adaptive feature fusion module takes as input the outputs of the RGB feature extraction module and the event feature extraction module, and outputs the multimodal fused features. .

[0036] RGB features and event characteristics First, perform a Concat operation to concatenate along the channel dimension, and then denote the result as the joint feature. : (3); Then combine features After global average pooling and The convolution operation yields an RGB adaptive convolution kernel. and event-adaptive convolution kernels .

[0037] RGB adaptive convolution kernel Event features are processed using depthwise separable convolutional operations and residual connections, with event-adaptive convolutional kernels. Perform depthwise separable convolution operations and residual connections on RGB features: (4); in, This indicates a depthwise separable convolution operation. For splicing operations, This is a convolution operation.

[0038] The final output contains multiple fused features. .

[0039] like Figure 6 As shown, the multi-scale feature fusion module performs feature fusion on features at different scales (feature sizes) to enhance target detection performance. It receives multiple outputs from the multi-modal adaptive feature fusion module. Where i = 1, 2, 3… Output multi-scale feature fusion features The upsample operation uses nearest neighbor upsampling.

[0040] go through Figure 6 The operation shown outputs features at multiple different scales; for example, three features at different scales with downsampling factors of 8, 16, and 32, respectively, denoted as multi-scale fused features. .

[0041] The target detection head module adopts an anchor-based approach, comprising three convolutional layers. The first two layers consist of convolution, batch normalization, and ReLU activation functions; the last convolutional layer operates without batch normalization or activation functions. It receives multi-scale feature fusion outputs from a multi-scale feature fusion network. Output the object's bounding box position information, category probability, and object existence probability.

[0042] The training process of the multimodal object detection model is as follows: based on the output of the object detection head module, the predicted bounding boxes and category information are matched with the corresponding ground truth labels, and the loss function is calculated accordingly; then, through the backpropagation algorithm, the gradient of the loss with respect to the model parameters is transmitted back to each layer of the network, thereby updating the model parameters.

[0043] Step 3, the event data spatiotemporal alignment upsampling module, performs spatiotemporal alignment upsampling on the event data to align the event features with the RGB features at the spatial resolution level; Step 3 includes the following steps 301 to 303: Step 301: In the event data spatiotemporal alignment upsampling module, the multimodal mixed attention score is first obtained based on the attention scores of the RGB data and the attention scores of the event data. .

[0044] In the event data spatiotemporal alignment upsampling module, using formula (1), RGB data undergoes spatial attention operation to obtain RGB attention scores. Attention scores are determined by assessing the spatial continuity of the event data. Attention score of RGB data Event data attention score Adding them together yields the multimodal mixed attention score. .

[0045] Step 302: The event data spatiotemporal alignment upsampling module performs an upsampling convolution operation on the event data to obtain the upsampled event data. .

[0046] The event data is first pooled and then convolved with 1x1 to obtain an upsampled convolution kernel. This upsampled kernel is then convolved with the original event features to obtain the upsampled event data. .

[0047] Step 303: The event data spatiotemporal alignment upsampling module multiplies the upsampled event data with the multimodal mixed attention score to obtain the weighted event features. .

[0048] At this point, the event features are consistent with the corresponding RGB features in terms of spatial resolution, thus providing an alignment basis for subsequent multimodal adaptive feature fusion.

[0049] Step 4: The RGB feature extraction module extracts RGB features from the RGB data to obtain RGB features; the event feature extraction module extracts event features from the two-dimensional event frame image to obtain event features.

[0050] RGB images are used as input to the RGB feature extraction module; event images are used as input to the event feature extraction module; event data is acquired in the form of an event stream, denoised, and saved as event frames. The event images can be any two-dimensional or multi-dimensional representation constructed based on event polarity, time information, or a combination of both, and this invention does not impose specific limitations on them.

[0051] Step 5: The multimodal adaptive feature fusion module performs multimodal feature fusion on the RGB features and event features to obtain multimodal fused features.

[0052] In the multimodal adaptive feature fusion module, multimodal adaptive feature fusion operations are performed on RGB features and event features at different scales. This module dynamically generates convolutional kernels from features of one modality in an adaptive manner and applies these kernels to the feature representation of another modality to achieve correlation modeling and information complementarity between RGB features and event features. Step 5 includes the following steps 501 to 502: Step 501: The multimodal adaptive feature fusion module fuses the RGB features with the event features to obtain joint features. ; and generate RGB adaptive convolution kernels and event adaptive convolution kernels based on the joint features.

[0053] Step 502: Using RGB adaptive convolution kernels and event adaptive convolution kernels, perform depthwise separable convolution operations and residual connection operations on event features and RGB features respectively.

[0054] Through the dynamic convolution operation, when one modality has insufficient information response in a specific scenario, another modality can enhance its feature representation capability.

[0055] Step 6: Multi-scale feature fusion module, which performs multi-scale feature fusion on multi-modal adaptive fusion features.

[0056] The multimodal adaptive fusion features are respectively input into the multi-scale feature fusion network. By fusing feature maps of different scales, the detection capability of small targets and the robustness to scale changes are enhanced.

[0057] Step 7: Input the multi-modal fusion features obtained by multi-scale feature fusion into the target detection head module, and the target detection head module completes the target detection task.

[0058] The target detection head module employs an anchor-based detection structure. Its anchor box parameters are generated by clustering based on the statistical characteristics of target size in the PKU-DAVIS dataset. The target detection head module consists of three convolutional layers. After passing through the target detection head module, the object information and location of the target object are obtained.

[0059] like Figure 7 As shown, the method provided by this invention can maintain high accuracy and stability in complex lighting or high-speed motion scenarios, thus achieving stable target detection results in complex environments. Specific quantitative indicators are shown in the table below:

[0060] The following further details the implementation of this invention: 1. Data Preprocessing and Model Building First, time window segmentation and event quantization encoding are performed: a fixed time window is used to frame the continuous event stream, with the window duration set to 30ms. The selection of this parameter is based on comparative experiments in the traffic scene subset of the PKU-DAVIS dataset (including 10 complex scenes such as low light and high speed motion, with a total of 5000 frames of labeled data). The 30ms window can achieve the optimal balance between "event information density" and "real-time detection efficiency". Compared with the 10ms window (insufficient event information leads to a 15.3% false negative rate for small targets) and the 100ms window (event redundancy leads to a 23.1% background noise ratio), the model mAP under the 30ms window reaches 89.7%, and the processing time per frame is ≤20ms, which meets the requirements of real-time detection.

[0061] The quantization encoding rules for event polarity are as follows: numerical quantization is used, with anode event pixel values ​​quantized to (255, 0, 0) and cathode event pixel values ​​quantized to (128, 0, 128). After the event stream data is divided into event frames through a time window, event data denoising is required. A common method is to use a spatial neighborhood-based noise filtering approach. For each non-zero pixel, the number of event pixels in its neighborhood is counted, and a noise threshold of 2 is set. If the number of valid event pixels is less than 2, filtering is performed. The filtered event matrix is ​​then adjusted for resolution using bilinear interpolation. Size. First, the RGB data and event data are respectively passed to the event data spatiotemporal alignment upsampling module to obtain the upsampled event data. .Then and The input is fed into the event feature extraction module and the RGB feature extraction module for feature extraction, and the results are obtained. .

[0062] The input is fed into the multimodal adaptive feature fusion module for adaptive fusion, and the obtained fused features are: Pass it to Figure 7 A multi-scale feature fusion network is used to perform scale-wise fusion to obtain new features. .

[0063] The current detection result is passed to the detection head, and then the loss function is calculated based on the ground truth annotations. Gradient backpropagation training is then performed. Regarding the detection head construction, the target detection head adopts an anchor-based detection structure, and its anchor box parameters are adaptively generated based on the statistical characteristics of target size in the PKU-DAVIS dataset. Specifically, the bounding box information of all labeled targets in the statistical data set is collected, and the K-means clustering algorithm is used to perform cluster analysis on the target size sample set. The clustering objective is to minimize the IOU loss between the bounding box and the anchor box. Based on the scale characteristics of the multi-scale feature maps, the number of cluster centers K = 9 is set and evenly distributed across the three scale feature maps. The resulting anchor box values ​​are (12,11), (23, 17), (13, 31), (35, 28), (56, 40), (37, 72), (86, 63), (107, 107), (206, 162).

[0064] The detection head model adopts, for example Figure 6The anchor box detection head shown in the left module has the following structure for each detection branch: 2 layers of 3×3 convolutional layers (same padding, Leaky ReLU activation function, and kernel stride of 1) + 1 layer of 1×1 convolutional layer; where the 3×3 convolutional layer is used to refine the cross-modal adaptive fusion features and enhance the distinction between target features and background, and the 1×1 convolutional layer is used to map the high-dimensional fusion features into the target detection prediction tensor.

[0065] 2. Model Training To achieve accurate optimization in object detection, a multi-task joint loss function is adopted, with the total loss consisting of a weighted average of bounding box loss, object classification loss, and confidence loss. The bounding box loss uses GIOU loss to avoid the gradient vanishing problem of traditional IOU loss. The classification loss uses the BCE cross-entropy loss function. Gradient descent training is performed using the SGD optimizer, with the training parameters shown below. Table 1: Parameter Settings .

Claims

1. A target detection method based on event and image fusion perception, characterized in that, The process includes the following: Step 1: Collect RGB data and event stream data of the object to be detected in real time, and construct event frame images using the event stream data; Step 2: Construct a multimodal object detection model and train it. The multimodal target detection model includes: an RGB feature extraction module, an event feature extraction module, an event data spatiotemporal alignment upsampling module, a multimodal adaptive feature fusion module, a multiscale feature fusion module, and a target detection head module; The input to the multimodal adaptive feature fusion module is the output of the RGB feature extraction module and the event feature extraction module, and the output is the multimodal fused feature. ; RGB features and event characteristics First, perform a Concat operation to concatenate along the channel dimension, and then denote the result as the joint feature. : （3）； Then combine features After global average pooling and The convolution operation yields an RGB adaptive convolution kernel. and event-adaptive convolution kernels ; RGB adaptive convolution kernel Event features are processed using depthwise separable convolutional operations and residual connections, with event-adaptive convolutional kernels. Perform depthwise separable convolution operations and residual connections on RGB features: （4）； in, This indicates a depthwise separable convolution operation. For splicing operations, This is a convolution operation; The final output contains multiple fused features. ; Step 3: The event data spatiotemporal alignment upsampling module performs spatiotemporal alignment upsampling on the event data to align the event features with the RGB features at the spatial resolution level. Step 4: The RGB feature extraction module extracts RGB features from the RGB data to obtain RGB features; the event feature extraction module extracts event features from the two-dimensional event frame image to obtain event features. Step 5: The multimodal adaptive feature fusion module performs multimodal adaptive feature fusion on the RGB features and event features to obtain multimodal adaptive fused features; Step 6, Multi-scale feature fusion module, performs multi-scale feature fusion on multi-modal adaptive fusion features; Step 7: Input the multi-modal fusion features obtained by multi-scale feature fusion into the target detection head module, and the target detection head module completes the target detection task.

2. The target detection method based on event and image fusion perception according to claim 1, characterized in that, In step 1, the RGB data and event stream data of the object to be detected are collected in real time using an RGB-event integrated device.

3. The target detection method based on event and image fusion perception according to claim 1, characterized in that, The RGB feature extraction module and the event feature extraction module include: SPPF submodule, C3K2 submodule, and bottleneck layer submodule.

4. The target detection method based on event and image fusion perception according to claim 1, characterized in that, The event data spatiotemporal alignment upsampling module generates RGB attention scores based on RGB data using spatial attention operations. RGB attention score The formula is as follows: （1）； in, To represent the sigmoid activation function, This is a convolution operation with a kernel size of 7; This is a global average pooling operation; This is a max pooling operation; F is the input. The event data generates event attention scores based on spatial continuity, including: selection A square neighborhood window is used to count the number of valid event pixels within the neighborhood. And based on the maximum neighborhood event density in the event frame Using the normalization factor, construct the spatial continuity attention score. Its expression is: （2）； in, Indicates Centered Neighborhood window, For valid event indicator functions, This is the lower limit constant for normalization; By adding the RGB attention score and the event attention score to guide the upsampling process of the event features, event features consistent with the spatial resolution of the RGB data can be obtained; The enhanced event features and RGB data are extracted layer by layer by the event feature extraction network and the RGB feature extraction network, and finally the event modality and RGB modality features at multiple scales are obtained.

5. The target detection method based on event and image fusion perception according to claim 1, characterized in that, The multi-scale feature fusion module performs feature fusion for features at different scales; it receives multiple outputs from the multi-modal adaptive feature fusion module. Where i = 1, 2, 3…; Output multi-scale feature fusion features .

6. The target detection method based on event and image fusion perception according to claim 1, characterized in that, The target detection head module adopts an anchor-based approach, comprising three convolutional layers. The first two convolutional layers consist of convolution, batch normalization, and ReLU activation functions; the last convolutional layer has no batch normalization or activation function.

7. The target detection method based on event and image fusion perception according to claim 4, characterized in that, Step 3 includes the following steps 301 to 303: Step 301: In the event data spatiotemporal alignment upsampling module, the multimodal mixed attention score is first obtained based on the attention scores of the RGB data and the attention scores of the event data. ; Step 302: The event data spatiotemporal alignment upsampling module performs an upsampling convolution operation on the event data to obtain the upsampled event data; Step 303: The event data spatiotemporal alignment upsampling module multiplies the upsampled event data with the multimodal mixed attention score to obtain the weighted event features.

8. The target detection method based on event and image fusion perception according to claim 1, characterized in that, Step 5 includes the following steps 501 to 502: Step 501: The multimodal adaptive feature fusion module fuses the RGB features with the event features to obtain joint features. ; and based on the joint features, generate RGB adaptive convolution kernels and event adaptive convolution kernels; Step 502: Using RGB adaptive convolution kernels and event adaptive convolution kernels, perform depthwise separable convolution operations and residual connection operations on event features and RGB features respectively.

Images