An event camera and color camera based information fusion target detection method

Abstract

The application discloses an information fusion target detection method based on an event camera and a color camera, and comprises the following steps: S1, performing data framing processing on a color stream of the color camera and an event stream of the event camera respectively; S2, constructing an information fusion target detection model; the information fusion target detection model comprises a double-stream feature extractor, a soft alignment feature fusion module, a feature pyramid and a center detection head; S3, processing modal data of the data framing processed color stream and event stream by using the information fusion target detection model to obtain a detection target. The method adopts a soft alignment mechanism between event information and color information by using a Transformer, solves the hardware data attribute limitation of the existing method, retains effective information of all original data, eliminates calibration errors, adopts a more advanced fusion calculation mode to improve fusion performance, and simultaneously enables the model to have adaptability to a complex multi-sensor environment.

Description

Technical Field

[0001] This invention belongs to the field of event-color fusion technology, specifically relating to a target detection method based on information fusion between an event camera and a color camera. Background Technology

[0002] An event camera, also known as a dynamic vision sensor (DVS), is a novel type of visual sensor capable of asynchronously detecting pixel-level brightness changes. Event cameras boast high dynamic range, microsecond-level temporal resolution, and no motion blur, making them suitable for various robotics and autonomous driving applications. On the other hand, color cameras offer rich semantic information but suffer from lower response times and are susceptible to harsh environments. Combining event and color cameras allows for the complementary advantages of both sensors, improving object detection performance and environmental adaptability, and holds significant importance for the development of high-speed autonomous driving.

[0003] Existing event-color fusion methods have inherent limitations on input data; they can only perform feature alignment on event frames and image frames with the same resolution and uniform viewpoint.

[0004] Most event-color fusion methods are based on the Dynamic and Active Pixel Vision Sensor (DAVIS), which includes an Active Pixel Sensor (APS) channel and a Dynamic Vision Sensor (DVS) channel, providing both hardware-aligned grayscale frames and event data with uniform resolution and viewpoint. While DAVIS is very convenient to use, this also limits DAVIS-dependent methods to its specific application. With the development of multi-sensor fusion, the industry increasingly needs fusion methods that can support independent event and color cameras.

[0005] To achieve event-color information fusion based on independent multi-sensor systems, ERF and DSEC-Det employ a hard alignment method, which involves manually aligning image frames and event frames using a calibration matrix. While hard alignment allows existing event-color fusion methods to be directly transferred to multi-sensor environments, it does not fundamentally address data limitations. As a trade-off, it introduces calibration errors and significant information loss, wasting substantial amounts of data and limiting the performance of existing fusion methods.

[0006] On the other hand, existing fusion methods' fusion modules are all based on direct, naive fusion approaches, such as element-wise addition, channel concatenation, inner product, or combinations thereof. These naive and traditional fusion methods limit the performance of the entire fusion model. Summary of the Invention

[0007] To address the aforementioned shortcomings in existing technologies, this invention provides a target detection method based on information fusion of event cameras and color cameras. This method solves the problem that existing fusion methods cannot guarantee the validity of all original data, resulting in calibration errors that affect the target detection performance.

[0008] To achieve the aforementioned objectives, the present invention employs the following technical solution: a target detection method based on information fusion of an event camera and a color camera, comprising the following steps:

[0009] S1. Perform data frame processing on the color stream of the color camera and the event stream of the event camera respectively;

[0010] S2. Construct an information fusion target detection model;

[0011] The information fusion target detection model includes a dual-stream feature extractor, a soft-aligned feature fusion module, a feature pyramid, and a central detection head.

[0012] S3. Using the information fusion target detection model, the modal data of the color stream and event stream after data pair data framing are processed to obtain the detection target;

[0013] Furthermore, in step S1, the method for performing data frame processing is as follows:

[0014] The color stream of a color camera is represented by RGB image frames, and the event stream of an event camera is represented by temporal voxel event frames.

[0015] Furthermore, in step S3, in the information fusion target detection model:

[0016] The dual-stream feature extractor is used to extract features from the modal data of the color stream and the event stream after data framing.

[0017] The soft-aligned feature fusion module is used to fuse the last three levels of features of the two modal data extracted by the dual-stream feature extractor to obtain fused features.

[0018] The feature pyramid is used to separate the feature scale of the fused features to obtain features corresponding to different target scale levels.

[0019] The central detection head is used to enable the feature extractor of the information fusion target detection model to concentrate features of different target scale levels into the feature vector of the object's center position, so as to adapt to the computational structure of the soft alignment feature fusion module; at the same time, the central detection head outputs the position information of the object based on the center point of the object to complete the target detection.

[0020] Furthermore, the dual-stream feature extractor includes ResNet50 convolutional networks that extract features from the color stream and the event stream, respectively; wherein the input channels of the ResNet50 convolutional network for extracting features from the color stream are 3, and the input channels of the ResNet50 convolutional network for extracting features from the event stream are 5.

[0021] Furthermore, the soft alignment feature fusion module includes a cascaded first fusion module, a second fusion module, and a third fusion module;

[0022] The input to the first fusion module is the penultimate layer of hierarchical features corresponding to the color stream and event stream modal data extracted by the dual-stream feature extractor;

[0023] The input to the second fusion module is the penultimate layer features corresponding to the color stream and event stream modal data extracted by the dual-stream feature extractor;

[0024] The input to the third fusion module is the last layer of features corresponding to the color stream and event stream modal data extracted by the dual-stream feature extractor;

[0025] The output features of the first fusion module, the second fusion module, and the third fusion module are all input into the feature pyramid.

[0026] Furthermore, the first fusion module, the second fusion module, and the third fusion module have the same structure, each including a cross-fusion submodule and a hybrid fusion submodule. Their specific method for feature fusion of the input features is as follows:

[0027] A1. In the cross-fusion submodule, the input features corresponding to the event stream and color stream are positionally encoded respectively, and the Group-Shuffle algorithm is used for processing to output the required features of the event stream and color stream.

[0028] A2. In the cross-fusion submodule, internal attention calculations are performed on the required features of the event stream and color stream respectively to enhance the modal internal features;

[0029] A3. In the cross-fusion submodule, cross-attention calculation is performed on the data after the internal features of the two modalities are enhanced, the features of the two modalities are mixed, and the data are input into the hybrid fusion submodule.

[0030] A4. In the hybrid fusion submodule, the color stream corresponding features in the modal fusion features are used as Q to perform attention calculation on the event stream, and feature information is extracted from the event stream again.

[0031] A5. In the hybrid fusion submodule, all extracted feature information is processed by channel concatenation and convolution to obtain the final fused features.

[0032] Further, in step A1, the input to the Group-Shuffle algorithm includes a feature map E from the event stream and a feature map C from the color stream, and specifies parameters n and i; wherein, the size of feature map E is (Dim, H... e ×W e The size of feature map C is (Dim, H). c ×W c ), where n represents the side length of the grouping unit, and i represents the number of the feature group being processed;

[0033] The specified parameters n and i satisfy:

[0034]

[0035] Furthermore, in step A1, the method for processing the input feature map using the Group-Shuffle algorithm is specifically as follows:

[0036] A11. Calculate the width and height (H, W) of the input feature map after padding, and set a mapping relationship index of length H×W; where (H, W) satisfies H|n, W|n;

[0037] A12, Grouping stage: Divide the input feature map into a grid with a cell side length of n;

[0038] For each cell, determine whether it is completely within the original width and height of the input feature map;

[0039] If so, convert the two-dimensional coordinates of each point in the cell to one-dimensional coordinates and add them to the end of the mapping relationship index;

[0040] If not, convert the two-dimensional coordinates of the points in the cell that do not exceed the original width and height range of the input feature map into one-dimensional coordinates and add them to the end of the mapping relationship index; at the same time, convert the two-dimensional coordinates of the point in the cell that is closest to the edge of the input feature map among the points that exceed the original width and height range of the input feature map into one-dimensional coordinates and add them to the end of the mapping relationship index.

[0041] A13. Shuffling Phase: For each n in the mapping relationship index... 2A series of consecutive mappings are randomly shuffled, and then n is used. 2 The step size interval is used to obtain the one-dimensional sequence coordinates of the features of the required group, and then the required features are obtained from the input feature map.

[0042] Furthermore, in the center detection head, a Gaussian distribution weight allocation strategy is used to make the information fusion target detection model focus on the center position of the target object, the expression of which is:

[0043]

[0044] In the formula, Y xyc The feature represents the x, y, z coordinates in the feature map, where (x, y) are the corresponding variables. σ represents the downsampled coordinates of the center point of each object label. p This represents the standard deviation of the width and height of all object labels.

[0045] Furthermore, the loss function L of the information fusion target detection model det for:

[0046] L det =L k +λ size L size +λ off L off

[0047] In the formula, L k L represents the object category prediction loss. size L represents the object size prediction loss. off λ represents the prediction loss at the object center. size and λ off L represents size and L off The corresponding weights.

[0048] The beneficial effects of this invention are as follows:

[0049] (1) The method of the present invention uses Transformer to implement a soft alignment mechanism between event information and color information, which solves the hardware data attribute limitation of the existing method, retains all the effective information of the original data, eliminates calibration error, adopts a more advanced fusion calculation method to improve fusion performance, and at the same time makes the fusion model adaptable to complex multi-sensor environments.

[0050] (2) In the information fusion target detection model provided by the present invention, the last three levels of features of the feature extractor adopt a soft-aligned feature fusion module based on Transformer attention calculation. On the one hand, it uses the global relationship of feature vectors to perform soft feature alignment in order to preserve data integrity and overcome the differences in hardware attributes of the two sensors; on the other hand, it uses the vector space transformation method of Query, Key and Value to perform cross and ring feature fusion simultaneously in order to make full use of the information of the event modality.

[0051] (3) In the method of the present invention, a target detection head based on center features is used for features at different scale levels. The center loss is used to compress the information of the entire object into the vector of the object's center position, so that it is adapted to the feature vector representation of the fusion module, thereby improving the consistency of feature representation and computation structure of each network component of the fusion model. Attached Figure Description

[0052] Figure 1 The flowchart of the target detection method based on information fusion of event camera and color camera provided by the present invention is shown.

[0053] Figure 2 The structural framework diagram of the information fusion target detection model provided by the present invention is shown.

[0054] Figure 3 The diagram shows the structure of the fusion module in the soft alignment feature fusion module provided by this invention. Detailed Implementation

[0055] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0056] This invention provides a target detection method based on information fusion from an event camera and a color camera, such as... Figure 1 As shown, it includes the following steps:

[0057] S1. Perform data frame processing on the color stream of the color camera and the event stream of the event camera respectively;

[0058] S2. Construct an information fusion target detection model;

[0059] S3. Using the information fusion target detection model, the modal data of the color stream and event stream after data pair data framing are processed to obtain the detection target;

[0060] like Figure 2 As shown, the information fusion target detection model in this embodiment includes a dual-stream feature extractor, a soft-aligned feature fusion module, a feature pyramid, and a central detection head.

[0061] In step S1 of this embodiment of the invention, the method for performing data framing processing is as follows:

[0062] The color stream of a color camera is represented by RGB image frames, and the event stream of an event camera is represented by temporal voxel event frames.

[0063] Specifically, in this embodiment, the event stream from the event camera is represented by the following temporal voxel event frames:

[0064]

[0065] δ(a) = max(0, 1 - |a|)

[0066]

[0067] Among them, event stream e i =(x i ,y i ,t i ,p i The time window is ΔT = 50ms, the total number of channels is C = 5, and the value of the (x,y)th pixel position of the cth channel of the event frame is E(x,y,c).

[0068] In the formula, t1 represents the timestamp of the transformed event, t1 represents the timestamp of the first event in the time window of the event stream, δ(·) represents a function defined as shown in the above formula, and a represents its independent variable.

[0069] In this embodiment, the color stream and event stream are framed. The framed representation method allows the data of the two modalities to be better processed and learned by the network structure of the subsequent information fusion target detection model.

[0070] In step S3 of this embodiment of the invention, Figure 2 In the information fusion target detection model shown:

[0071] The dual-stream feature extractor is used to extract features from the modal data of the color stream and the event stream after data framing.

[0072] The soft-aligned feature fusion module is used to fuse the last three levels of features of the two modal data extracted by the dual-stream feature extractor to obtain fused features.

[0073] The feature pyramid is used to separate the feature scale of the fused features to obtain features corresponding to different target scale levels.

[0074] The central detection head is used to enable the feature extractor of the information fusion target detection model to concentrate features of different target scale levels into the feature vector of the object's center position, so as to adapt to the computational structure of the soft alignment feature fusion module; at the same time, the central detection head outputs the position information of the object based on the center point of the object to complete the target detection.

[0075] In this embodiment, the dual-stream feature extractor includes ResNet50 convolutional networks that extract features from the color stream and the event stream, respectively; wherein, the input channel of the ResNet50 convolutional network for extracting features from the color stream is 3, and the input channel of the ResNet50 convolutional network for extracting features from the event stream is 5.

[0076] Specifically, when extracting features from color streams and event streams, the number of output channels is the same in the last three levels (C3, C4, C5), which are (512, 1024, 2048) respectively.

[0077] The soft alignment feature fusion module in this embodiment of the invention includes a cascaded first fusion module, a second fusion module, and a third fusion module;

[0078] The input to the first fusion module is the penultimate layer of hierarchical features corresponding to the color stream and event stream modal data extracted by the dual-stream feature extractor;

[0079] The input to the second fusion module is the penultimate layer features corresponding to the color stream and event stream modal data extracted by the dual-stream feature extractor;

[0080] The input to the third fusion module is the last layer of features corresponding to the color stream and event stream modal data extracted by the dual-stream feature extractor;

[0081] The output features of the first fusion module, the second fusion module, and the third fusion module are all input into the feature pyramid.

[0082] like Figure 3 As shown, the first fusion module, second fusion module, and third fusion module in this embodiment have the same structure, each including a cross fusion submodule and a mixed fusion submodule; specifically, in Figure 3In this context, PE stands for Sine positional embedding, Norm is for addition and normalization, GS is for Group-Shuffle algorithm, and Poc-FC is a fully connected network for each vector position. The upper fusion module in this embodiment implements feature fusion based on the Transformer's Attention operation. Specifically, its method for feature fusion of input features is as follows:

[0083] A1. In the cross-fusion submodule, the input features corresponding to the event stream and color stream are positionally encoded respectively, and processed using the Group-Shuffle algorithm to output the required features for the event stream and color stream.

[0084] A2. In the cross-fusion submodule, internal attention calculations are performed on the required features of the event stream and color stream respectively to enhance the modal internal features;

[0085] A3. In the cross-fusion submodule, cross-attention calculation is performed on the data after the internal features of the two modalities are enhanced, the features of the two modalities are mixed, and the data are input into the hybrid fusion submodule.

[0086] A4. In the hybrid fusion submodule, the color stream corresponding features in the modal fusion features are used as Q to perform attention calculation on the event stream, and feature information is extracted from the event stream again.

[0087] Specifically, the attention calculation in the hybrid fusion submodule and the event flow calculation in the Cross Submodule form a circular attention calculation structure. The two attention calculations in these two stages can make full use of the information in the event flow.

[0088] A5. In the hybrid fusion submodule, all extracted feature information is processed by channel concatenation and convolution to obtain the final fused features.

[0089] In this embodiment, since the fusion module uses a large amount of computationally intensive attention calculations, the Group-Shuffle algorithm is used in step A1 to process the input feature map in order to ensure sufficient performance while reducing the computational load of the fusion module.

[0090] In this embodiment, the input to the Group-Shuffle algorithm includes a feature map E from the event stream and a feature map C from the color stream, and specifies parameters n and i; wherein, the size of feature map E is (Dim, H e ×W e The size of feature map C is (Dim, H). c ×W c ), where n represents the side length of the grouping unit, and i represents the number of the feature group being processed;

[0091] The specified parameters n and i satisfy:

[0092]

[0093] In the Group-Shuffle algorithm described above, each feature map group uses a fusion module with shared weights. The parameter settings of the fusion module at each level need to be adjusted according to the actual situation. The larger the resolution of the framed modal data, the larger the value of n needs to be set, generally set to 2 to 5. It is recommended not to use the GS algorithm for the highest level feature level and set its n to 0.

[0094] In this embodiment, the output of the Group-Shuffle algorithm is the desired feature E of the processed specified group. i C i Based on this, the specific method for processing the input feature map using the Group-Shuffle algorithm is as follows:

[0095] A11. Calculate the width and height (H, W) of the input feature map after padding, and set a mapping relationship index of length H×W; where (H, W) satisfies H|n, W|n;

[0096] A12, Grouping stage: Divide the input feature map into a grid with a cell side length of n;

[0097] For each cell, determine whether it is completely within the original width and height of the input feature map;

[0098] If so, convert the two-dimensional coordinates of each point in the cell to one-dimensional coordinates and add them to the end of the mapping relationship index;

[0099] If not, convert the two-dimensional coordinates of the points in the cell that do not exceed the original width and height range of the input feature map into one-dimensional coordinates and add them to the end of the mapping relationship index; at the same time, convert the two-dimensional coordinates of the point in the cell that is closest to the edge of the input feature map among the points that exceed the original width and height range of the input feature map into one-dimensional coordinates and add them to the end of the mapping relationship index.

[0100] A13. Shuffling Phase: For each n in the mapping relationship index... 2 A series of consecutive mappings are randomly shuffled, and then n is used. 2 The step size interval is used to obtain the one-dimensional sequence coordinates of the features of the required group, and then the required features are obtained from the input feature map.

[0101] Specifically, for the i-th group, in Python, index[i::n] can be used. 2The one-dimensional sequence coordinates corresponding to the features of the i-th group can be obtained, and then the features of the required group can be obtained from the input feature map. During the inference stage, the shuffling stage can be turned off directly to eliminate useless computation.

[0102] In this embodiment of the invention, the fused features are input to a Feature Pyramid Network (FPN) to separate large-scale and small-scale features, making the detection results more accurate. In this embodiment, the FPN uses bilinear interpolation for upsampling to ensure the stability of the central feature position and reduce information loss transmitted to the central detection head.

[0103] In this embodiment of the invention, the center detection head is a CenterNet-based detection head, and its output class tensor is... Where W and H are the width and height of the input color frame, R is the minimum span of the output, i.e., the downsampling factor from the model to the detector head, and C is the number of object classes. One value of this tensor represents the class probability of the object at that location.

[0104] In the center detection head of this embodiment, a Gaussian distribution weight allocation strategy is used to make the information fusion target detection model focus on the center position of the target object, and its expression is:

[0105]

[0106] In the formula, Y xyc The feature represents the x, y, z coordinates in the feature map, where (x, y) are the corresponding variables. σ represents the downsampled coordinates of the center point of each object label. p This represents the standard deviation of the width and height of all object labels. If the Gaussian weights of two objects of the same category overlap, the weight at the overlapping position is the one with the larger weight.

[0107] In this embodiment of the invention, the loss function L of the information fusion target detection model det for:

[0108] L det =L k +λ size L size +λ off L off

[0109] In the formula, L k L represents the object category prediction loss. size L represents the object size prediction loss. off λ represents the prediction loss at the object center. size and λ off L representssize and L off The corresponding weights.

[0110] Specifically, in this embodiment, the object category prediction loss is determined based on the above Gaussian distribution formula, and its calculation formula is as follows:

[0111]

[0112] In the formula, α = 2, β = 4, and N is the number of centers of the color frame.

[0113] In this embodiment, in order to reduce the error caused by downsampling, it is also necessary to predict an offset tensor for each center point. The predicted loss for the object center is:

[0114]

[0115] In this embodiment, when determining the object size prediction loss, let category c be... k The bounding box of target k is Its width and height are Each feature level needs to predict the width and height tensors. The object size prediction loss is:

[0116]

[0117] In this embodiment of the invention, the center detection head uses a loss function that applies a weight preference to the center of the object, allowing the neural network model to concentrate the information of the entire object into the feature vector located at the center of the object.

[0118] In the information fusion target detection model in this embodiment of the invention, the attention calculation in the fusion module can be used to calculate the relationship between each feature vector. At the same time, the attention matrix can represent the matching degree of the feature vectors. Applying this principle to the fusion module enables the fusion module to calculate the correlation degree of feature vectors of color and event information from different perspectives, achieving the adaptation of the attention calculation structure and the central feature representation, thereby optimizing the performance of the model.

[0119] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0120] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A target detection method based on information fusion from an event camera and a color camera, characterized in that, Includes the following steps: S1. Perform data frame processing on the color stream of the color camera and the event stream of the event camera respectively; S2. Construct an information fusion target detection model; The information fusion target detection model includes a dual-stream feature extractor, a soft-aligned feature fusion module, a feature pyramid, and a central detection head; The soft alignment feature fusion module includes a cascaded first fusion module, a second fusion module, and a third fusion module; The input to the first fusion module is the penultimate layer of hierarchical features corresponding to the color stream and event stream modal data extracted by the dual-stream feature extractor; The input to the second fusion module is the penultimate layer features corresponding to the color stream and event stream modal data extracted by the dual-stream feature extractor; The input to the third fusion module is the last layer of features corresponding to the color stream and event stream modal data extracted by the dual-stream feature extractor; The output features of the first fusion module, the second fusion module, and the third fusion module are all input into the feature pyramid; The first, second, and third fusion modules have the same structure, each including a cross-fusion submodule and a hybrid fusion submodule. Their specific method for feature fusion of input features is as follows: A1. In the cross-fusion submodule, the input features corresponding to the event stream and color stream are positionally encoded respectively, and the Group-Shuffle algorithm is used for processing to output the required features of the event stream and color stream. A2. In the cross-fusion submodule, internal attention calculations are performed on the required features of the event stream and color stream respectively to enhance the modal internal features; A3. In the cross-fusion submodule, cross-attention calculation is performed on the data after the internal features of the two modalities are enhanced, the features of the two modalities are mixed, and the data are input into the hybrid fusion submodule. A4. In the hybrid fusion submodule, the color stream corresponding features in the modal fusion features are used as Q to perform attention calculation on the event stream, and feature information is extracted from the event stream again. A5. In the hybrid fusion submodule, all extracted feature information is processed by channel concatenation and convolution to obtain the final fused features; In step A1, the input to the Group-Shuffle algorithm includes feature map E from the event stream and feature map C from the color stream, and specifies parameters. n and i The size of feature map E is... The size of feature map C is , n Indicates the side length of the grouping unit. i Indicates the number of the feature group currently being processed; The specified parameters n and i satisfy: ； In step A1, the method for processing the input feature map using the Group-Shuffle algorithm is as follows: A11. Calculate the width and height of the input feature map after filling. and set a length of mapping relationship ;in, satisfy ; A12, Grouping stage: Divide the input feature map into a grid with a cell side length of n; For each cell, determine whether it is completely within the original width and height of the input feature map; If so, convert the two-dimensional coordinates of each point in that cell to one-dimensional coordinates and add them to the mapping relationship. The end; If not, convert the two-dimensional coordinates of the points in the cell that do not exceed the original width and height range of the input feature map to one-dimensional coordinates and add them to the mapping relationship. At the end; simultaneously, convert the two-dimensional coordinates of the point closest to the edge of the input feature map among the points within the cell that extend beyond the original width and height range of the input feature map into one-dimensional coordinates and add them to the mapping relationship. The end; A13, Shuffling Phase: Mapping Relationships each A series of consecutive mappings are randomly shuffled, and then... The step size interval is used to obtain the one-dimensional sequence coordinates of the features of the required group, and then the required features are obtained from the input feature map. S3. Using the information fusion target detection model, the modal data of the color stream and event stream after data frame processing are processed to obtain the detected target.

2. The target detection method based on information fusion of event camera and color camera according to claim 1, characterized in that, In step S1, the method for performing data framing processing is as follows: The color stream of a color camera is represented by RGB image frames, and the event stream of an event camera is represented by temporal voxel event frames.

3. The target detection method based on information fusion of event camera and color camera according to claim 2, characterized in that, In step S3, in the information fusion target detection model: The dual-stream feature extractor is used to extract features from the modal data of the color stream and the event stream after data framing. The soft-aligned feature fusion module is used to fuse the last three levels of features of the two modal data extracted by the dual-stream feature extractor to obtain fused features. The feature pyramid is used to separate the feature scale of the fused features to obtain features corresponding to different target scale levels. The central detection head is used to enable the feature extractor of the information fusion target detection model to concentrate features of different target scale levels into the feature vector of the object's center position, so as to adapt to the computational structure of the soft alignment feature fusion module; at the same time, the central detection head outputs the position information of the object based on the center point of the object to complete the target detection.

4. The target detection method based on information fusion of event camera and color camera according to claim 3, characterized in that, The dual-stream feature extractor includes ResNet50 convolutional networks that extract features from the color stream and the event stream, respectively; wherein the ResNet50 convolutional network for extracting features from the color stream has 3 input channels and the ResNet50 convolutional network for extracting features from the event stream has 5 input channels.

5. The target detection method based on information fusion of event camera and color camera according to claim 3, characterized in that, In the central detection head, a Gaussian distribution weight allocation strategy is used to make the information fusion target detection model focus on the center position of the target object, and its expression is: In the formula, Representation of feature map Characteristics of coordinates It is the corresponding variable. This represents the downsampled coordinates of the center point of each object label. This represents the standard deviation of the width and height of all object labels.

6. The target detection method based on information fusion of event camera and color camera according to claim 3, characterized in that, The loss function of the information fusion target detection model for: In the formula, This represents the loss for predicting object categories. This represents the object size prediction loss. Indicates the predicted loss at the center of the object. and express and The corresponding weights.

Images