Lightweight end-to-end object detection method and device, electronic equipment and storage medium

Abstract

The invention discloses an object detection method, and particularly relates to a lightweight end-to-end object detection method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining an RGB frame and an event frame corresponding to a detection region; inputting the RGB frame into a first backbone network for feature extraction to obtain K RGB sub-features of different scales, and inputting the event frame into a second backbone network for feature extraction to obtain K event sub-features of different scales, K being a positive integer; fusing the RGB sub-features and the event sub-features of the same scale to obtain K preliminary fusion features; performing cross-scale cross-modal fusion processing on the K preliminary fusion features to obtain target features; and inputting the target features into a decoder for decoding processing to obtain classification and positioning of the detection object in the detection area. According to the method, the object detection accuracy in a complex environment can be improved.

Description

Technical Field

[0001] The present invention relates to an object detection method, and in particular to a lightweight end-to-end object detection method, device, electronic device and storage medium. Background Art

[0002] In the field of autonomous driving, object detection under adverse conditions plays a vital role, where distortion caused by low-light conditions and motion blur can severely affect detection performance. Due to the limitation of traditional cameras to capture photons within a specific exposure time to generate a frame, object detection often encounters ambiguity caused by extreme lighting and high-speed motion, which hinders the performance of object detection in autonomous driving applications.

[0003] Event cameras are a new type of frame sensing technology that has a pixel array that responds to changes in light intensity at each pixel, enabling them to quickly respond to changes in ambient light, especially edge information of objects in complex environments. However, event cameras lack texture information and are sensitive to noise in changes in ambient light, which results in object detection using only event data being significantly lower than methods using RGB frames in the same scene.

[0004] Therefore, the existing technology considers using the unique characteristics of event data and RGB frames to continuously detect targets in an asynchronous manner, thereby achieving object detection in complex environments. However, the methods provided by the existing technology require a large number of parameters (more than 80 million), which limits edge deployment in autonomous driving. At the same time, the non-end-to-end structure used in the methods provided by the existing technology also leads to very long training time and difficulty in integrating the advantages of each mode, resulting in poor object detection in complex environments. Summary of the invention

[0005] The technical problem to be solved by the present invention is that the above-mentioned methods commonly used in the prior art cannot well integrate the advantages of each mode, resulting in poor object detection effect in complex environments. In order to solve the above-mentioned problem, the present invention provides a lightweight end-to-end object detection method, device, electronic device and storage medium.

[0006] The content of the present invention includes:

[0007] In a first aspect, an embodiment of the present invention provides a lightweight end-to-end object detection method, including:

[0008] Get the RGB frame and event frame corresponding to the detection area;

[0009] Inputting the RGB frame into the first backbone network for feature extraction to obtain K RGB sub-features of different scales, and inputting the event frame into the second backbone network for feature extraction to obtain K event sub-features of different scales, where K is a positive integer;

[0010] Fusing the RGB sub-features and the event sub-features of the same scale to obtain K preliminary fused features;

[0011] Performing cross-scale and cross-modal fusion processing on the K preliminary fusion features to obtain target features;

[0012] The target features are input into a decoder for decoding processing to obtain the classification and positioning of the detection object in the detection area.

[0013] Optionally, the preliminary fused feature obtained by fusing the RGB sub-feature and the event sub-feature at the i-th scale is the i-th fused feature, where i is a positive integer less than or equal to K, and the RGB sub-feature and the event sub-feature at the same scale are fused to obtain K preliminary fused features, including:

[0014] The RGB sub-feature at the first level scale is fused with the event sub-feature in a first manner to obtain a first fused feature, wherein the first level scale is the highest scale and the Kth level scale is the lowest scale;

[0015] The RGB sub-feature at the j-th scale is fused with the event sub-feature in a second manner to obtain a j-th fused feature, where j=2, 3, ..., K.

[0016] Optionally, fusing the RGB sub-feature at the first scale with the event sub-feature in a first manner to obtain a first fused feature includes:

[0017] Inputting the RGB sub-feature of the first-level scale into a first single-stage Transformer encoder for processing to obtain a target RGB sub-feature, and inputting the event sub-feature of the first-level scale into a second single-stage Transformer encoder for processing to obtain a target event sub-feature;

[0018] The target RGB sub-feature and the target event sub-feature are concatenated to obtain the first fusion feature.

[0019] Optionally, fusing the RGB sub-feature at the j-th scale with the event sub-feature in a second manner to obtain a j-th fused feature includes:

[0020] After splicing the RGB sub-features and the event sub-features at the j-th scale, the number of channels is adjusted and down-sampled using convolution to obtain the j-th fused feature.

[0021] Optionally, the value of K is 3, the K preliminary fusion features include the first fusion feature, the second fusion feature and the third fusion feature, and the cross-scale and cross-modal fusion processing is performed on the K preliminary fusion features to obtain the target feature, including:

[0022] Fusing the first fused feature and the second fused feature to obtain a first high-order feature;

[0023] Fusing the first high-order feature and the third fused feature to obtain a first low-order feature;

[0024] Fusing the first low-order feature with the first high-order feature to obtain a second low-order feature;

[0025] Fusing the second low-order feature with the first fused feature to obtain a second high-order feature;

[0026] The second low-order feature and the second high-order feature are concatenated to obtain the target feature.

[0027] Optionally, acquiring the RGB frame and the event frame corresponding to the detection area includes:

[0028] Obtaining a video stream and a corresponding event stream of the detection area;

[0029] The corresponding RGB frame is acquired based on the video stream, and the event stream is compressed to obtain the event frame.

[0030] Optionally, compressing the event stream to obtain the event frame includes:

[0031] Events occurring within a preset time in the event stream are mapped onto a two-dimensional plane to obtain the event frame.

[0032] In a second aspect, an embodiment of the present invention further provides a lightweight end-to-end object detection device, including:

[0033] An acquisition module is used to acquire the RGB frame and event frame corresponding to the detection area;

[0034] A feature extraction module, used to input the RGB frame into the first backbone network for feature extraction to obtain K RGB sub-features of different scales, and input the event frame into the second backbone network for feature extraction to obtain K event sub-features of different scales, where K is a positive integer;

[0035] A preliminary fusion module, used to fuse the RGB sub-features and the event sub-features of the same scale to obtain K preliminary fusion features;

[0036] A cross-fusion module is used to perform cross-scale and cross-modal fusion processing on the K preliminary fusion features to obtain target features;

[0037] The decoding module is used to input the target features into a decoder for decoding processing to obtain the classification and positioning of the detection object in the detection area.

[0038] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a program stored in the memory and executable on the processor; the processor is used to read the program in the memory to implement the steps in the lightweight end-to-end object detection method as described in the first aspect.

[0039] In a fourth aspect, an embodiment of the present invention provides a readable storage medium for storing a program, which, when executed by a processor, implements the steps in the lightweight end-to-end object detection method as described in the first aspect.

[0040] The beneficial effect of the present invention is that, in an embodiment of the present invention, the RGB frame and event frame corresponding to the detection area are obtained, and the detection is performed in combination with the texture information in the RGB frame and the edge information in the event data, so as to improve the detection accuracy. The RGB frame is input into the first backbone network for feature extraction to obtain K RGB sub-features of different scales, and the event frame is input into the second backbone network for feature extraction to obtain K event sub-features of different scales. In the preliminary fusion stage, the RGB sub-features and event sub-features of the same scale are fused to obtain K preliminary fusion features; in the cross-fusion stage, the K preliminary fusion features are cross-scale and cross-modal fusion processing is performed to obtain the target features; the target features are input into the decoder for decoding processing to obtain the classification and positioning of the detection object in the detection area. Through the fusion of the two stages, the information in the different scale features from the two heterogeneous modalities can be fully utilized to fully utilize the potential spatial relationship between the features at different hierarchical levels across the modalities, so that the detection results are more accurate and have better detection effects in the complex environment of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Attached Figure 1 One of the flowcharts of the lightweight end-to-end object detection method provided by an embodiment of the present invention;

[0042] Attached Figure 2a The second flowchart of the lightweight end-to-end object detection method provided by the embodiment of the present invention;

[0043] Attached Figure 2b For attachment Figure 2a Schematic diagram of the process of the initial fusion stage;

[0044] Attached Figure 2c For attachment Figure 2aSchematic diagram of the process of cross-fusion stage;

[0045] Attached Figure 2d For attachment Figure 2a Schematic diagram of the structure of the decoding stage;

[0046] Attached Figure 3 A schematic diagram of a lightweight end-to-end object detection device provided by an embodiment of the present invention;

[0047] Attached Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In the embodiments of the present application, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In the embodiments of the present application, the term "multiple" refers to two or more, and other quantifiers are similar. The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first" and "second" are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0051] The embodiments of the present application provide a lightweight end-to-end object detection method, device, electronic device and storage medium, which are intended to improve the accuracy of object detection in adverse environments. Figure 1 , Figure 1 The present invention provides a lightweight end-to-end object detection method, which specifically includes the following steps:

[0052] Step 101, obtaining the RGB frame and event frame corresponding to the detection area.

[0053] Step 102: Input the RGB frame into the first backbone network for feature extraction to obtain K RGB sub-features of different scales, and input the event frame into the second backbone network for feature extraction to obtain K event sub-features of different scales, where K is a positive integer.

[0054] Step 103: fuse the RGB sub-features and the event sub-features of the same scale to obtain K preliminary fused features.

[0055] Step 104: Perform cross-scale and cross-modal fusion processing on the K preliminary fusion features to obtain target features.

[0056] Step 105: input the target feature into a decoder for decoding processing to obtain the classification and location of the detection object in the detection area.

[0057] In the lightweight end-to-end object detection method provided in this embodiment, step 101 may be referred to as a data acquisition phase. Optionally, in some embodiments, step 101 includes:

[0058] Obtaining a video stream and a corresponding event stream of the detection area;

[0059] The corresponding RGB frame is acquired based on the video stream, and the event stream is compressed to obtain the event frame.

[0060] As an optional implementation, the video stream of the detection area and the event stream of the detection area can be obtained from different photographic devices respectively, wherein the video stream and the event stream correspond to the same time period. As another optional implementation, the video stream of the detection area and the corresponding event stream can be generated simultaneously by a DAVIS camera.

[0061] By processing the video stream, the video stream can be converted into multiple RGB frames (or RGB images). By compressing the event stream, multiple event frames (or event images) can be obtained. Optionally, in some embodiments, compressing the event stream to obtain the event frame includes:

[0062] Events occurring within a preset time in the event stream are mapped onto a two-dimensional plane to obtain the event frame.

[0063] Specifically, the event stream The four dimensions of represent the pixel position (x, y), the light polarity p and the timestamp t respectively. By mapping the events occurring within a certain period of time onto the XY plane, multiple event frames can be obtained.

[0064] In the lightweight end-to-end object detection method provided in this embodiment, step 102 can be called a feature extraction stage. In step 102, the RGB frame and the event frame are respectively input into the backbone network for feature extraction. Among them, the structure and parameters of the first backbone network and the second backbone network can be the same or different, and are not specifically limited here. As a specific embodiment, the first backbone network and the second backbone network are both backbone networks of ResNet-50.

[0065] like Figure 2a As shown in the figure, as a specific embodiment, in the feature extraction stage, the first backbone network (Backbone) is used to extract features from the RGB frame, and the obtained features are decomposed and arranged in order of different scales to obtain K RGB sub-features of different scales. Similarly, the second backbone network (Backbone) is used to extract features from the event frame, and the obtained features are decomposed and arranged in order of different scales to obtain K event sub-features of different scales.

[0066] In the lightweight end-to-end object detection method provided in this embodiment, step 103 may be referred to as a preliminary fusion stage, in which the RGB sub-features and event sub-features of the same scale are fused to obtain K preliminary fused features. Optionally, in some embodiments, for the convenience of description, the preliminary fused features obtained by fusing the RGB sub-features of the i-th scale and the event sub-features of the i-th scale are referred to as i-th fused features, where i is a positive integer less than or equal to K.

[0067] In some embodiments, the RGB sub-features and the event sub-features of the same scale are fused to obtain K preliminary fused features, including:

[0068] The RGB sub-feature at the first level scale is fused with the event sub-feature in a first manner to obtain a first fused feature, wherein the first level scale is the highest scale and the Kth level scale is the lowest scale;

[0069] The RGB sub-feature at the j-th scale is fused with the event sub-feature in a second manner to obtain a j-th fused feature, where j=2, 3, ..., K.

[0070] In this embodiment, the highest scale RGB sub-feature and the highest scale event sub-feature are fused in a first manner to obtain a first fused feature. Optionally, in some embodiments, the fusion of the RGB sub-feature of the first scale with the event sub-feature in a first manner to obtain a first fused feature includes:

[0071] Inputting the RGB sub-feature of the first-level scale into a first single-stage Transformer encoder for processing to obtain a target RGB sub-feature, and inputting the event sub-feature of the first-level scale into a second single-stage Transformer encoder for processing to obtain a target event sub-feature;

[0072] The target RGB sub-feature and the target event sub-feature are concatenated to obtain the first fusion feature.

[0073] In this embodiment, the RGB sub-features at the first scale are the highest scale features in the RGB modal data, and the event sub-features at the first scale are the highest scale features in the event modal data, so the RGB sub-features at the first scale and the event sub-features at the first scale contain very rich semantic information. In the above manner, by using a unipolar Transformer to process the largest scale features, richer semantic information can be obtained.

[0074] See also Figure 2b In the initial fusion stage, the RGB sub-features of the first-level scale and the event sub-features of the first-level scale are respectively input into the single-stage Transformer encoder for processing and then spliced. Among them, the structure of the first single-stage Transformer encoder and the second single-stage Transformer encoder are the same, and their parameters can be the same or different.

[0075] As a specific embodiment, the RGB sub-features of the first-level scale and the event sub-features of the first-level scale are respectively input into a single-stage Transformer encoder together with their corresponding Position Embeds. Position Embed is the position embedding in the Transformer model. Its specific structure can be found in the description in the relevant technology and is not limited here.

[0076] Except for the RGB sub-features and event sub-features of the highest scale, the RGB sub-features and event sub-features of other scales are fused respectively according to the second method. Optionally, in some embodiments, the RGB sub-features of the j-th scale are fused with the event sub-features according to the second method to obtain the j-th fused feature, including:

[0077] After splicing the RGB sub-features and the event sub-features at the j-th scale, the number of channels is adjusted and down-sampled using convolution to obtain the j-th fused feature.

[0078] In this embodiment, the same fusion method is adopted for all scales except the highest scale. Specifically, the RGB sub-features of the j-th scale and the event sub-features of the j-th scale are concatenated, input into a 3×3 convolution layer for convolution processing, and then input into a 1×1 convolution layer for convolution processing to obtain the j-th fused feature.

[0079] The low-scale features extracted by the backbone network contain a lot of noise and irrelevant information. In order to reduce complexity, in this embodiment, the highest-scale RGB sub-features and event sub-features are processed using a single-stage Transformer encoder, and features of other scales are roughly fused, reducing the number of Transformers, making the model parameters used in this method smaller and the inference speed faster.

[0080] As a specific example, see Figure 2b In this embodiment, the backbone of ResNet-50 is used as the first backbone network (Backbone) and the second backbone network (Backbone) to extract features from the RGB frame and the event frame respectively, and then the features are decomposed and arranged in order of different scales. Figure 2b In the initial fusion stage, the feature-scale decomposition spatial encoder (FSSE) performs feature decomposition and arrangement, and the highest-scale RGB sub-features are passed through the first single-stage Transformer encoder (Encoder f ) is processed, and the highest-scale event sub-features are processed by the second single-stage Transformer encoder (Encoder e ), the specific process can be characterized as:

[0081]

[0082] in, Used to represent the RGB frame X f and event frames Feature extraction and scale decomposition. RGB sub-features used to characterize the first-level scale, RGB sub-features used to characterize the second-level scale, RGB sub-features used to characterize the third-level scale, Sub-features used to characterize events at level 1, Sub-features used to characterize events at level 2, Used to characterize event sub-features at level 3 scale.

[0083]

[0084] Among them, Flatten() is used to indicate that the features are flattened so that the features can be input into the Transformer encoder. f The RGB sub-features used to represent the query in the Transformer encoder are K f Key, V, used to represent the RGB sub-features of level 1 input into the Transformer encoder f The RGB sub-features used to represent the value of the first level input to the Transformer encoder, Q e The query K used to represent the event sub-features of level 1 is input into the Transformer encoder e The key used to characterize the event sub-features of level 1 and input into the Transformer encoder, V e The value used to represent the event sub-features at level 1 input into the Transformer encoder.

[0085]

[0086] in, is the first fusion feature, and E represents the Transformer encoder. In the Transformer encoder, MSA is a multi-head self-attention module. The specific structure of the Transformer encoder can be found in the description of the relevant technology, which will not be repeated here.

[0087] The fusion process of RGB sub-features and event sub-features at the second and third scales can be characterized as follows:

[0088]

[0089] in, is the second fusion feature, For the third fusion feature, Reshape() is used to represent the adjustment of the number of channels and use convolution for downsampling, and Concat() is used to represent feature concatenation.

[0090] In the lightweight end-to-end object detection method provided in this embodiment, step 104 can be called a cross fusion stage. Figure 2c In the specific embodiment shown, the value of K is 3, and the K preliminary fusion features include the first fusion feature Second fusion feature and the third fusion feature The step 104 includes:

[0091] Fusing the first fused feature and the second fused feature to obtain a first high-order feature;

[0092] Fusing the first high-order feature and the third fused feature to obtain a first low-order feature;

[0093] Fusing the first low-order feature with the first high-order feature to obtain a second low-order feature;

[0094] Fusing the second low-order feature with the first fused feature to obtain a second high-order feature;

[0095] The second low-order feature and the second high-order feature are concatenated to obtain the target feature.

[0096] like Figure 2c As shown in Figure 1, in the cross-fusion stage, the two-stage multi-modal cross-scale fusion module (MCFM) uses an interactive structure to fuse features from different modalities and scales. Specifically, the three preliminary fusion features are refined to obtain the final target features. Figure 2c As shown, the High-scale stage first aggregates features The low-scale stage will The output of the high-scale stage is fused together. The two stages further interact with each other to obtain the target features, which can be specifically expressed as:

[0097]

[0098]

[0099] in, is the first high-order feature, is the second highest-order feature, is the first low-order feature, is the second lowest-order feature, is the target feature.

[0100] In the lightweight end-to-end object detection method provided in this embodiment, step 105 may be referred to as a decoding stage. Figure 2dIn step 105, the target features are input into the decoder for decoding to obtain the prediction results, including the classification and positioning of the detection object in the detection area. Among them, DMSA is a deformable multi-head self-attention module, ObjectQueries is a set of learnable vectors used to represent each target predicted by the model; Ground Truth Boxes is the real target detection box of the data set; Fused Features is the target feature output by MCFM, and CDN is a contrast denoising training method that improves the training efficiency and detection performance of the model by introducing positive and negative samples. The specific structure of the decoder and the specific process of the decoding stage can be found in the description of the relevant technology, which will not be repeated here.

[0101] In the method provided in the embodiment of the present application, only the highest-scale RGB sub-features and event sub-features are subjected to global feature extraction, so the parameters of the model can be greatly reduced and the reasoning speed can be improved. In addition, the model in the present method can be trained in an end-to-end manner without the need to split and freeze each module for separate training. The end-to-end training method enables the method provided in the present application to be effectively integrated in the aligned feature potential space.

[0102] like Figure 2a-2d As shown, an embodiment of the present application also provides a lightweight end-to-end object detection model with Event-RGB fusion (abbreviated as EvR-DETR) for implementing the present method, including the above-mentioned FSSE, MCFM and decoder. According to the foregoing content, in an embodiment of the present application, compared with the design of only fusing the highest-scale features in the prior art SODFormer (the most advanced (SOTA) method on the PKU-SOD dataset), the method provided in the embodiment of the present application fuses features of multiple different scales, and can effectively learn information of different modalities and scales. Furthermore, since the method provided in the present application only applies the Transformer to the highest-level scale features of each modality, rather than using it in all stages, the model provided in the present application does not include a large number of Transformers, which effectively reduces the number of parameters of the model provided in the present application.

[0103] In addition, due to the large number of parameters in SODFormer, the SODFormer trains each module separately and then freezes the parameters to train the fusion module, which may cause the feature extractor to be unable to extract the most suitable features for fusion. According to the experimental results, compared with the SODFormer in the prior art, the model provided by the present application reduces the number of parameters by 50%, and although the number of parameters of EvR-DETR is only half of that of SODFormer, EvR-DETR can still achieve a higher accuracy rate. It can be seen that compared with the prior art, the method provided in the embodiment of the present application improves speed without sacrificing performance.

[0104] like Figure 3 As shown, the embodiment of the present invention provides a lightweight end-to-end object detection device 300, including:

[0105] An acquisition module 301 is used to acquire an RGB frame and an event frame corresponding to a detection area;

[0106] A feature extraction module 302 is used to input the RGB frame into the first backbone network for feature extraction to obtain K RGB sub-features of different scales, and input the event frame into the second backbone network for feature extraction to obtain K event sub-features of different scales, where K is a positive integer;

[0107] A preliminary fusion module 303 is used to fuse the RGB sub-features and the event sub-features of the same scale to obtain K preliminary fusion features;

[0108] A cross-fusion module 304 is used to perform cross-scale and cross-modal fusion processing on the K preliminary fusion features to obtain target features;

[0109] The decoding module 305 is used to input the target feature into a decoder for decoding processing to obtain the classification and location of the detection object in the detection area.

[0110] Optionally, the preliminary fusion feature obtained by fusing the RGB sub-feature and the event sub-feature at the i-th scale is the i-th fusion feature, i is a positive integer less than or equal to K, and the preliminary fusion module 303 includes:

[0111] A first fusion unit, configured to fuse the RGB sub-feature at the first level of scale with the event sub-feature in a first manner to obtain a first fused feature, wherein the first level of scale is the highest scale and the Kth level of scale is the lowest scale;

[0112] The second fusion unit is used to fuse the RGB sub-feature of the j-th scale with the event sub-feature in a second manner to obtain a j-th fused feature, where j=2, 3, ..., K.

[0113] Optionally, the first fusion unit is specifically used for:

[0114] Inputting the RGB sub-feature of the first-level scale into a first single-stage Transformer encoder for processing to obtain a target RGB sub-feature, and inputting the event sub-feature of the first-level scale into a second single-stage Transformer encoder for processing to obtain a target event sub-feature;

[0115] The target RGB sub-feature and the target event sub-feature are concatenated to obtain the first fusion feature.

[0116] Optionally, the second fusion unit is specifically used for:

[0117] After splicing the RGB sub-features and the event sub-features at the j-th scale, the number of channels is adjusted and down-sampled using convolution to obtain the j-th fused feature.

[0118] Optionally, the value of K is 3, the K preliminary fusion features include the first fusion feature, the second fusion feature and the third fusion feature, and the cross fusion module 304 is specifically used for:

[0119] Fusing the first fused feature and the second fused feature to obtain a first high-order feature;

[0120] Fusing the first high-order feature and the third fused feature to obtain a first low-order feature;

[0121] Fusing the first low-order feature with the first high-order feature to obtain a second low-order feature;

[0122] Fusing the second low-order feature with the first fused feature to obtain a second high-order feature;

[0123] The second low-order feature and the second high-order feature are concatenated to obtain the target feature.

[0124] Optionally, the acquisition module 301 includes:

[0125] An acquisition unit, used to acquire a video stream and a corresponding event stream of the detection area;

[0126] A processing unit is used to obtain the corresponding RGB frame based on the video stream, and to compress the event stream to obtain the event frame.

[0127] Optionally, compressing the event stream to obtain the event frame includes:

[0128] Events occurring within a preset time in the event stream are mapped onto a two-dimensional plane to obtain the event frame.

[0129] The lightweight end-to-end object detection device 300 provided in the embodiment of the present application can execute the above-mentioned method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here. The lightweight end-to-end object detection device 300 provided in this embodiment uses event data and RGB data for cross-modal and cross-scale fusion, effectively utilizing the texture information in the RGB frame and the edge information in the event frame. Compared with the unimodal benchmark model (a model that only uses event data or RGB data), the lightweight end-to-end object detection device 300 provided in this embodiment significantly improves the detection accuracy in extreme scenarios of autonomous driving. Compared with the multimodal benchmark test method (such as SODFormer), the lightweight end-to-end object detection device 300 provided in this embodiment reduces the number of parameters and improves the detection accuracy and reasoning speed.

[0130] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0131] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0132] like Figure 4As shown, an embodiment of the present application provides an electronic device 400, including: a memory 402, a processor 401, and a program stored in the memory 402 and executable on the processor 401; the processor 401 is used to read the program in the memory 402 to implement the steps in the lightweight end-to-end object detection method as described above.

[0133] An embodiment of the present application also provides a readable storage medium, on which a program is stored. When the program is executed by a processor, the various processes of the above-mentioned lightweight end-to-end object detection method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here. Among them, the readable storage medium can be any available medium or data storage device that can be accessed by the processor, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO), etc.), optical storage (such as compact disk (CD), digital video disc (DVD), Blu-ray Disc (BD), high-definition versatile disc (HVD), etc.), and semiconductor memory (such as read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read only memory (EEPROM), non-volatile memory (NAND FLASH), solid-state drive (Solid State Disk or Solid State Drive, SSD)), etc.

[0134] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0135] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0136] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A lightweight end-to-end object detection method, characterized in that: include: Get the RGB frame and event frame corresponding to the detection area; Inputting the RGB frame into the first backbone network for feature extraction to obtain K RGB sub-features of different scales, and inputting the event frame into the second backbone network for feature extraction to obtain K event sub-features of different scales, where K is a positive integer; Fusing the RGB sub-features and the event sub-features of the same scale to obtain K preliminary fused features; Performing cross-scale and cross-modal fusion processing on the K preliminary fusion features to obtain target features; The target features are input into a decoder for decoding processing to obtain the classification and positioning of the detection object in the detection area.

2. The method according to claim 1, characterized in that: The preliminary fused feature obtained by fusing the RGB sub-features and the event sub-features at the i-th scale is the i-th fused feature, where i is a positive integer less than or equal to K. The RGB sub-features and the event sub-features at the same scale are fused to obtain K preliminary fused features, including: The RGB sub-feature at the first level scale is fused with the event sub-feature in a first manner to obtain a first fused feature, wherein the first level scale is the highest scale and the Kth level scale is the lowest scale; The RGB sub-feature at the j-th scale is fused with the event sub-feature in a second manner to obtain a j-th fused feature, where j=2, 3, ..., K.

3. The method according to claim 2, characterized in that: The step of fusing the RGB sub-feature at the first level scale with the event sub-feature in a first manner to obtain a first fused feature includes: Inputting the RGB sub-feature of the first-level scale into a first single-stage Transformer encoder for processing to obtain a target RGB sub-feature, and inputting the event sub-feature of the first-level scale into a second single-stage Transformer encoder for processing to obtain a target event sub-feature; The target RGB sub-feature and the target event sub-feature are concatenated to obtain the first fusion feature.

4. The method according to claim 2, characterized in that: The step of fusing the RGB sub-feature at the j-th scale with the event sub-feature in a second manner to obtain the j-th fused feature includes: After splicing the RGB sub-features and the event sub-features at the j-th scale, the number of channels is adjusted and down-sampled using convolution to obtain the j-th fused feature.

5. The method according to claim 2, characterized in that: The value of K is 3, the K preliminary fusion features include the first fusion feature, the second fusion feature and the third fusion feature, and the cross-scale and cross-modal fusion processing is performed on the K preliminary fusion features to obtain the target feature, including: Fusing the first fused feature and the second fused feature to obtain a first high-order feature; Fusing the first high-order feature and the third fused feature to obtain a first low-order feature; Fusing the first low-order feature with the first high-order feature to obtain a second low-order feature; Fusing the second low-order feature with the first fused feature to obtain a second high-order feature; The second low-order feature and the second high-order feature are concatenated to obtain the target feature.

6. The method according to claim 1, characterized in that: The obtaining of the RGB frame and the event frame corresponding to the detection area includes: Obtaining a video stream and a corresponding event stream of the detection area; The corresponding RGB frame is acquired based on the video stream, and the event stream is compressed to obtain the event frame.

7. The method according to claim 6, characterized in that: The compressing the event stream to obtain the event frame includes: Events occurring within a preset time in the event stream are mapped onto a two-dimensional plane to obtain the event frame.

8. A lightweight end-to-end object detection device, characterized in that: include: An acquisition module is used to acquire the RGB frame and event frame corresponding to the detection area; A feature extraction module, used to input the RGB frame into the first backbone network for feature extraction to obtain K RGB sub-features of different scales, and input the event frame into the second backbone network for feature extraction to obtain K event sub-features of different scales, where K is a positive integer; A preliminary fusion module, used to fuse the RGB sub-features and the event sub-features of the same scale to obtain K preliminary fusion features; A cross-fusion module is used to perform cross-scale and cross-modal fusion processing on the K preliminary fusion features to obtain target features; The decoding module is used to input the target features into a decoder for decoding processing to obtain the classification and positioning of the detection object in the detection area.

9. An electronic device, comprising: A memory, a processor, and a program stored in the memory and executable on the processor; wherein the processor is used to read the program in the memory to implement the steps in the lightweight end-to-end object detection method as described in any one of claims 1 to 7.

10. A readable storage medium for storing a program, characterized in that: When the program is executed by a processor, the steps in the lightweight end-to-end object detection method according to any one of claims 1 to 7 are implemented.

Images