Transform-based multi-modal fusion BEV target detection system
Through a multimodal fusion BEV target detection system based on Transformer, using DSVT and Swin-Transformer and other technologies to extract and fusion multiple sensor data, the problems of high computational complexity and poor detection effect of existing systems are solved, and the problem of high performance and poor detection effect is achieved. More robust object detection and adaptability.
Patent Information
- Application Number
- CN202410972891.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-19
- Publication Date
- 2024-12-13
AI Technical Summary
The existing multimodal fusion BEV target detection system has problems such as high computational complexity, poor real-time performance, and inability to fully utilize multiple sensor information, and the detection effect is poor in harsh environments.
Using a multimodal fusion BEV target detection system based on Transformer, including sensor feature extraction module, feature fusion encoding module and task decoding module, the feature extraction and fusion of multiple sensor data through DSVT algorithm and Swin-Transformer model, using AIFI and The CCFM module performs feature processing and uses DINO's denoising training algorithm to accelerate the model's convergence.
It realizes more robust multi-sensor data fusion, improves the accuracy and robustness of target detection, adapts to different environmental conditions, reduces information conflicts and inconsistencies, and improves detection effect.
Smart Images
Figure CN119128615A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of autonomous driving, and in particular to a transformer-based multimodal fusion BEV target detection system. Background Art
[0002] With the rapid development of autonomous driving, intelligent transportation and other fields, on-board sensors and computer vision technology play an increasingly important role in vehicle environmental perception. The main task of vehicle environmental perception is to monitor the surrounding environment in real time in order to provide accurate and timely decision support for the vehicle. Among them, Bird's Eye View (BEV) target detection, as an important technical means, can form a bird's-eye view around the vehicle and convert three-dimensional spatial information into two-dimensional image information, which is convenient for processing and analysis. Traditional BEV target detection methods are mainly based on single-modal data, such as using only camera data or lidar data for target detection. However, in practical applications, single-modal data often has limitations and cannot fully utilize the complementary information of other modal data. For example, lidar data has better performance in dark environments or bad weather, but camera data has more advantages in obtaining texture information. Therefore, multimodal fusion BEV target detection technology came into being, which improves the accuracy and robustness of target detection by fusing multiple sensor data. At present, multimodal fusion BEV target detection technology is still in the research stage, and some related research results have been achieved. Among them, most methods focus on how to effectively fuse data of different modalities, such as feature-level fusion, decision-level fusion, etc. However, these methods still have certain limitations in practical applications, such as high computational complexity and poor real-time performance. In addition, although some existing technical solutions can achieve the fusion of multimodal data, they still need to be improved in terms of target detection accuracy and robustness.
[0003] In the prior art, the complex road target detection method based on the multimodal fusion bird's-eye view obtains multi-view camera images and lidar point clouds from different actual driving scenes, extracts features from the images and point clouds respectively, and maps them into dense vectors in the bird's-eye view space, obtains the heat map and point cloud feature vector of the image and point cloud fusion, realizes the query vector initialization, calculates the similarity between the query vector and the image feature vector using the spatial multi-channel attention mechanism, and inputs the query vector at the position with the strongest correlation into the feedforward neural network, predicts the center coordinates, three-dimensional size and yaw angle of the target to form a bounding box; establishes a target detection model based on the collected data set to detect vehicles on the road. The present invention only supports the fusion of cameras and lidars. The multimodal collaborative perception method and system for autonomous driving based on BEV perspective include: obtaining original point cloud data and multi-view camera image data; voxelizing the original point cloud data and extracting features to obtain point cloud BEV features; extracting features of multi-view camera image data and performing forward-view feature conversion to obtain image BEV features; performing depth completion and perspective conversion on the sparse depth map corresponding to the original point cloud data and the multi-view camera image data to obtain depth BEV features; performing feature fusion on each BEV feature to obtain multimodal BEV fusion features; and processing the multimodal BEV fusion features through a multi-task head to obtain three-dimensional target detection results and BEV perspective semantic segmentation results. This method relies on converting features into BEV features in the bird's-eye view space, so it is not effective for sparse data fusion such as millimeter-wave radar.
[0004] There are some problems with the existing multimodal fusion BEV target detection system. First, traditional detection systems mostly use manually designed features, which are difficult to adapt to different scenarios and changing environmental conditions. Second, existing systems can often only process data of a single modality, such as images or lidar data, and cannot fully utilize the information provided by multiple sensors. In addition, when processing multimodal data, existing systems often have information conflicts and inconsistencies between modalities, resulting in poor detection results.
[0005] Therefore, one or more methods are needed to solve the above problems.
[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0007] The purpose of the present disclosure is to provide a transformer-based multimodal fusion BEV target detection system, thereby overcoming one or more problems caused by the limitations and defects of related technologies at least to a certain extent.
[0008] According to one aspect of the present disclosure, a transformer-based multimodal fusion BEV target detection system is provided, including a sensor feature extraction module, a feature fusion encoding module, and a task decoding module, wherein:
[0009] The sensor feature extraction module includes a point cloud feature extraction network and an image feature extraction network, and the sensor feature extraction module is used to extract features from detection signals of multimodal sensors;
[0010] The feature fusion encoding module is used to fuse the features of each sensor extracted by the sensor feature extraction module to generate an image feature sequence;
[0011] The task decoding module is used to take the query vector and the position code as input, decode the image feature sequence, and generate a retrieval result.
[0012] In an exemplary embodiment of the present disclosure, the point cloud feature extraction network of the sensor feature extraction module of the system is used to extract features from the laser radar point cloud and the millimeter wave radar point cloud respectively;
[0013] The image feature extraction network of the sensor feature extraction module of the system is used to extract features from camera images;
[0014] The sensor feature extraction module of the system also includes other sensor data feature extraction networks, and the other sensor data feature extraction networks are used to extract features from other sensor data.
[0015] In an exemplary embodiment of the present disclosure, the point cloud feature extraction network of the sensor feature extraction module of the system performs feature extraction on the laser radar point cloud and the millimeter wave radar point cloud based on the DSVT algorithm.
[0016] In an exemplary embodiment of the present disclosure, the DSVT algorithm implements parallel computing by segmenting a sparse point cloud into multiple local regions and dividing each window into a series of subsets with the same number of sparse voxels.
[0017] In an exemplary embodiment of the present disclosure, the DSVT algorithm is based on a rotation set partitioning strategy, and alternately uses two different partitioning configurations in consecutive self-attention layers to achieve downsampling of sparse voxels and encoding of geometric information.
[0018] In an exemplary embodiment of the present disclosure, the image feature extraction network of the sensor feature extraction module of the system is a Swin-Transformer visual model based on a visual model of the Transformer architecture.
[0019] In an exemplary embodiment of the present disclosure, the Swin-Transformer visual model of the system applies a self-attention mechanism within a local window and allows the model to share information between different regions.
[0020] In an exemplary embodiment of the present disclosure, the feature fusion encoding module of the system uses a transformer encoder to embed the features of different modes and scales of each sensor extracted by the sensor feature extraction module into a feature fusion encoder for embedding vector learning and feature fusion.
[0021] In an exemplary embodiment of the present disclosure, the feature fusion encoding module of the system implements high-level feature processing in the image through attention-based feature processing of the AIFI module, and realizes intra-scale feature interaction of each feature of each sensor;
[0022] The feature fusion coding module of the system realizes cross-scale feature fusion of various features of each sensor through the CCFM module.
[0023] In an exemplary embodiment of the present disclosure, the system accelerates the convergence of the model based on the denoising training algorithm in DINO.
[0024] In an exemplary embodiment of the present disclosure, a transformer-based multimodal fusion BEV target detection system is disclosed, the system includes a sensor feature extraction module, a feature fusion encoding module, and a task decoding module, wherein: the sensor feature extraction module includes a point cloud feature extraction network and an image feature extraction network, the sensor feature extraction module is used to extract features from the detection signal of the multimodal sensor; the feature fusion encoding module is used to fuse the various features of each sensor extracted by the sensor feature extraction module to generate an image feature sequence; the task decoding module is used to decode the image feature sequence with the query vector and position code as input to generate a retrieval result. The present disclosure solves the problem of difficult convergence of model training in Transformer through a new denoising training algorithm, and can fuse data from multiple sensors to perform more robust target detection.
[0025] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] The above and other features and advantages of the present disclosure will become more apparent by describing in detail example embodiments thereof with reference to the attached drawings.
[0027] Figure 1A schematic block diagram of a transformer-based multimodal fusion BEV target detection system according to an exemplary embodiment of the present disclosure is shown;
[0028] Figure 2 The figure shows an overall block diagram of a technical solution of a transformer-based multimodal fusion BEV target detection system according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0029] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be comprehensive and complete and will fully convey the concepts of the example embodiments to those skilled in the art. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted.
[0030] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, materials, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations, materials or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.
[0031] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.
[0032] In this exemplary embodiment, a transformer-based multimodal fusion BEV target detection system 100 is first provided; Figure 1 As shown in , the transformer-based multimodal fusion BEV target detection system 100 includes a sensor feature extraction module, a feature fusion encoding module, and a task decoding module, wherein:
[0033] The sensor feature extraction module includes a point cloud feature extraction network and an image feature extraction network, and the sensor feature extraction module is used to extract features from detection signals of multimodal sensors;
[0034] The feature fusion encoding module is used to fuse the features of each sensor extracted by the sensor feature extraction module to generate an image feature sequence;
[0035] The task decoding module is used to take the query vector and the position code as input, decode the image feature sequence, and generate a retrieval result.
[0036] A transformer-based multimodal fusion BEV target detection system 100 in an exemplary embodiment of the present disclosure ， The system includes a sensor feature extraction module, a feature fusion encoding module, and a task decoding module, wherein: the sensor feature extraction module includes a point cloud feature extraction network and an image feature extraction network, and the sensor feature extraction module is used to extract features from detection signals of multimodal sensors; the feature fusion encoding module is used to fuse the features of each sensor extracted by the sensor feature extraction module to generate an image feature sequence; the task decoding module is used to decode the image feature sequence with a query vector and a position code as input to generate a retrieval result. The present disclosure solves the problem of difficult convergence of model training in Transformer through a new denoising training algorithm, and can fuse data from multiple sensors for more robust target detection.
[0037] Next, a transformer-based multimodal fusion BEV target detection system 100 in this example embodiment will be further described.
[0038] Embodiment 1:
[0039] The transformer-based multimodal fusion BEV target detection system 100 includes a sensor feature extraction module, a feature fusion encoding module, and a task decoding module, wherein:
[0040] The sensor feature extraction module includes a point cloud feature extraction network and an image feature extraction network, and the sensor feature extraction module is used to extract features from detection signals of multimodal sensors.
[0041] In the embodiment of this example, the point cloud feature extraction network of the sensor feature extraction module of the system is used to extract features from the laser radar point cloud and the millimeter wave radar point cloud respectively;
[0042] The image feature extraction network of the sensor feature extraction module of the system is used to extract features from camera images;
[0043] In the embodiment of this example, the point cloud feature extraction network of the sensor feature extraction module of the system performs feature extraction on the laser radar point cloud and the millimeter wave radar point cloud based on the DSVT algorithm.
[0044] The feature fusion encoding module is used to fuse the features of each sensor extracted by the sensor feature extraction module to generate an image feature sequence.
[0045] The task decoding module is used to take the query vector and the position code as input, decode the image feature sequence, and generate a retrieval result.
[0046] In the embodiment of this example, the DSVT algorithm implements parallel computing by segmenting the sparse point cloud into multiple local areas and dividing each window into a series of subsets with the same number of sparse voxels.
[0047] In the embodiment of this example, the DSVT algorithm is based on a rotation set partitioning strategy, which alternates between two different partitioning configurations in consecutive self-attention layers to achieve downsampling of sparse voxels and encoding of geometric information.
[0048] In the embodiment of this example, the image feature extraction network of the sensor feature extraction module of the system is a Swin-Transformer visual model based on a visual model of the Transformer architecture.
[0049] In an embodiment of this example, the Swin-Transformer visual model of the system applies a self-attention mechanism within a local window and allows the model to share information between different regions.
[0050] In the embodiment of this example, the feature fusion encoding module of the system uses a transformer encoder to embed the features of different modes and scales of each sensor extracted by the sensor feature extraction module into a feature fusion encoder for embedding vector learning and feature fusion.
[0051] In the embodiment of this example, the feature fusion encoding module of the system implements high-level feature processing in the image through the attention-based feature processing of the AIFI module, and realizes the intra-scale feature interaction of each feature of each sensor;
[0052] The feature fusion coding module of the system realizes cross-scale feature fusion of various features of each sensor through the CCFM module.
[0053] The sensor feature extraction module of the system also includes other sensor data feature extraction networks, and the other sensor data feature extraction networks are used to extract features from other sensor data.
[0054] In the embodiment of this example, the system accelerates the convergence of the model based on the denoising training algorithm in DINO.
[0055] Embodiment 2:
[0056] like Figure 2 As shown, the transformer-based multimodal fusion BEV target detection system 100 includes a sensor feature extraction module, a feature fusion encoding module, and a task decoding module, wherein:
[0057] In the embodiment of this example, the sensor feature extraction module performs feature extraction for different sensor data. Among them, the point cloud feature extraction network selects the DSVT algorithm. The DSVT (Dynamic Sparse Voxel Transformer) algorithm is an efficient and easy-to-deploy network architecture for point cloud feature extraction, which is particularly suitable for 3D target detection tasks of laser radar (LiDAR) data. The algorithm improves the efficiency and accuracy of processing large-scale point cloud data by combining the attention mechanism of Transformer and the specialized strategy designed for point cloud sparsity. The DSVT algorithm achieves efficient parallel computing by dividing the sparse point cloud into multiple local areas and dividing a series of subsets with the same number of sparse voxels in each window. This method allows the network to dynamically adjust computing resources according to the sparsity in each window, thereby improving processing efficiency. In order to allow cross-set connections, DSVT designs a rotation set partitioning strategy that alternates between two different partition configurations in consecutive self-attention layers. This strategy not only enhances the modeling ability of the model, but also introduces feature propagation within and between windows while maintaining efficient computing. DSVT proposes a learnable 3D pooling operation for effectively downsampling sparse voxels and encoding geometric information. Compared with traditional sparse convolution-based methods, this pooling module is powerful and easy to deploy, without any customized CUDA operations, which is conducive to efficient computing on modern GPUs. The image feature extraction network selects Swin-Transformer. The Swin-Transformer algorithm is a visual model based on the Transformer architecture, designed for image recognition and processing tasks. Swin-Transformer adopts a hierarchical design, in which the image is divided into windows of different sizes, and the pixels within each window are processed by a self-attention mechanism. This hierarchical approach enables the model to capture features of different scales, thereby better understanding the content of the image. Unlike the traditional Transformer model, Swin-Transformer applies a self-attention mechanism within a local window instead of on the entire image. This approach reduces the amount of computation while still being able to capture long-distance dependencies. In order to process windows at different positions, Swin-Transformer introduces a sliding window mechanism. This allows the model to share information between different regions and enhances the global representation of features. Swin-Transformer forms a hierarchical feature representation by gradually reducing the resolution of feature maps. At each stage, the model generates a lower-resolution feature map, which is used to capture more abstract information in subsequent stages. Swin-Transformer can be trained end-to-end without the need for additional pre-training or complex data augmentation strategies. This makes the model training process more efficient and straightforward.Swin-Transformer performs well in multiple computer vision tasks, including image classification, object detection, and semantic segmentation. Its hierarchical feature representation and effective self-attention mechanism give it an advantage in processing large-scale image data, especially in scenarios that need to capture complex spatial relationships. Other sensors can use feature extraction networks of corresponding modalities as needed.
[0058] In the embodiment of this example, the feature fusion encoder adopts the classic transformer encoder. Features of different modalities and scales are embedded in the feature fusion encoder for vector learning and feature fusion. We use the AIFI module and CCFM module in RT-DETR to efficiently process multi-scale features by decoupling intra-scale interactions and cross-scale fusion, thereby optimizing the initialization of decoder queries and supporting flexible adjustment of inference speed by using different decoder layers. The core idea of the AIFI module is attention-based feature processing, which uses the self-attention mechanism to process high-level features in images. The self-attention mechanism allows the model to take into account other relevant parts of the data while processing a specific part of the data, which is particularly effective for processing high-level image features with rich semantic information. The main functions and features of the AIFI module include reducing computational redundancy, capturing the connection between conceptual entities in the image by applying self-attention operations to high-level features with rich semantic concepts, and thus helping subsequent modules to more effectively detect and recognize objects in the image. The CCFM module is designed to effectively integrate features of different scales, including detail features and contextual information, thereby improving the overall performance of the model. The CCFM module receives multi-scale features from the encoder and transforms them into image feature sequences through intra-scale feature interaction (AIFI) and cross-scale feature fusion (CCFM). These feature sequences are then used for the initial object query of the decoder.
[0059] In this example, the decoder uses the decoder in the Transformer. We use a learnable positional encoding to add to the query vector, which is then input into the decoder, and the output of the feature fusion encoder is input into the decoder as a key-value pair. After being decoded by this transformer, the decoded output is obtained, and finally the detection result is obtained through a decoding head.
[0060] In the embodiment of this example, during training, we used the denoising training algorithm in DINO to accelerate the convergence of the model. DINO designed a method for the model to identify negative samples in denoising training. By comparing and learning the negative sample anchor box with the positive sample with less noise, the model not only learns to choose a better anchor box, but also reduces repeated predictions. At the same time, DINO selects some of the most obvious image features and uses their positions as the initialization of the anchor box, further accelerating the convergence of the model.
[0061] It should be noted that although several modules or units of a transformer-based multimodal fusion BEV target detection system 100 are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.
[0062] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.
[0063] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.
[0064] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A transformer-based multimodal fusion BEV target detection system, characterized in that: The system includes a sensor feature extraction module, a feature fusion encoding module, and a task decoding module, wherein: The sensor feature extraction module includes a point cloud feature extraction network and an image feature extraction network, and the sensor feature extraction module is used to extract features from detection signals of multimodal sensors; The feature fusion encoding module is used to fuse the features of each sensor extracted by the sensor feature extraction module to generate an image feature sequence; The task decoding module is used to take the query vector and the position code as input, decode the image feature sequence, and generate a retrieval result.
2. The system according to claim 1, characterized in that The point cloud feature extraction network of the sensor feature extraction module of the system is used to extract features from the laser radar point cloud and the millimeter wave radar point cloud respectively; The image feature extraction network of the sensor feature extraction module of the system is used to extract features from camera images; The sensor feature extraction module of the system also includes other sensor data feature extraction networks, and the other sensor data feature extraction networks are used to extract features from other sensor data.
3. The system according to claim 2, characterized in that The point cloud feature extraction network of the sensor feature extraction module of the system extracts features of the laser radar point cloud and the millimeter wave radar point cloud based on the DSVT algorithm.
4. The system according to claim 1, characterized in that The DSVT algorithm achieves parallel computing by dividing the sparse point cloud into multiple local areas and dividing a series of subsets with the same number of sparse voxels in each window.
5. The system according to claim 4, characterized in that The DSVT algorithm is based on a rotation set partitioning strategy, which alternates between two different partitioning configurations in consecutive self-attention layers to downsample sparse voxels and encode geometric information.
6. The system according to claim 1, wherein: The image feature extraction network of the sensor feature extraction module of the system is a Swin-Transformer visual model based on a visual model of the Transformer architecture.
7. The system according to claim 1, characterized in that The Swin-Transformer vision model of the system applies a self-attention mechanism within a local window and allows the model to share information between different regions.
8. The system of claim 1, wherein: The feature fusion encoding module of the system is based on a transformer encoder, which embeds the features of different modes and scales of each sensor extracted by the sensor feature extraction module into a feature fusion encoder for embedding vector learning and feature fusion.
9. The system according to claim 8, characterized in that The feature fusion encoding module of the system implements high-level feature processing in the image through the attention-based feature processing of the AIFI module, and realizes the intra-scale feature interaction of each feature of each sensor; The feature fusion coding module of the system realizes cross-scale feature fusion of various features of each sensor through the CCFM module.
10. The system according to claim 8, characterized in that The system accelerates the convergence of the model based on the denoising training algorithm in DINO.