Multi-mode self-adaptive fusion automatic driving perception method

Abstract

The invention discloses a multi-mode adaptive fusion automatic driving perception method. The method comprises the following steps: acquiring a public picture data set of a road scene; and inputting the public picture data set into a constructed multi-mode adaptive fusion model to obtain an automatic driving perception result. Through the adaptive selection and fusion strategy, the robustness of the automatic driving perception system under the condition of sensor fault or mode missing is improved, and the perception performance in a complex scene is optimized.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal sensor fusion in autonomous driving, and in particular relates to an autonomous driving perception method with multimodal adaptive fusion. Background Art

[0002] With the rapid development of autonomous driving technology, the accuracy and stability of the environmental perception system, as a key part of achieving safe driving, are particularly important. In order to obtain comprehensive environmental information, current autonomous driving perception systems usually use multiple types of sensors, including LiDAR, cameras, and millimeter-wave radars (Radar). These sensors lay the foundation for efficient and stable perception tasks by providing environmental information in different dimensions. However, effectively fusing data from different sensors to improve overall perception performance is still a key research problem in the current field of autonomous driving technology.

[0003] Existing multimodal fusion technologies are mainly divided into three types: data-level fusion, feature-level fusion, and decision-level fusion. Data-level fusion directly merges the raw data of sensors, which can preserve the integrity of the data to a certain extent. However, due to its extremely high requirements on the accuracy and synchronization of sensor data and its high computational complexity, it faces many challenges in practical applications. Feature-level fusion extracts features from different sensors for fusion, which can reduce the computational burden, but it is prone to deviation when aligning features of different modalities, and it relies heavily on the complementarity between modalities. Decision-level fusion is to make a comprehensive decision on the results after each modality completes an independent perception task. Although the method is simple, it is difficult to ensure perception accuracy in complex scenes. These fusion strategies mostly rely on traditional convolutional neural networks (CNNs) and feature concatenation technology. However, CNN has obvious limitations in long-distance target feature alignment and dynamic adaptive attention allocation, which makes it difficult to accurately align the fused multimodal features, thereby causing feature misalignment problems and affecting the final perception effect.

[0004] In addition, existing multimodal fusion technologies usually adopt a rigid fusion strategy, which is fixedly dependent on preset sensor combinations and data types. Although this method performs well when the sensor functions are complete, it lacks adaptability when the sensor fails or the data modality is missing, and the perception performance may drop significantly, or even cause the entire perception system to fail. In complex dynamic scenarios (such as night, rain, snow and other bad weather), traditional rigid fusion strategies lack flexible adjustment capabilities and are difficult to meet the needs of autonomous driving systems for high robustness and high-precision perception.

[0005] Existing multimodal fusion systems generally lack the ability to adapt to atypical scenarios. For example, in the dark, low light or extreme weather conditions, sensor performance will drop significantly, and environmental noise will interfere with the perception results, making it difficult for traditional methods to effectively adjust the weights of each sensor to optimize system performance. In addition, the differences in the advantages of different sensors in specific scenarios make the optimization of fusion strategies particularly important. For example, cameras perform well in capturing semantic and color information, but their performance is significantly reduced under extremely dark or bright light conditions; LiDARs are outstanding in spatial positioning and depth information acquisition, but are easily disturbed by weather such as rain and fog; millimeter-wave radars have strong anti-interference capabilities in bad weather, but their resolution and accuracy are lower than other sensors. Therefore, how to dynamically adjust the combination and weights of sensors according to real-time scenarios to optimize the overall perception performance has become an important research direction for multimodal fusion technology.

[0006] In general, the existing technology has the following deficiencies:

[0007] Insufficient adaptability to sensor failures: Most methods cannot flexibly adjust the fusion strategy when the sensor modality is missing, resulting in poor system robustness.

[0008] Feature alignment problem: In feature-level fusion, features of different modalities are difficult to align effectively, especially in the perception of long-distance or dynamic targets, which affects the perception accuracy.

[0009] Limited scenario adaptability: Traditional methods find it difficult to adjust fusion strategies according to different environmental conditions, resulting in reduced system performance in complex dynamic environments. Summary of the invention

[0010] In order to solve the above technical problems, the present invention proposes an autonomous driving perception method with multi-modal adaptive fusion, which improves the robustness of the autonomous driving perception system in the case of sensor failure or modal loss through adaptive selection and fusion strategies, and optimizes the perception performance in complex scenarios.

[0011] To achieve the above object, the present invention provides a multi-modal adaptive fusion autonomous driving perception method, comprising: acquiring a public image dataset of a road scene;

[0012] The public image dataset is input into the constructed multimodal adaptive fusion model to obtain the autonomous driving perception result.

[0013] Optionally, the constructed multimodal adaptive fusion model includes: an adaptive selection module, a feature extraction module, an adaptive fusion module and a perception task head module;

[0014] The adaptive selection module is used to perform modal fault detection on the public image data set, detect the integrity of the public image data set, and perform weighted processing after the integrity detection to obtain a combined data stream;

[0015] The feature extraction module is used to extract features from the combined data stream, and perform perspective conversion of the extracted features using different methods to obtain a multimodal data stream;

[0016] The adaptive fusion module is used to fuse the multimodal data streams into a unified bird's-eye view feature;

[0017] The perception task head module is used to perform target positioning and map segmentation on the unified bird's-eye view features.

[0018] Optionally, obtaining the combined data stream includes:

[0019] w i =MMA(C,s i )·SA(s i ),

[0020] Among them, MMA(C,s i ) is based on scene C and sensor s i The weight assignment rule of features, SA(s i ) is the sensor s i The state function, w i For the current S i The weight value of the sensor.

[0021] Optionally, the feature extraction module includes a feature extraction unit and a perspective conversion unit;

[0022] The feature extraction unit is used to extract features from the obtained combined data stream through a corresponding feature extraction channel to obtain features of different modalities;

[0023] The perspective conversion unit is used to perform perspective conversion using corresponding perspective conversion methods according to the characteristics of different modes to obtain a multi-modal data stream.

[0024] Optionally, the adaptive fusion module includes a temporal self-attention unit, a modality cross-attention unit, and a feed-forward neural network layer;

[0025] The temporal self-attention unit is used to align the historical BEV features with the BEV query features according to the self-motion given the BEV query features at the current timestamp t and the historical BEV features retained at the timestamp t-1, initialize the public BEV query of the current frame based on the historical BEV features, and obtain the initialized public BEV feature map through temporal self-attention;

[0026] The modal cross attention unit is used to establish an association between the available modal BEV features and the initialized public BEV feature map according to the initialized public BEV feature map, so as to obtain a fused public BEV feature map;

[0027] The feedforward neural network layer is used to further process the fused common BEV feature map.

[0028] Optionally, the perception task head module includes a 3D object detection unit and a map segmentation task unit;

[0029] The 3D object detection unit is used to detect and locate various road participants in the driving scene through a category-based center heat map head and a regression head;

[0030] The map segmentation task unit is used to segment the driving scene into regions with different semantic meanings based on a focal loss training method.

[0031] Optionally, the public BEV feature map initialized by temporal self-attention includes:

[0032]

[0033] Among them, Q P represents the BEV query point at p = (x, y), DeformAttn(·) represents the function of the deformable attention mechanism, TA(Q p ,{Q,B t ' -1}) represents the public BEV feature map B initialized by the output of the current temporal self-attention module P , Q represents the BEV query at a given current timestamp t, B t ' -1 Represents the aligned historical BEV feature B t-1 , p represents the reference point, and V represents the value vector.

[0034] Optionally, obtaining the fused common BEV feature map includes:

[0035]

[0036] Where m represents the attention head, K represents the number of sampling points, p represents the reference point, and W m and W m ′ is the learnable projection matrix, B P represents the initialized common BEV features, x represents the features from one modality, M represents the set of value vectors, k represents the index of the sampled keys, represents the attention weight predicted by the current modality x, It represents the predicted offset of the reference point p, CMA(B p ,p,x) represents the fused common BEV feature map.

[0037] Technical effect of the invention: The present invention discloses an autonomous driving perception method with multi-modal adaptive fusion, which improves the adaptability of the autonomous driving perception system in the case of sensor failure or modality loss, realizes adaptive selection and fusion of perception data, and ensures the continuity and robustness of the autonomous driving perception system; solves the problem of feature alignment in the process of multi-modal data fusion, and ensures that the fused features have high precision and consistency in different scenarios; optimizes the perception performance in complex dynamic scenes, and improves the adaptability and perception accuracy of the autonomous driving perception system to various scenarios (such as night and bad weather) through real-time scene classification and sensor weight adjustment; improves the perception accuracy and robustness of the autonomous driving perception system in autonomous driving tasks, and realizes efficient target detection and map segmentation under multiple tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0039] Figure 1 A schematic diagram of a process flow of a multi-modal adaptive fusion autonomous driving perception method according to an embodiment of the present invention;

[0040] Figure 2 It is an overall flow chart of the multi-modal adaptive fusion model according to an embodiment of the present invention;

[0041] Figure 3 This is a flow chart of the adaptive selection (ASM) module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0042] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0043] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] like Figure 1-2 As shown, this embodiment provides a multi-modal adaptive fusion autonomous driving perception method, including: obtaining a public image dataset of a road scene;

[0045] The public image dataset is input into the constructed multimodal adaptive fusion model to obtain the autonomous driving perception result.

[0046] Specifically, the public image datasets for road scenes include:

[0047] In order to verify the effectiveness of the MAS-Fusion framework, a public image dataset of road scenes is obtained, and the nuScenes dataset is preferred for experimental verification. The nuScenes dataset is a widely used benchmark dataset in the current field of autonomous driving, providing data including six cameras, a 32-beam laser radar (LiDAR), and five millimeter-wave radars. The dataset contains a total of 1,000 driving scenes, each of which lasts 20 seconds, and annotates 3D bounding boxes at a frequency of 2Hz, covering 10 object categories, with a total of about 1.4 million 3D annotation boxes, covering a variety of driving scenes such as daytime, night, sunny days, and rainy days. The preprocessing method is to randomly rotate, deform, and crop the image, and reduce or expand the size of the image to the target size. The target size is set according to the needs. The smaller the size, the lower the computational cost, but the more blurred the image. In order to ensure the clarity of the image and reduce the computational cost, the target size is preferably 256×704×3. Preprocessing the image by the above method can increase the diversity of the image, and using the preprocessed image to train the model can make the trained model more robust. For multi-scale feature extraction. In addition, the LiDAR point cloud and Radar point cloud are voxelized at a resolution of 0.075 meters for subsequent feature extraction. In order to realize the encoding of camera and LiDAR data, Swin-T transformer and VoxelNet model are selected respectively to ensure the efficiency and robustness of feature extraction.

[0048] Furthermore, the constructed multimodal adaptive fusion model includes: an adaptive selection module, a feature extraction module, an adaptive fusion module and a perception task head module;

[0049] The adaptive selection module is used to perform modal fault detection on the public image data set, detect the integrity of the public image data set, and perform weighted processing after the integrity detection to obtain a combined data stream;

[0050] The feature extraction module is used to extract features from the combined data stream, and perform perspective conversion of the extracted features using different methods to obtain a multimodal data stream;

[0051] The adaptive fusion module is used to fuse the multimodal data streams into a unified bird's-eye view feature;

[0052] The perception task head module is used to perform target positioning and map segmentation on the unified bird's-eye view features.

[0053] like Figure 3 As shown in the figure, the adaptive selection module ASM involves receiving raw data from a series of sensors (including LiDAR, millimeter-wave radar, and camera). The data first passes through the adaptive selection module (ASM), which plays a vital role in monitoring the sensor status and identifying missing data modalities. After verifying that the data modality is complete, the system assigns different weights to each sensor based on real-time scene classification (such as natural weather conditions and lighting scenes). The determination of weights follows preset rules to ensure that the fusion process adapts to specific scenarios to achieve optimal performance. Among them, modal loss represents: traditional rigid fusion methods often ignore the impact of sensor damage and modality loss, which usually leads to suboptimal perception effects. To solve this problem, when some modalities are missing, ASM automatically selects existing modalities for fusion and perception tasks. This method can still maintain high perception robustness in the case of modal loss, with less performance degradation. In the event of sensor failure, ASM dynamically adjusts the sensor combination to maintain perception continuity. The data of the faulty sensor is automatically excluded, triggering the adaptive selection fusion of the remaining sensors. For example, when LiDAR fails, the system will automatically switch to the fusion of millimeter-wave radar and camera data. This dynamic adjustment mechanism significantly improves the robustness of the system and ensures reliable perception results can be output when sensors fail. Scene adaptation means that ASM performs real-time scene perception and sensor weight assignment when all sensor modalities are available. ASM classifies scenes by processing environmental features (including weather and lighting conditions) and dynamically assigns weights to each sensor based on these classifications, thereby adjusting the fusion process according to specific scenarios. This adaptive method allows the optimization of sensor combinations to achieve the best perception effect in different scenarios. Compared with traditional rigid fusion methods, this method is a significant improvement and can actively adapt to diverse environmental conditions. In theory, different types of sensors have unique characteristics and advantages. For example, cameras are good at capturing semantic and color information, but their perception effect is poor in extremely dark and extremely bright conditions, so their weight in the fusion process should be reduced under such conditions. Although LiDAR has the ability to accurately locate objects, it is susceptible to noise interference in bad weather, so its weight also needs to be adjusted accordingly. Many examples show that different sensors work better in specific complex scenes. This evidence supports the conclusion that sensor weights should be adjusted according to different scene requirements to enhance the overall fusion perception effect. Sensor weight assignment rules are crucial to actively select the functionality of the fusion module and these rules are formulated via a knowledge-based approach.

[0054] Based on a partial understanding of specific scenes and their associated sensor combinations, this knowledge is used to predetermine sensor array weights that adapt to different scenarios. The nuScenes dataset is used for validation, and the annotated scene classification in the dataset is used instead of the scene classification of the real vehicle. The nuScenes dataset classifies complex scenes into sunny days, rainy days, nights, and ordinary days. These labels provide the basis for scene classification, and sensor weights are assigned to each classified scene, and the reasons for each setting are explained. The size of the weight value (between 0 and 1) indicates its relative importance in the fusion process. The higher the weight value, the greater the contribution of the sensor to the fusion process. The weighting of the cross-scene multimodal combination is detailed in Table 1.

[0055] Table 1

[0056]

[0057] Furthermore, first build a model on software such as Pytorch, TensorFlow, Keras, etc., preferably build an autonomous driving drivable area detection model on Pytorch; detect the integrity of the input through the modal loss perception unit, and perform the input operation of the sample data. The data stream then enters the scene adaptation module for weighted operation of the module weight. By detecting what scene the current scene C belongs to, and combining the feedback result of the modal loss perception module to the current sensor (S1, S2...) combination for weighted operation, the combined data stream obtained includes:

[0058] w i =MMA(C,s i )·SA(s i ),

[0059] Among them, MMA(C,s i ) is based on scene C and sensor s i The weight assignment rule of features, SA(s i ) is the sensor s i The state function determines whether it is currently available, w i For the current S i The weight value of the sensor.

[0060] Further, the feature extraction module includes a feature extraction unit and a perspective conversion unit;

[0061] The feature extraction unit is used to extract features from the obtained combined data stream through a corresponding feature extraction channel to obtain features of different modalities;

[0062] The perspective conversion unit is used to perform perspective conversion using corresponding perspective conversion methods according to the characteristics of different modes to obtain a multi-modal data stream.

[0063] Specifically, the BEV feature extraction of camera images consists of two stages: feature extraction and view conversion. First, the Swin-Transformer backbone network extracts features from the data of each camera, denoted as F C ∈R (B×N×C×H×W) , where B is the batch size, N is the number of cameras, C is the number of channels, and H and W are the dimensions of the image. After feature extraction, the feature becomes F C′ ∈R (B×N×D×H×W) , where D represents the number of channels. Then, these features are converted to BEV representation F by LSS method B ∈R (B×D′×H×W) Where D′ is the number of feature channels per pixel.

[0064] LiDAR: The original point cloud data is processed by feature encoding and feature extractor. First, the original point cloud is evenly divided into several voxels, denoted as F V ∈R (C×N×H×W) , where H and W are the voxel dimensions, N is the number of points per voxel, and C is the feature of each point. After feature extraction, the feature is represented as F B ∈R (C×H×W) , represented as a pseudo image with H and W dimensions and C feature channels.

[0065] Radar: Radar point cloud data is processed in a similar way to LiDAR. The original radar point cloud data is F R ∈(X R ,Y R ,V X ,V Y ), and finally get the feature F R ∈R (C×H×W) .

[0066] Further, the adaptive fusion module includes a temporal self-attention unit, a modality cross-attention unit and a feed-forward neural network layer;

[0067] The temporal self-attention unit is used to align the historical BEV features with the BEV query features according to the self-motion given the BEV query features at the current timestamp t and the historical BEV features retained at the timestamp t-1, initialize the public BEV query of the current frame based on the historical BEV features, and obtain the initialized public BEV feature map through temporal self-attention;

[0068] The modal cross attention unit is used to establish an association between the available modal BEV features and the initialized public BEV feature map according to the initialized public BEV feature map, so as to obtain a fused public BEV feature map;

[0069] The feedforward neural network layer is used to further process the fused common BEV feature map.

[0070] Specifically, the adaptive fusion module is responsible for fusing multimodal data into a unified bird's-eye view (BEV) feature. AFM combines the temporal self-attention mechanism and the cross-modal attention mechanism to ensure that the data of the remaining modalities can be seamlessly integrated when the modality is missing, thereby maintaining efficient perception performance under the condition of sensor loss. It is worth emphasizing that the multimodal bird's-eye view (BEV) feature fusion module is specially designed and optimized to meet the fusion requirements of free combination and can integrate any combination of modalities. Finally, the perception result is output based on the fused features.

[0071] The present invention is to give a BEV query Q at a current timestamp t and a historical BEV feature B retained at a timestamp t-1. t-1 First, according to the self-motion t-1 Align with Q so that features on the same grid correspond to the same position. Align the historical BEV features B t-1 Indicated as B t ' -1 Based on historical BEV characteristics B t-1 To initialize the public BEV query Q of the current frame, the association between historical information and current features is established through temporal self-attention. In order to speed up this process, a deformable attention mechanism is adopted, which is as follows:

[0072]

[0073] Among them, Q P represents the BEV query point at p = (x, y), DeformAttn(·) represents the function of the deformable attention mechanism, TA(Q p ,{Q,B t ' -1}) represents the public BEV feature map B initialized by the output of the current temporal self-attention module P , Q represents the BEV query at a given current timestamp t, B t ' -1 Represents the aligned historical BEV feature B t-1 , p represents the reference point, and V represents the value vector.

[0074] Based on the obtained initialization common BEV feature B P , through the multi-head deformable attention mechanism, establish the available modal BEV feature x∈(B C ,B L ,B R ) and B P The overall processing formula is as follows:

[0075]

[0076] Where m represents the attention head, K represents the number of sampling points, p represents the reference point, and W m and W m ′ is the learnable projection matrix, B P represents the initialized common BEV features, x represents the features from one modality, M represents the set of value vectors, k represents the index of the sampled keys, represents the attention weight of the current modality x prediction, It represents the predicted offset of the reference point p, CMA(B p ,p,x) represents the fused common BEV feature map.

[0077] Further, the perception task head module includes a 3D object detection unit and a map segmentation task unit;

[0078] The 3D object detection unit is used to detect and locate various road participants in the driving scene through a category-based center heat map head and a regression head;

[0079] The map segmentation task unit is used to segment the driving scene into regions with different semantic meanings based on a focal loss training method.

[0080] Specifically, the core task of 3D object detection is to achieve the positioning and feature estimation of the target through the category-based center heatmap head and regression head. The main function of the center heatmap head is to accurately locate the spatial center of the target in the bird's-eye view (BEV), while the regression head further estimates the size, rotation angle and speed of the target. By working together, these two modules provide the key information required for object detection in 3D environments. The center heatmap head generates a dense probability map that represents the likelihood of each grid point being the center of the target.

[0081] First, the Gaussian distribution is used to fuse the BEV feature map B P The center of the target is encoded on the heat map, where the center point of each target corresponds to a Gaussian peak on the heat map. In the bird's eye view (BEV) space, for each center position of the target (x c ,y c ), and calculate its probability value in the heat map through Gaussian distribution. The formula is usually as follows:

[0082]

[0083] Among them, H(x,y) represents the c ,y c ), that is, the possibility of this position being the target center, (x c ,y c) is the true center coordinate of the target. σ is the standard deviation of the Gaussian distribution, which is used to control the extension range of the distribution. By adjusting the σ value, it can adapt to targets of different scales.

[0084] This method can effectively deal with the problems of partial occlusion and multiple target overlap. The specific steps include mapping the target center position to the gridded BEV space, generating the target heat map center area using Gaussian distribution, and adjusting the distribution range by standard deviation to adapt to different target scales. The network finally outputs a dense heat map representing the probability of the target center, and removes duplicate detections through non-maximum suppression (NMS) to ensure that each target has only one center point.

[0085] After detecting the center of the target, the regression head further predicts the size (length, width, height), rotation (orientation angle, such as the heading angle yaw) and speed (speed and direction of movement in the time series) of the target. In order to stabilize the optimization process, these attributes are usually normalized to reduce instability. The network uses the Smooth L1 loss function to smoothly optimize these continuous variables, as follows:

[0086]

[0087] Thus, the fluctuation caused by discrete jumps can be avoided. Through the combination of the center heat map and the regression head, the network can not only accurately locate the target but also describe its physical characteristics in detail.

[0088] The training process of the multimodal adaptive fusion model in the present invention includes: following the image, lidar and millimeter wave radar data enhancement strategies proposed by the framework to expand the diversity of training samples. AdamW is used for optimization, and the weight decay is 0.05. 24 epochs are used for training, and the training and reasoning tasks are performed on 6 NVIDIA-A800 GPUs. After obtaining the entire training model, the optimal architecture and parameter set of the model are determined. The entire network adopts an end-to-end training method, and the network weights are continuously updated until the model converges. Through end-to-end training, the model can autonomously learn the ability of target prediction and map segmentation from the training data.

[0089] The multimodal adaptive fusion model deployment in the present invention includes: deploying the trained perception model into the actual new energy vehicle automatic driving system to achieve robust and accurate target detection and map segmentation. Select a hardware platform suitable for the automatic driving system, including processors, storage devices, sensors and communication modules. Ensure that the hardware meets the computing requirements of the model and has sufficient performance and storage capacity. Verify the compatibility of the hardware with the automatic driving system. The trained perception model is integrated into the software architecture of the automatic driving system. Determine the input and output interfaces of the model and connect with other modules. Implement data interaction and communication mechanisms to ensure that the model can work seamlessly with other modules. Optimize and compress the model to reduce the size and computational complexity of the model in response to real-time performance requirements. Maintain a balance between performance and accuracy of the model to ensure that the model can run efficiently on new energy vehicles. Verify and test the entire automatic driving system to ensure the performance and safety of the model in actual scenarios. Perform simulator testing, closed-loop testing and field testing, etc., covering various driving scenarios and situations. Check whether the output and behavior of the model meet expectations, and evaluate the accuracy and robustness of the model under various test cases. After deployment, continuously monitor the model’s performance and output results to ensure the model’s stability and reliability in the real environment. Collect feedback data and use it to improve and optimize the model. Update the model regularly to adapt to new driving scenarios and requirements and improve perception performance and accuracy.

[0090] The present invention discloses an autonomous driving perception method with multi-modal adaptive fusion, which improves the adaptability of the autonomous driving perception system in the case of sensor failure or modality loss, realizes adaptive selection and fusion of perception data, and ensures the continuity and robustness of the autonomous driving perception system; solves the problem of feature alignment in the process of multi-modal data fusion, and ensures that the fused features have high accuracy and consistency in different scenarios; optimizes the perception performance in complex dynamic scenes, and improves the adaptability and perception accuracy of the autonomous driving perception system to various scenarios (such as night and bad weather) through real-time scene classification and sensor weight adjustment; improves the perception accuracy and robustness of the autonomous driving perception system in autonomous driving tasks, and realizes efficient target detection and map segmentation under multiple tasks.

[0091] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A multi-modal adaptive fusion autonomous driving perception method, characterized in that: include: Obtain a public image dataset of road scenes; The public image dataset is input into the constructed multimodal adaptive fusion model to obtain the autonomous driving perception result.

2. The multi-modal adaptive fusion autonomous driving perception method according to claim 1, characterized in that: The constructed multimodal adaptive fusion model includes: an adaptive selection module, a feature extraction module, an adaptive fusion module and a perception task head module; The adaptive selection module is used to perform modal fault detection on the public image data set, detect the integrity of the public image data set, and perform weighted processing after the integrity detection to obtain a combined data stream; The feature extraction module is used to extract features from the combined data stream, and perform perspective conversion of the extracted features using different methods to obtain a multimodal data stream; The adaptive fusion module is used to fuse the multimodal data streams into a unified bird's-eye view feature; The perception task head module is used to perform target positioning and map segmentation on the unified bird's-eye view features.

3. The multi-modal adaptive fusion autonomous driving perception method according to claim 2, characterized in that: Obtaining a combined data stream includes: w i =MMA(C,s i )·SA(s i ), Among them, MMA(C,s i ) is based on scene C and sensor s i The weight assignment rule of features, SA(s i ) is the sensor s i The state function, w i For the current S i The weight value of the sensor.

4. The multi-modal adaptive fusion autonomous driving perception method according to claim 2, characterized in that: The feature extraction module includes a feature extraction unit and a view conversion unit; The feature extraction unit is used to extract features from the obtained combined data stream through a corresponding feature extraction channel to obtain features of different modalities; The perspective conversion unit is used to perform perspective conversion using corresponding perspective conversion methods according to the characteristics of different modes to obtain a multi-modal data stream.

5. The multi-modal adaptive fusion autonomous driving perception method according to claim 2, characterized in that: The adaptive fusion module includes a temporal self-attention unit, a modal cross-attention unit, and a feed-forward neural network layer; The temporal self-attention unit is used to align the historical BEV features with the BEV query features according to the self-motion given the BEV query features at the current timestamp t and the historical BEV features retained at the timestamp t-1, initialize the public BEV query of the current frame based on the historical BEV features, and obtain the initialized public BEV feature map through temporal self-attention; The modal cross attention unit is used to establish an association between the available modal BEV features and the initialized public BEV feature map according to the initialized public BEV feature map, so as to obtain a fused public BEV feature map; The feedforward neural network layer is used to further process the fused common BEV feature map.

6. The multi-modal adaptive fusion autonomous driving perception method according to claim 5, characterized in that: The perception task head module includes a 3D object detection unit and a map segmentation task unit; The 3D object detection unit is used to detect and locate various road participants in the driving scene through a category-based center heat map head and a regression head; The map segmentation task unit is used to segment the driving scene into regions with different semantic meanings based on a focal loss training method.

7. The multi-modal adaptive fusion autonomous driving perception method according to claim 5, characterized in that: The public BEV feature map initialized by temporal self-attention includes: Among them, Q P represents the BEV query point at p = (x, y), DeformAttn(·) represents the function of the deformable attention mechanism, TA(Q p ,{Q,B t ' -1 }) represents the public BEV feature map B initialized by the output of the current temporal self-attention module P , Q represents the BEV query at a given current timestamp t, B t ' -1 Represents the aligned historical BEV feature B t-1 , p represents the reference point, and V represents the value vector.

8. The multi-modal adaptive fusion autonomous driving perception method according to claim 5, characterized in that: The fused public BEV feature maps include: Where m represents the attention head, K represents the number of sampling points, p represents the reference point, and W m and W m ′ is the learnable projection matrix, B P represents the initialized common BEV features, x represents the features from one modality, M represents the set of value vectors, k represents the index of the sampled keys, represents the attention weight predicted by the current modality x, It represents the predicted offset of the reference point p, CMA(B p ,p,x) represents the fused common BEV feature map.

Images