End-to-end multi-modal multi-task autonomous driving perception method and device based on long short-term hybrid coding

Abstract

The application relates to an end-to-end multi-modal multi-task automatic driving perception method and device based on long-short time sequence hybrid coding. The steps of the method comprise the following: long-short time sequence feature extraction is carried out on multiple frames of input of a vehicle-mounted sensor by using multiple coding networks; then, time sequence fusion based on an attention mechanism is carried out on the extracted features, cross-modal fusion is carried out, and a BEV feature map is generated for each task; and then, a decoder of a different task is used to obtain a prediction result of each task. The application can dynamically adjust the network reasoning speed and precision, can improve the network performance under the same amount of calculation, can carry out time sequence fusion on the multi-frame BEV feature maps obtained by different coding networks, and can better carry out cross-frame alignment on dynamic objects. The application can generate multi-scale BEV features, retains multi-scale features after time sequence and cross-modal fusion, and can respectively give different-scale feature maps to each task decoder through feature selection.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and relates to the establishment of an autonomous driving perception neural network model using computer vision and deep learning technologies. In particular, it relates to a multimodal autonomous driving perception method and device based on long and short temporal mixed resolution and mixed backbone network encoding, and temporal fusion based on bird's-eye view attention mechanism. Background Technology

[0002] In recent years, the research and application of machine learning have continued to develop, with deep learning methods achieving significant breakthroughs in many application areas (such as generative natural language dialogue, computer vision, and speech processing). In many fields, models have improved accuracy by increasing the number of parameters and computational cost. However, in the field of autonomous driving, due to safety and immediacy considerations, a non-networked perception algorithm must be deployed on the vehicle. Constraints such as vehicle hardware power consumption, cost, and inference speed impose strict limitations on the number of parameters and computational cost of this algorithm.

[0003] The main problem in autonomous driving 3D perception is analyzing and processing inputs from a series of sensors to analyze the surrounding environment and objects. This includes tasks such as 3D object detection, occupancy networks, and bird's-eye view (BEV) segmentation. The core lies in the design of the encoder network for the sensor inputs. Taking camera sensors as an example, using higher-resolution image inputs and deeper, wider, and more computationally intensive backbone and neck networks can effectively improve accuracy. Furthermore, since scene or object information may be lost due to occlusion in a single frame, this information can be acquired in previous frames. Different frames from the same camera viewpoint also contain rich spatial information; therefore, longer and better temporal fusion can also effectively improve accuracy.

[0004] However, higher image resolution, larger backbone and neck networks, and longer time series all lead to increased computational costs. Currently, the most accurate methods on various public datasets use large backbone networks such as SwinTransformer-base, ViT-L, and InternImage-L, whose inference latency generally exceeds 300ms, making them difficult to deploy on in-vehicle computing devices. If these large backbone networks are used simultaneously, combined with longer time series and multi-frame inputs exceeding 10 frames, even the most advanced GPUs' memory capacity and computing power are insufficient for training. How to leverage the advantages of high-resolution input and large encoding networks while combining them with longer time series within the limitations of parameter count and computational cost on in-vehicle devices is a practically significant problem.

[0005] In recent years, feature representations in Bird's Eye View (BEV) space have been widely used in autonomous driving methods due to their ease of convolution and compatibility with downstream tasks in autonomous driving, such as BEVFormer, BEVDepth, BEVerse, and BEVFusion published in 2022. However, temporal fusion of BEV features over long timeframes remains a challenging problem. Within a few seconds, objects in a scene, such as vehicles, may have moved tens of meters. Methods like BEVDepth and BEVStereo directly add or concatenate BEV feature maps along the channel dimension. In this case, features of the same object may be scattered across different regions of the BEV image along a path. Convolving these features to the object's current position using convolution would require extremely large kernels or deep convolutions, resulting in high computational cost and inevitably incorporating much irrelevant interference. Therefore, aligning moving objects between different frames using methods such as optical flow prediction, attention mechanisms, and two-stage feature extraction is necessary.

[0006] Furthermore, for different tasks in perception problems, the end-to-end shared encoding network approach is commonly used. This approach extracts features from various sensors using a single encoding network and connects to multiple decoders for different tasks, enabling simultaneous inference for multiple tasks, thus saving memory consumption and reducing inference latency. However, different decoders may have different requirements for feature distribution. In the end-to-end network of the shared encoding network, conflicts may arise between different tasks. Existing methods only use basic methods such as adjusting the weights of the loss function to make trade-offs, without in-depth analysis and resolution of conflicts between multiple tasks. Summary of the Invention

[0007] To overcome the shortcomings of the prior art, the present invention provides a deep learning method for autonomous driving, specifically an end-to-end multimodal multitasking autonomous driving perception method and device based on long and short temporal hybrid coding.

[0008] The technical solution of this invention is:

[0009] An end-to-end multimodal multitasking autonomous driving perception method based on long and short temporal hybrid coding includes the following steps:

[0010] For multi-frame input from vehicle sensors, multiple coding networks are used to extract features in both long and short time series.

[0011] Temporal fusion based on attention mechanism is performed on the extracted features;

[0012] Cross-modal fusion is performed on the temporally fused features, and BEV feature maps are generated for each task;

[0013] By utilizing BEV feature maps, different task decoders are used to obtain the prediction results for each task.

[0014] Furthermore, the vehicle-mounted sensor includes at least one of a multi-view camera, a lidar, and a millimeter-wave radar.

[0015] Furthermore, the feature extraction using multiple coding networks for both long and short time series includes:

[0016] Based on the type of sensor and the required task, a backbone network and a neck network are selected for different inputs. For each modality, multiple backbone networks and neck networks of different sizes are selected. Larger backbone networks and neck networks are used to analyze a few frames of shorter time series, while smaller backbone networks and neck networks are used to analyze more frames of longer time series.

[0017] For each backbone network, a depth predictor is selected to match it. The features of various sensors are extracted by the backbone network to extract multi-scale features, and the neck network performs multi-scale feature fusion. Then, the features are transformed into a unified BEV space coordinate system by the camera view conversion module. The camera view conversion module needs to cooperate with the depth prediction results to project the two-dimensional features into three-dimensional features.

[0018] Furthermore, the temporal fusion of the extracted features based on an attention mechanism includes:

[0019] Based on the required task, determine the required feature scale; for smaller encoding networks, generate larger-scale feature maps through downsampling operations; for each feature scale of each modality, perform temporal feature fusion based on an attention mechanism.

[0020] Furthermore, the step of performing cross-modal fusion on the temporally fused features and generating a BEV feature map for each task includes:

[0021] For each scale, cross-modal feature fusion is performed, connecting features from different modalities along the channel dimension, and then adjusting the number of feature channels and fusing local features through convolutional layers;

[0022] For each task, a BEV feature map of a scale suitable for that task is selected. By using a feature selection module and a BEV convolutional network that are independent of each task, a BEV feature map for each task is obtained.

[0023] Furthermore, during model training, a multimodal data augmentation strategy is constructed, and appropriate loss functions are selected for different tasks. Appropriate loss function weights, optimizers, and learning rate schedulers are used to load pre-trained weights onto each encoding network for end-to-end multi-task training.

[0024] An end-to-end multimodal multitasking autonomous driving perception device based on long and short time-series hybrid coding, comprising:

[0025] The feature encoding module is used to extract features of varying temporal lengths using multiple encoding networks for multi-frame input from vehicle sensors.

[0026] The temporal fusion module is used to perform temporal fusion of the extracted features based on an attention mechanism.

[0027] The cross-modal fusion module is used to perform cross-modal fusion on the temporally fused features and generate BEV feature maps for each task.

[0028] The multi-task decoding module is used to obtain the prediction results for each task by using BEV feature maps and different task decoders.

[0029] A vehicle comprising the aforementioned end-to-end multimodal multitasking autonomous driving perception device based on a hybrid long-short time-series coding system.

[0030] The beneficial effects of this invention are:

[0031] (1) This invention proposes a hybrid long and short time series feature coding network. For long time series, a high-efficiency coding network with a small number of parameters is used, while for short time series, a high-precision coding network with a large number of parameters is used. This can dynamically adjust the network inference speed and accuracy, and improve the network performance under the same computational load.

[0032] (2) This invention proposes a temporal fusion method based on attention mechanism, which can perform temporal fusion on multi-frame BEV feature maps obtained from the above different coding networks, and can better perform cross-frame alignment for dynamic objects.

[0033] (3) By analyzing the conflict between different tasks, it was found that different tasks have different requirements for the scale and distribution of features. This invention proposes a solution to this problem, namely, by using long and short temporal hybrid coding to generate multi-scale BEV features, performing temporal and cross-modal fusion, retaining multi-scale features, and after feature selection, giving feature maps of different scales to the decoders of each task respectively. Attached Figure Description

[0034] Figure 1 This is a flowchart of the deep learning algorithm proposed in this invention.

[0035] Figure 2 This is a diagram illustrating the specific method of feature encoding steps proposed in this invention.

[0036] Figure 3 This is a diagram illustrating the specific method of timing fusion steps proposed in this invention.

[0037] Figure 4 This is a diagram illustrating the specific method of cross-modal fusion steps proposed in this invention. Detailed Implementation

[0038] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0039] This invention proposes an end-to-end multimodal multitasking autonomous driving perception method based on hybrid long and short temporal coding. This method is applicable to any type of sensor input that can be represented as 3D features and any multitasking combination that can be decoded through 3D features. This invention targets multi-frame inputs from one or more types of vehicle-mounted sensors, such as multi-view cameras, LiDAR, and millimeter-wave radar, using multiple coding networks of varying sizes to extract features in both long and short temporal sequences, perform multi-frame temporal feature fusion, and conduct cross-modal feature fusion (e.g., multimodal information). Finally, different BEV feature maps are generated for use by decoders for different tasks (e.g., 3D object detection, BEV high-precision map semantic segmentation, etc.). Specifically, it includes the following steps:

[0040] S1. Feature Encoding: Based on the type of input sensor and the required task, select the backbone network and neck network for Radar point cloud input, LiDAR point cloud input, and image input. For each modality (modality refers to the type of sensor, such as image, LiDAR, millimeter-wave radar, etc.), select multiple backbone networks and neck networks of different sizes. Larger backbone networks and neck networks are used to analyze a few frames in a shorter time series, while smaller backbone networks and neck networks are used to analyze more frames in a longer time series. For each backbone network, select a depth predictor that is compatible with it. After the features of various sensors are processed by the backbone network to extract multi-scale features and the neck network to perform multi-scale feature fusion, they are transformed into a unified BEV space coordinate system by the camera view conversion module. The camera view conversion module needs to work with the depth prediction results to project the two-dimensional features into three-dimensional features.

[0041] S2. Temporal Fusion: Determine the required feature scale based on the task at hand. For smaller encoding networks, some larger-scale features may be missing. In this case, downsampling operations such as pooling can generate larger-scale feature maps. For each feature scale of each modality, perform temporal feature fusion based on an attention mechanism.

[0042] S3. Cross-modal fusion: For each scale, cross-modal feature fusion is performed. Then, for each task, a BEV feature map suitable for the task scale is selected. Through feature selection modules and BEV convolutional networks that are independent of each task, feature maps for each task are obtained.

[0043] S4. Multi-task decoding: For each task, select the feature map generated for that task in S3, connect it to the corresponding task's decoding network, and predict the results of each task.

[0044] S5. Loss Function Design, Model Training, and Inference: A multimodal data augmentation strategy is constructed. For LiDAR and Radar point cloud inputs, rotation, translation, and symmetric data augmentation strategies are employed. For multi-camera image inputs, cropping, color transformation, and symmetric data augmentation strategies are used. For a unified BEV space, random rotation, scaling, and symmetric data augmentation strategies are employed. Appropriate loss functions are selected for different tasks, and suitable loss function weights, optimizers, and learning rate schedulers are used. Pre-trained weights are loaded onto each encoding network for end-to-end multi-task training.

[0045] The invention is further described below with reference to the accompanying drawings, using three types of sensors—LiDAR, millimeter-wave radar, and multiple cameras—as input to process tasks such as 3D target detection, vehicle semantic segmentation in BEV view, semantic segmentation of passable areas in BEV view, and lane line semantic segmentation in BEV view. However, this does not limit the scope of the invention in any way.

[0046] The specific embodiments of the present invention include the following steps (e.g. Figure 1 As shown):

[0047] S1. Feature Encoding: Based on the type of input sensor and the required task, select the backbone network and neck network for Radar point cloud input, LiDAR point cloud input, and image input. For each modality, select multiple backbone networks and neck networks of varying sizes. Larger backbone networks and neck networks are used to analyze a few frames in a shorter time series, while smaller backbone networks and neck networks are used to analyze more frames in a longer time series. For each backbone network, select a depth predictor that is compatible with it. After multi-scale feature extraction by the backbone network and multi-scale feature fusion by the neck network, the features from various sensors are transformed into a unified BEV space coordinate system by the viewpoint transformation module. The camera viewpoint transformation module needs to work with the depth prediction results to project the two-dimensional features into three-dimensional features. The feature encoding process is as follows: Figure 2 As shown, the specific steps include:

[0048] S1.1. Selection of Image Backbone and Neck Networks: Based on task requirements, select the necessary backbone and neck networks. The backbone network is used to extract multi-scale features from the image. For long-time series, lightweight models such as ResNet, DarkNet, MobileNet, and Res2Net are available; for short-time series, high-precision models such as SwinTransformer, InternImage, VovNetv2, and CBNet are available. The neck network is used to fuse multi-scale features, often employing feature pyramid networks and their various variants to aggregate multi-scale features. For longer time series, use a backbone and neck network with lower resolution input and fewer parameters; for shorter time series, use a backbone and neck network with higher resolution input and more parameters.

[0049] S1.2. Viewpoint Transformation from Image Features to Unified BEV Spatial Features: Options include projecting 2D features onto 3D features based on depth estimation and then projecting according to the depth distribution; alternatively, positional encoding of the BEV grid can be performed, and features can be extracted directly from multi-view images using an attention mechanism. For depth estimation methods, more efficient methods, such as monocular convolution in BEVDepth, can be used for longer time series, while more complex methods, such as multi-view, multi-frame depth estimation in MVSNet, can be used for shorter time series. For attention-based viewpoint transformation methods, sparse self-attention mechanisms with a small number of offsets k or lower-resolution feature maps can be used for longer time series; sparse attention mechanisms with a larger number of offsets k or higher-resolution feature maps can be used for shorter time series.

[0050] S1.3. Selection of Backbone and Neck Networks for LiDAR and Millimeter-Wave Radar Point Clouds: First, the radar point cloud is voxelized to obtain feature voxels. Then, the required backbone and neck networks are selected according to the needs of the task. Common methods include SECOND, etc. For short time series, a deeper and wider network is used, while for long time series, a shallower and narrower network is used.

[0051] S1.4. Transformation of 3D Features from LiDAR and Millimeter-Wave Radar to Unified BEV Spatial Features: Utilizing the pose parameters of the radar sensors, voxel coordinate system transformation is performed to align different sensors to a unified BEV coordinate system. Then, cylinder pooling is performed to convert the feature voxels into BEV feature maps.

[0052] S2. Temporal Fusion: Determine the required feature scale based on the task at hand. For smaller encoding networks, some larger-scale features may be missing. In this case, downsampling operations such as pooling can generate larger-scale feature maps. For each feature scale of each modality, perform temporal feature fusion based on an attention mechanism. The temporal fusion process is as follows: Figure 3 As shown, the specific steps include:

[0053] S2.1. Selecting BEV feature maps at the required scale for each task: Because different tasks have different requirements for the scale of BEV feature maps, it is necessary to retain the multi-scale BEV feature maps required for each task. For example, in the relevant experiments of this patent, it was found that on the nuScenes dataset (note: nuScenes is a large-scale autonomous driving dataset that includes tasks such as 3D object detection, 2D object detection, object tracking, and semantic segmentation), the BEV view semantic segmenter has the best performance when the BEV grid length and width are 0.8m, while the 3D object detector performs better when the BEV grid length and width are 0.4m. In this step, if the multi-scale BEV features obtained by the encoding network in S1 contain the required scale, they are directly used; otherwise, the BEV feature maps at the required scale are obtained through pooling, interpolation, and other methods.

[0054] S2.2. Temporal Fusion Module Design: This method employs an attention-based approach to fuse temporal BEV features for better alignment of moving object features. The multi-head sparse attention mechanism (Deformable Attention) is represented as DeformAttn(Q,K,V), where Q represents the query in the attention mechanism, and K and V represent the key and value being queried. For n+1 frames of multi-frame BEV features at each scale for each modality, the BEV features of the current frame are represented as F0, and the BEV features of past frames are represented from nearest to furthest as F1, F2, ..., F... n The time-series fusion method proposed in this patent, which proceeds from back to front and then from front to back, can be expressed as follows:

[0055] For k values ​​from 1 to n:

[0056] F k ←MLP(Concat(DeformAttn(F k ,F k-1 ,F k-1 ,),F k ))

[0057] For k values ​​from n to 1:

[0058] F k-1 ←MLP(Concat(DeformAttn(F k-1,F k ,F k ,),F k-1 ))

[0059] Here, MLP stands for Multilayer Perceptron, and Concat means connecting features in the channel dimension.

[0060] Finally, the updated F0 is taken as the feature after temporal fusion.

[0061] S3. Cross-modal fusion: For each scale, cross-modal feature fusion is performed. Then, for each task, a BEV feature map suitable for that task scale is selected. Through independent feature selection modules and BEV convolutional networks for each task, feature maps usable by each task are obtained. The cross-modal fusion process is as follows: Figure 4 As shown, the specific steps include:

[0062] S3.1. Cross-modal fusion module design: If multiple modalities exist, for each scale, the features of different modalities are connected along the channel dimension, and then the number of feature channels is adjusted and the local features are fused through a convolutional layer with a kernel size of 3. For input from only a single-modal sensor, step S3.1 is skipped.

[0063] S3.2. Feature Selection Module Design: To address the issue that different tasks have different requirements for feature distribution, an independent feature selection module is designed for each task. This module obtains a channel-dimensional mask through global pooling, linear layers, and sigmoid layers to perform feature selection on BEV features.

[0064] S3.3. BEV Convolutional Network Selection: BEV convolutional networks often use smaller image backbone networks and neck networks, with ResNet18 and Feature Pyramid (FPN) being the most commonly used.

[0065] S4. Multi-task decoding: For each task, select the feature map generated for that task in S3, connect it to the corresponding task's decoding network, and predict the results of each task.

[0066] S4.1. Selection of detection decoder: Depending on the specific needs of the scenario, different detection decoders can be selected, such as the PointPillars detection head with anchor points in the first stage, the Centerpoint detection head without anchor points in the first stage, the PETR detection head based on Transformer in the first stage, or the Transfusion-L detection head in the second stage, etc.

[0067] S4.2. Semantic segmentation decoder selection: Depending on the specific needs of the scenario, different semantic segmentation decoders can be selected, such as SegNet based on convolution or PETR SegHead based on Transformer.

[0068] S5. Loss Function Design, Model Training and Inference: Construct a multimodal data augmentation strategy; select appropriate loss functions for different tasks, and use appropriate loss function weights, optimizers and learning rate schedulers; load pre-trained weights for each encoding network and perform end-to-end multi-task training.

[0069] S5.1. Loss Function Design: If a depth estimation-based approach was adopted in S1.2, and ground truth depth values ​​exist (either manually labeled or generated by LiDAR), then the loss function L for depth estimation is calculated. depth For 3D object detection, there exists a classification loss function L that supervises the classification process. cls , and the regression loss function L that supervises the 3D bounding box bbox In addition, there may be loss functions specific to certain detector heads, such as the Centerpoint and Transfusion-L detector heads, which rely on center heatmap prediction for additional classification loss function L for supervising center heatmap prediction. heatmap For semantic segmentation and placeholder network tasks, there exists a classification loss function L for the prediction results. seg or L occ Commonly used classification loss functions include L1 loss function, L2 loss function, Huber loss function, etc.; common regression loss functions include cross-entropy loss function, Focal loss function, etc.

[0070] S5.2. Multi-task Loss Function Weight Selection: The weights of the multi-task loss function can be manually adjusted based on the increase or decrease in accuracy between the multi-task and single-task operations. Taking 3D object detection and semantic segmentation of BEV view vehicles, road passable areas, and lane lines on the nuScenes dataset as an example, using Centerpoint as the 3D object detection head, the final determined loss function weights are L. depth :L cls :L bbos :I seg_veh :L seg_dri :L seg_div = 1:6:1.5:180:45:90. Furthermore, automated methods, such as search algorithms, can be used to adjust the weights of the loss function. Where L... seg_veh L seg_dri L seg_div Let represent the loss functions for vehicle semantic segmentation, drivable area semantic segmentation, and lane line semantic segmentation, respectively.

[0071] S5.3. Data Augmentation Strategy: Construct a multimodal data augmentation strategy. For LiDAR and Radar point cloud inputs, use rotation, translation, and symmetry data augmentation strategies. For multi-camera image inputs, use cropping, color transformation, and symmetry data augmentation strategies. For a unified BEV space, use random rotation, scaling, and symmetry data augmentation strategies.

[0072] S5.4. Model Pre-training: If a camera modality exists, the camera backbone network and neck network are pre-trained on 2D classification, detection, or depth estimation datasets. For each encoder of each modality, single-modality single-task pre-training is first performed on a key task (such as 3D object detection); then, when training the multi-task model end-to-end, the above pre-training parameters are loaded for each encoder of each modality.

[0073] S5.5. Optimizer and Learning Rate Adjustment: Different optimizers and learning rate schedulers can be selected based on the actual training situation of the model. In the relevant experiments of this patent, the AdamW optimizer and Circle learning rate scheduler were used. A larger initial learning rate was used during pre-training, and a smaller initial learning rate was used during multi-task end-to-end training.

[0074] In a 3D object detection task with multi-camera input, the method of this invention was specifically implemented on the nuScenes object detection dataset. Results on nuScenes show that, taking the CenterPoint detector as an example, the baseline method, with two 896x1600 resolution input frames and using Vovnetv2-99 as the backbone network, achieves a nuScenes detection score (NDS) of 59.2. Following the S1 method of this patent, which mixes nine 256x704 resolution input frames and uses ResNet50 as the backbone network, the NDS is improved to 60.0 at the cost of only about 10% increase in computation.

[0075] Meanwhile, by combining the backbone network Vovnetv2-99 based on 2 frames of input at 640x1152 resolution and the backbone network Resnet50 based on 9 frames of input at 256x704 resolution according to the S1 method of this patent, the inference latency is only about 60% of the baseline method, while NDS is 59.8, which exceeds the baseline method.

[0076] The method of this invention was specifically implemented on the nuScenes dataset for multi-task implementations, including 3D object detection with multi-camera input, vehicle semantic segmentation in BEV view, drivable area semantic segmentation in BEV view, and lane line semantic segmentation in BEV view. Implementation results on nuScenes show that the proposed S2 temporal fusion module and S3 task-independent fusion module, compared to the existing best-in-class BEV baseline method BEVStereo, achieve improvements in NDS detection (56.5→58.5), vehicle segmentation mIoU (44.3→46.5), drivable area segmentation mIoU (76.4→77.8), and lane line segmentation mIoU (33.5→36.8) across multiple tasks.

[0077] Another embodiment of the present invention provides an end-to-end multimodal multitasking autonomous driving perception device based on long and short timing hybrid coding, comprising:

[0078] The feature encoding module is used to extract features of varying temporal lengths using multiple encoding networks for multi-frame input from vehicle sensors.

[0079] The temporal fusion module is used to perform temporal fusion of the extracted features based on an attention mechanism.

[0080] The cross-modal fusion module is used to perform cross-modal fusion on the temporally fused features and generate BEV feature maps for each task.

[0081] The multi-task decoding module is used to obtain the prediction results for each task by using BEV feature maps and different task decoders.

[0082] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0083] Another embodiment of the present invention provides a vehicle in which the above-described end-to-end multimodal multitasking autonomous driving perception device based on long and short time sequence hybrid coding is deployed or installed.

[0084] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0085] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0086] It should be noted that the purpose of disclosing the embodiments is to help further understand the present invention. However, those skilled in the art will understand that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims, including but not limited to the fusion of more modalities and end-to-end networks for more tasks. Therefore, the present invention should not be limited to the content disclosed in the embodiments, and the scope of protection of the present invention is defined by the scope of the claims.

Claims

1. An end-to-end multimodal multitasking autonomous driving perception method based on long and short temporal hybrid coding, characterized in that, Includes the following steps: For multi-frame input from vehicle sensors, multiple coding networks are used to extract features in both long and short time series. Temporal fusion based on attention mechanism is performed on the extracted features; Cross-modal fusion is performed on the temporally fused features, and BEV feature maps are generated for each task; Using BEV feature maps, different task decoders are used to obtain the prediction results for each task; The method of using multiple coding networks for long and short time series feature extraction includes: Based on the type of sensor and the required task, a backbone network and a neck network are selected for different inputs. For each modality, multiple backbone networks and neck networks of different sizes are selected. Larger backbone networks and neck networks are used to analyze a few frames of shorter time series, while smaller backbone networks and neck networks are used to analyze more frames of longer time series. For each backbone network, a depth predictor is selected to match it. The features of various sensors are extracted by the backbone network to extract multi-scale features, and the neck network performs multi-scale feature fusion. Then, the features are transformed into a unified BEV space coordinate system by the camera view conversion module. The camera view conversion module needs to cooperate with the depth prediction results to project the two-dimensional features into three-dimensional features. The temporal fusion of the extracted features based on an attention mechanism includes: Determine the required feature scale based on the task at hand; for smaller encoding networks, generate larger-scale feature maps through downsampling; perform temporal feature fusion based on attention mechanism for each feature scale of each modality. The step of performing cross-modal fusion on the temporally fused features and generating a BEV feature map for each task includes: For each scale, cross-modal feature fusion is performed, connecting features from different modalities along the channel dimension, and then adjusting the number of feature channels and fusing local features through convolutional layers; For each task, a BEV feature map of a scale suitable for that task is selected. By using a feature selection module and a BEV convolutional network that are independent of each task, a BEV feature map for each task is obtained.

2. The method according to claim 1, characterized in that, The vehicle-mounted sensors include at least one of a multi-view camera, a lidar, and a millimeter-wave radar.

3. The method according to claim 1, characterized in that, The attention-based temporal fusion of features includes: The multi-head sparse attention mechanism is represented as DeformAttn(Q, K, V), where Q represents the query in the attention mechanism, and K and V represent the key and value being queried in the attention mechanism. For the n+1 frames of multi-frame BEV features for each modality and each scale, the BEV features of the current frame are represented as... The BEV features of past frames are represented from near to far as follows: The timing fusion method adopts a back-to-forward and then front-to-back approach, which is represented as follows: For The value is from to : For The value is from to : in This represents a multilayer perceptron. This indicates that features are connected in the channel dimension; Finally, take the updated version. As a feature after temporal fusion.

4. The method according to claim 1, characterized in that, During model training, a multimodal data augmentation strategy is constructed, and appropriate loss functions are selected for different tasks. Appropriate loss function weights, optimizers, and learning rate schedulers are used to load pre-trained weights onto each encoding network for end-to-end multi-task training.

5. An end-to-end multimodal multitasking autonomous driving perception device based on long and short time-series hybrid coding, characterized in that, The apparatus for performing the method according to any one of claims 1 to 4 comprises: The feature encoding module is used to extract features of varying temporal lengths using multiple encoding networks for multi-frame input from vehicle sensors. The temporal fusion module is used to perform temporal fusion of the extracted features based on an attention mechanism. The cross-modal fusion module is used to perform cross-modal fusion on the temporally fused features and generate BEV feature maps for each task. The multi-task decoding module is used to obtain the prediction results for each task by using BEV feature maps and different task decoders.

6. A vehicle, characterized in that, Includes the end-to-end multimodal multitasking autonomous driving perception device based on long and short time sequence hybrid coding as described in claim 5.

7. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 4.

Images