Multi-modal end-to-end automatic driving method and system based on unified aerial view representation

By adopting unified bird's-eye view representation and multimodal feature fusion technology in end-to-end autonomous driving systems, the problems of information loss and low learning efficiency in traditional systems are solved, and more efficient environmental scenario understanding and planning performance improvements are achieved.

CN119049000AActive Publication Date: 2024-11-29SUN YAT SEN UNIV

Patent Information

Application Number
CN202411050035.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-01
Publication Date
2024-11-29
Estimated Expiration
2044-08-01

AI Technical Summary

Technical Problem

Existing end-to-end autonomous driving systems have limitations in environmental scenario understanding and planning performance, especially due to the problems of information loss and low model learning efficiency when information fusion of different sensors.

Method used

Using a multimodal end-to-end autonomous driving method based on unified aerial view representation, RGB cameras and LiDAR point cloud features are extracted through a dual-branch network and converted to aerial view space for fusion, using a convolutional self-attention mechanism and Transformer The decoder performs feature fusion and behavior planning, and introduces a multi-task learning framework to improve model performance.

Benefits of technology

It effectively improves the environmental scenario understanding and planning performance of end-to-end autonomous driving systems, reduces information losses, and improves the accuracy of autonomous driving trajectory prediction and the robustness of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119049000A_ABST
    Figure CN119049000A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-modal end-to-end automatic driving method and system based on unified aerial view representation, and the method comprises the steps: obtaining an RGB camera image and LiDAR point cloud data of a target vehicle object, carrying out the feature extraction through a dual-branch network, converting the image features and LiDAR point cloud features into an aerial view space, and obtaining multi-modal aerial view features; the multi-modal aerial view features are aligned in the channel dimension, and multi-modal aerial view feature fusion is carried out through a convolution self-attention mechanism fusion module; and adopting a behavior planning network based on a Transform decoder, taking the fused aerial view features as decision input of the behavior planning network, and outputting an automatic driving prediction behavior of the target vehicle object. The method effectively improves the environment scene understanding capability and planning performance of an end-to-end automatic driving system, improves the accuracy of automatic driving track prediction, and can be widely applied to the technical field of automatic driving.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a multimodal end-to-end autonomous driving method and system based on unified bird's-eye view representation. Background Art

[0002] Traditional autonomous driving systems construct the intermediate process from sensor input to actuator output into multiple independent sub-modules, such as perception, prediction, planning, and control. However, this type of modular solution naturally has the problem of error accumulation during task execution, and the planning output is completely dependent on the results of perception and prediction. Once perception and prediction are wrong, planning will be seriously affected. In order to reduce the accumulated errors during the execution process, researchers have proposed the concept of end-to-end autonomous driving, which directly maps raw sensor data to future driving trajectories or control instructions through a single neural network model. Compared with traditional modular solutions, end-to-end solutions not only simplify the decision-making process, but also reduce the problems caused by error transmission between modules.

[0003] Early end-to-end autonomous driving systems have made significant progress using monocular cameras, but the environment representation obtained only from RGB images has great limitations in capturing spatial geometric information. Therefore, existing autonomous vehicles are generally equipped with multiple complementary sensors, such as RGB cameras, laser radar (LiDAR), and millimeter wave radar (Radar). In order to achieve effective fusion of information from different sensors, most existing studies project the features of one modality onto another modality to achieve modality enhancement, but this method may cause information loss in the projected modality. Some other studies have tried to associate features of different sensors through attention mechanisms, but this kind of attention learning that associates features by feature similarity under different views is less efficient. Compared with the above methods, using bird's-eye view (BEV) representation for sensor information fusion has less information loss. By establishing a unified coordinate system and scale for all sensors, it makes it easier to associate data between different sensors, greatly reducing the complexity of system design, and also promoting integration with other sensors. In addition, most existing end-to-end autonomous driving models use complex feature extraction and fusion networks, while the behavior planning network structure consists of only a few simple multi-layer perceptrons (MLPs) and gated recurrent units (GRUs). This imbalance between resources and tasks limits the learning efficiency of the model and affects the overall learning ability of the model. Even if the feature extraction and fusion network can provide high-quality feature representation, due to the limited capacity of the behavior planning network, the model may not be able to effectively learn the mapping relationship from input to output. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a multimodal end-to-end autonomous driving method and system based on a unified bird's-eye view representation, so as to effectively improve the environmental scene understanding capability and planning performance of the end-to-end autonomous driving system.

[0005] An aspect of an embodiment of the present invention provides a multimodal end-to-end autonomous driving method based on a unified bird's-eye view representation, including:

[0006] Obtain the RGB camera image and LiDAR point cloud data of the target vehicle object, use a dual-branch network to extract image features and LiDAR point cloud features, and transform the image features and LiDAR point cloud features into the bird's-eye view space to obtain multimodal bird's-eye view features;

[0007] Align the multimodal bird's-eye view features in the channel dimension, and fuse the multimodal bird's-eye view features through the convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features;

[0008] A behavior planning network based on a Transformer decoder is adopted, and the fused bird's-eye view features are used as the decision input of the behavior planning network to output the autonomous driving predicted behavior of the target vehicle object.

[0009] Optionally, the step of acquiring an RGB camera image and LiDAR point cloud data of a target vehicle object, extracting image features and LiDAR point cloud features using a dual-branch network, and converting the image features and LiDAR point cloud features into a bird's-eye view space to obtain multimodal bird's-eye view features includes:

[0010] ResNet-34 is used as the backbone network for image feature extraction to perform preliminary feature extraction;

[0011] Based on high-resolution image features, a feature pyramid network containing a dilated spatial pyramid pooling node is used to extract multi-scale resolution features.

[0012] Predicting discrete depth distributions corresponding to image features through a depth estimation network;

[0013] The frustum feature can be obtained by performing an outer product operation on the image feature and the corresponding depth distribution;

[0014] Each voxel in the frustum feature is assigned to a corresponding BEV grid through the camera's internal and external parameter matrix, and the features in the same grid are summed and pooled to obtain the camera BEV feature;

[0015] Use several convolutional layers and downsampling layers to reduce the dimension of the original point cloud BEV features to obtain low-resolution features;

[0016] The multi-view image features from the image feature extraction network are concatenated along the width direction to form new image features, the query vector, key vector and value vector are configured, and a multi-head attention mechanism and a convolutional layer function are applied to obtain the aggregated features;

[0017] The aggregated features are concatenated with the low-resolution features after dimensionality reduction in the channel dimension, and then the convolution layer function and the upsampling layer function are applied to obtain the enhanced point cloud BEV features.

[0018] Optionally, aligning the multimodal bird's-eye view features in the channel dimension and fusing the multimodal bird's-eye view features through a convolutional self-attention mechanism fusion module to obtain fused bird's-eye view features includes:

[0019] The obtained image BEV features and point cloud BEV features are concatenated in the channel dimension to fuse the two modal features in the BEV space;

[0020] The residual neural network is used to enhance the local feature extraction capability of the network, and the global perception capability based on the self-attention mechanism is used to strengthen the fused BEV features, realize the deep fusion of the two modal features in the BEV space, and obtain the fused bird's-eye view features.

[0021] Optionally, the residual neural network is used to enhance the local feature extraction capability of the network, and the global perception capability based on the self-attention mechanism is used to strengthen the fused BEV features, so as to achieve deep fusion of the two modal features in the BEV space, and obtain the fused bird's-eye view features, including:

[0022] A residual neural network block is used to fuse the concatenated BEV features to reduce the resolution of BEV features;

[0023] By using several layers of window self-attention blocks, the feature map is divided into four independent windows, and the window self-attention calculation is performed inside each window to enhance the feature map after dimensionality reduction;

[0024] Apply another downsampled residual neural network block to reduce the resolution of the BEV features again;

[0025] Several global self-attention blocks are used to capture the global dependencies between BEV features and modeling features, and deep fusion of the two modal features in the BEV space is achieved to obtain the fused bird's-eye view features.

[0026] Optionally, the behavior planning network based on the Transformer decoder is adopted, the fused bird's-eye view features are used as decision inputs of the behavior planning network, and the automatic driving predicted behavior of the target vehicle object is outputted, including:

[0027] Initialize a feature vector using a local target point, use the feature vector as a query embedding for a decoder, generate state features using vehicle feature information of the target vehicle object through a fully connected layer, and superimpose the state features and a learnable position encoding embedding onto the query embedding;

[0028] Using the cross-attention mechanism, the BEV features obtained by fusion of multimodal features are globally averaged and pooled, and the sinusoidal position encoding embedding is superimposed as the key and value of the cross-attention calculation, and the query embedding is used to query the spatial information in the BEV features;

[0029] While taking future waypoints as the main prediction output, control command prediction is introduced as an auxiliary task, and the output of control command prediction is fused through a specific fusion strategy;

[0030] A multi-task learning framework is adopted and the loss of auxiliary tasks is introduced to guide the multimodal feature learning process and enhance the performance and robustness of the model.

[0031] Optionally, while taking the future waypoints as the main prediction output, introducing control instruction prediction as an auxiliary task, and fusing the output of the control instruction prediction through a specific fusion strategy, including:

[0032] Two prediction heads are connected after the decoder output to predict multiple future waypoints and control commands respectively;

[0033] The waypoint prediction head uses a layer of GRU network, takes 4 waypoint query embeddings as input, and autoregressively predicts multiple future waypoints. Then two PID controllers perform lateral and longitudinal control to generate the steering value, throttle value, and brake value of the target vehicle object.

[0034] The control command prediction head consists of three fully connected neural network layers, taking a control command query embedding as input and directly predicting the vehicle's steering value, throttle value, and brake value;

[0035] The query embedding of the control instruction is appended to the query embedding of the waypoint and used together as the input of the GRU network to improve the prediction accuracy of the waypoint;

[0036] During the vehicle turning phase, it is determined whether the target vehicle object is in a turning state. If it is in a turning state, the steering angle predicted by the control instruction is integrated into the steering angle output by the trajectory prediction combined with the PID controller with a specific weight coefficient as the control output;

[0037] When the target vehicle object is in the straight-moving stage, trajectory prediction combined with PID controller is used as the control output.

[0038] Optionally, in the step of adopting a multi-task learning framework and introducing the loss of auxiliary tasks to guide the multimodal feature learning process, the expression of the overall loss function is:

[0039] L=λ depth L depth +λ wp L wp +λ ctr L ctr

[0040] Among them, L depth represents the loss function of the depth estimation network; L wp represents the waypoint prediction network loss function; L ctr represents the control instruction prediction network loss function; λ depth , wp and λ ctr are the weight coefficients of the corresponding tasks respectively.

[0041] Another aspect of the embodiment of the present invention further provides a multimodal end-to-end autonomous driving system based on a unified bird's-eye view representation, including:

[0042] The first module is used to obtain the RGB camera image and LiDAR point cloud data of the target vehicle object, extract the image features and LiDAR point cloud features using a dual-branch network, and transform the image features and LiDAR point cloud features into the bird's-eye view space to obtain multimodal bird's-eye view features;

[0043] The second module is used to align the multimodal bird's-eye view features in the channel dimension, and fuse the multimodal bird's-eye view features through the convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features;

[0044] The third module is used to adopt a behavior planning network based on a Transformer decoder, use the fused bird's-eye view features as the decision input of the behavior planning network, and output the predicted autonomous driving behavior of the target vehicle object.

[0045] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0046] The memory is used to store programs;

[0047] The processor executes the program to implement the method described above.

[0048] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0049] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0050] The embodiment of the present invention obtains the RGB camera image and LiDAR point cloud data of the target vehicle object, uses a dual-branch network to extract image features and LiDAR point cloud features, and converts the image features and LiDAR point cloud features into a bird's-eye view space to obtain multimodal bird's-eye view features; aligns the multimodal bird's-eye view features in the channel dimension, and fuses the multimodal bird's-eye view features through a convolutional self-attention mechanism fusion module to obtain fused bird's-eye view features; adopts a behavior planning network based on a Transformer decoder, uses the fused bird's-eye view features as the decision input of the behavior planning network, and outputs the autonomous driving predicted behavior of the target vehicle object. The present invention effectively improves the environmental scene understanding capability and planning performance of the end-to-end autonomous driving system, and improves the accuracy of autonomous driving trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 An overall step flow chart provided for an embodiment of the present invention;

[0053] Figure 2 An architecture diagram of a multimodal end-to-end autonomous driving system provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of a point cloud BEV feature enhancement module provided in an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of a multimodal BEV feature fusion module provided in an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of a behavior planning network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0058] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0059] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0061] The multimodal end-to-end autonomous driving method based on unified bird's-eye view representation provided by the embodiment of the present invention relates to the field of autonomous driving technology. The multimodal end-to-end autonomous driving method based on unified bird's-eye view representation provided by the embodiment of the present invention can be applied to a terminal, can also be applied to a server, and can also be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a multimodal end-to-end autonomous driving method based on a unified bird's-eye view representation, etc., but is not limited to the above forms.

[0062] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0063] Specifically, an aspect of an embodiment of the present invention provides a multimodal end-to-end autonomous driving method based on a unified bird's-eye view representation, including:

[0064] Obtain the RGB camera image and LiDAR point cloud data of the target vehicle object, use a dual-branch network to extract image features and LiDAR point cloud features, and transform the image features and LiDAR point cloud features into the bird's-eye view space to obtain multimodal bird's-eye view features;

[0065] Align the multimodal bird's-eye view features in the channel dimension, and fuse the multimodal bird's-eye view features through the convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features;

[0066] A behavior planning network based on a Transformer decoder is adopted, and the fused bird's-eye view features are used as the decision input of the behavior planning network to output the autonomous driving predicted behavior of the target vehicle object.

[0067] Optionally, the step of acquiring an RGB camera image and LiDAR point cloud data of a target vehicle object, extracting image features and LiDAR point cloud features using a dual-branch network, and converting the image features and LiDAR point cloud features into a bird's-eye view space to obtain multimodal bird's-eye view features includes:

[0068] ResNet-34 is used as the backbone network for image feature extraction to perform preliminary feature extraction;

[0069] Based on high-resolution image features, a feature pyramid network containing a dilated spatial pyramid pooling node is used to extract multi-scale resolution features.

[0070] Predicting discrete depth distributions corresponding to image features through a depth estimation network;

[0071] The frustum feature can be obtained by performing an outer product operation on the image feature and the corresponding depth distribution;

[0072] Each voxel in the frustum feature is assigned to a corresponding BEV grid through the camera's internal and external parameter matrix, and the features in the same grid are summed and pooled to obtain the camera BEV feature;

[0073] Use several convolutional layers and downsampling layers to reduce the dimension of the original point cloud BEV features to obtain low-resolution features;

[0074] The multi-view image features from the image feature extraction network are concatenated along the width direction to form new image features, the query vector, key vector and value vector are configured, and a multi-head attention mechanism and a convolutional layer function are applied to obtain the aggregated features;

[0075] The aggregated features are concatenated with the low-resolution features after dimensionality reduction in the channel dimension, and then the convolution layer function and the upsampling layer function are applied to obtain the enhanced point cloud BEV features.

[0076] Optionally, aligning the multimodal bird's-eye view features in the channel dimension and fusing the multimodal bird's-eye view features through a convolutional self-attention mechanism fusion module to obtain fused bird's-eye view features includes:

[0077] The obtained image BEV features and point cloud BEV features are concatenated in the channel dimension to fuse the two modal features in the BEV space;

[0078] The residual neural network is used to enhance the local feature extraction capability of the network, and the global perception capability based on the self-attention mechanism is used to strengthen the fused BEV features, realize the deep fusion of the two modal features in the BEV space, and obtain the fused bird's-eye view features.

[0079] Optionally, the residual neural network is used to enhance the local feature extraction capability of the network, and the global perception capability based on the self-attention mechanism is used to strengthen the fused BEV features, so as to achieve deep fusion of the two modal features in the BEV space, and obtain the fused bird's-eye view features, including:

[0080] A residual neural network block is used to fuse the concatenated BEV features to reduce the resolution of BEV features;

[0081] By using several layers of window self-attention blocks, the feature map is divided into four independent windows, and the window self-attention calculation is performed inside each window to enhance the feature map after dimensionality reduction;

[0082] Apply another downsampled residual neural network block to reduce the resolution of the BEV features again;

[0083] Several global self-attention blocks are used to capture the global dependencies between BEV features and modeling features, and deep fusion of the two modal features in the BEV space is achieved to obtain the fused bird's-eye view features.

[0084] Optionally, the behavior planning network based on the Transformer decoder is adopted, the fused bird's-eye view features are used as decision inputs of the behavior planning network, and the automatic driving predicted behavior of the target vehicle object is outputted, including:

[0085] Initialize a feature vector using a local target point, use the feature vector as a query embedding for a decoder, generate state features using vehicle feature information of the target vehicle object through a fully connected layer, and superimpose the state features and a learnable position encoding embedding onto the query embedding;

[0086] Using the cross-attention mechanism, the BEV features obtained by fusion of multimodal features are globally averaged and pooled, and the sinusoidal position encoding embedding is superimposed as the key and value of the cross-attention calculation, and the query embedding is used to query the spatial information in the BEV features;

[0087] While taking future waypoints as the main prediction output, control command prediction is introduced as an auxiliary task, and the output of control command prediction is fused through a specific fusion strategy;

[0088] A multi-task learning framework is adopted and the loss of auxiliary tasks is introduced to guide the multimodal feature learning process and enhance the performance and robustness of the model.

[0089] Optionally, while taking the future waypoints as the main prediction output, the control command prediction is introduced as an auxiliary task, and the output of the control command prediction is fused through a specific fusion strategy, including

[0090] Two prediction heads are connected after the decoder output to predict multiple future waypoints and control commands respectively;

[0091] The waypoint prediction head uses a layer of GRU network, takes 4 waypoint query embeddings as input, and autoregressively predicts multiple future waypoints. Then two PID controllers perform lateral and longitudinal control to generate the steering value, throttle value, and brake value of the target vehicle object.

[0092] The control command prediction head consists of three fully connected neural network layers, taking a control command query embedding as input and directly predicting the vehicle's steering value, throttle value, and brake value;

[0093] The query embedding of the control instruction is appended to the query embedding of the waypoint and used together as the input of the GRU network to improve the prediction accuracy of the waypoint;

[0094] During the vehicle turning phase, it is determined whether the target vehicle object is in a turning state. If it is in a turning state, the steering angle predicted by the control instruction is integrated into the steering angle output by the trajectory prediction combined with the PID controller with a specific weight coefficient as the control output;

[0095] When the target vehicle object is in the straight-moving stage, trajectory prediction combined with PID controller is used as the control output.

[0096] Optionally, in the step of adopting a multi-task learning framework and introducing the loss of auxiliary tasks to guide the multimodal feature learning process, the expression of the overall loss function is:

[0097] L=λ depth L depth +λ wp L wp +λ ctr L ctr

[0098] Among them, L depth represents the loss function of the depth estimation network; L wp represents the waypoint prediction network loss function; L ctr represents the control instruction prediction network loss function; λ depth , wp and λ ctr are the weight coefficients of the corresponding tasks respectively.

[0099] The following is an implementation process of the present invention in a specific application scenario in conjunction with the accompanying drawings of the specification:

[0100] Reference Figure 1 The present invention provides a multimodal end-to-end autonomous driving system based on a unified bird's-eye view representation, the method comprising the following steps:

[0101] S1, obtain RGB camera images and LiDAR point cloud data, use a dual-branch network to extract image features and LiDAR point cloud features, and convert them into BEV space;

[0102] S2, align the multimodal BEV features in the channel dimension, and then perform multimodal BEV feature fusion through the convolutional self-attention mechanism fusion module;

[0103] S3. Use the behavior planning network based on Transformer decoder, use the fused BEV features as the decision input of the behavior planning network, and finally output the predicted behavior.

[0104] Optionally, the step of taking the RGB camera image and the LiDAR point cloud data, extracting the image features and the LiDAR point cloud features using a dual-branch network, and converting them into the BEV space specifically includes:

[0105] S11, collecting RGB camera images from multiple perspectives of the left front, front and right front of the vehicle, and converting the 2D image features of the multiple perspectives into 3D space to construct the camera BEV features.

[0106] Specifically, the image resolution captured by each camera is 300×400, and the horizontal viewing angle is 60°. The three cameras provide a total horizontal viewing angle of 180° for the autonomous vehicle. For RGB image inputs from multiple views, in order to construct camera BEV features, the present invention draws on the 2D to 3D view conversion method LSS proposed in the related art. This method promotes the multi-view camera image features to a predicted three-dimensional space, uses depth estimation to assign an assumed depth to each feature pixel, and then assigns these three-dimensional features to a predefined three-dimensional grid, so that the features from different cameras are unified into a common three-dimensional space, and then the features of this three-dimensional space are converted into two-dimensional BEV features by performing a pooling operation. The construction process is as follows: Figure 2 shown.

[0107] First, ResNet-34 is used as the backbone network for image feature extraction to perform preliminary feature extraction. Then, a feature pyramid network containing a dilated spatial pyramid pooling node is used to extract multi-scale resolution features. Different from the LSS method, this paper uses higher resolution image features. To retain more spatial information, N, H i , W i and C i Represent the number of cameras, the height and width of the i-th camera image, and the feature dimension. Secondly, the depth estimation network predicts the discrete depth distribution corresponding to the image features. Where D represents the discrete depth value. Finally, the image feature F i The corresponding depth distribution D i The outer product operation can be performed to obtain the frustum feature Frum, and the calculation formula is as follows:

[0108]

[0109] in, Through the camera's internal and external parameter matrices, each voxel in the frustum feature is assigned to the corresponding BEV grid, and the features in the same grid are summed and pooled to finally obtain the camera BEV feature B camera ∈R H×W×C , where H, W and C represent the size of the camera BEV feature and the feature dimension respectively.

[0110] In order to improve the estimation accuracy of depth distribution, the present invention projects the LiDAR point cloud into a multi-view image, and uses a depth completion method to densify the projected sparse depth map, and uses the dense depth map to supervise depth estimation.

[0111] S12. Use the PointPillars network to convert LiDAR point data into point cloud BEV features that match the image BEV features, and adopt a point cloud BEV feature enhancement module guided by image features to improve the learning efficiency of LiDAR point cloud features.

[0112] Specifically, for LiDAR point cloud input, the present invention uses the PointPillars network as the LiDAR point cloud feature extraction backbone network to generate point cloud BEV features B′∈R that match the image BEV features H×W×C However, due to the sparsity of LiDAR point cloud itself, relying solely on PointPillars network for feature extraction may result in the model being unable to effectively learn the feature information of sparse areas in the point cloud. To this end, the present invention designs a point cloud feature enhancement module guided by image features to improve the learning efficiency of LiDAR point cloud features. The specific details are as follows Figure 3 shown.

[0113] First, several convolutional layers and downsampling layers are used to reduce the dimension of the original point cloud BEV feature B′ to obtain a low-resolution feature The calculation process is as follows:

[0114] N = f conv (f down (f conv (B′)))

[0115] Among them, f conv Represents a 3×3 convolutional layer function, f down Represents a downsampling layer function, which aims to reduce the dimension and complexity of the data to make subsequent processing more efficient.

[0116] Secondly, N is set as the query vector Q, and the multi-view image features F from the image feature extraction network are i They are connected in series along the width direction to form a new image feature i, which is used as the key vector K and the value vector V. The multi-head attention mechanism and a convolutional layer function are applied to obtain the aggregated feature A. The calculation process is as follows:

[0117] A=f conv (MultiHead(Q,K,V))

[0118] In the calculation process of the multi-head attention mechanism, Q, V, and K are first transformed h times linearly to obtain h groups of Q i , K i , V i For each group Q i , K i , V i , the corresponding output Head is obtained through attention calculation i , splice all the Head i , and linearly map it to the final output. The calculation formula is as follows:

[0119] MultiHead(Q,K,V)=MLP(Concat(Head 1 ,...,Head h ))

[0120]

[0121] This step achieves the effect of aggregating image features and point cloud features by considering the mutual connection between image features and point cloud features.

[0122] Finally, the aggregate feature A is concatenated with the low-resolution feature N after dimensionality reduction in the channel dimension, and then the convolution layer function and the upsampling layer function are applied to obtain the enhanced point cloud BEV feature B. lidar ∈R H×W×C :

[0123] B lidar =f conv (f up (fconv (Concat[A,N])))

[0124] Among them, f up Represents an upsampling layer function, and Concat[,] represents a concatenation operation. This step enhances the learning of point cloud BEV features through image feature guidance, thereby effectively enhancing the expression ability of point cloud BEV features in sparse areas.

[0125] Optionally, the step of aligning the multimodal BEV features in the channel dimension and then fusing the multimodal BEV features through a convolutional self-attention mechanism fusion module specifically includes:

[0126] S21. Concatenate the obtained camera BEV features and point cloud BEV features in the channel dimension to achieve a simple fusion of the two modal features in the BEV space.

[0127] Specifically, when generating image BEV features and LiDAR point cloud BEV features After that, the two features are basically aligned in the BEV space, where H and W represent the feature resolution, and C c and C l Represents the number of feature channels of image BEV features and LiDAR point cloud BEV features respectively. By directly concatenating these two features in the channel dimension, a unified BEV representation can be synthesized:

[0128] Z=Concat[B camera ,B lidar ]

[0129] S22. Use residual neural network to enhance the local feature extraction capability of the network. At the same time, with the help of the global perception capability of the self-attention mechanism, further strengthen the fused BEV features and realize the deep fusion of the two modal features in the BEV space.

[0130] Specifically, in order to solve the problem that some features may be spatially misaligned due to the limitation of depth estimation and to further strengthen the fused BEV features, the present invention designs a fusion module that combines convolution operation and self-attention mechanism, such as Figure 4As shown, (a) is a schematic diagram of the processing flow of the BEV feature fusion module, (b) is a schematic diagram of the processing flow of the window self-attention block, and (c) is a schematic diagram of the processing flow of the global self-attention block. This module cross-uses the residual network block (Residual Network Block) and the Transformer-based self-attention block (Self-Attention Block) to perform feature fusion on the BEV features obtained in step S21. This module combines the powerful local feature extraction capability of the residual neural network and the global dynamic feature extraction capability of the self-attention mechanism, compensates for the feature misalignment caused by inaccurate depth estimation, and also improves the entire model's understanding and processing capabilities of complex scenes through deep feature fusion. The specific implementation process is as follows:

[0131] First, a residual neural network block is used to fuse the BEV features concatenated in step S21 and reduce the feature resolution to

[0132] Z 1 =ResNetDownBlock(Z)

[0133] Among them, ResNetDownBlock represents a downsampled residual network block.

[0134] Secondly, the feature map after dimensionality reduction is further enhanced by using several layers of window self-attention blocks to split the feature map into four independent windows, each with a feature resolution of Window self-attention calculation is performed inside each window. This step can effectively reduce the overall computational load by decomposing the original self-attention calculation into each local window for self-attention calculation, while enhancing the module's ability to model local correlations. The calculation formula of the window self-attention block is as follows:

[0135]

[0136]

[0137] Among them, f W-MSA represents the window multi-head self-attention, f LN Representative layer normalization, f FFN Represents a feedforward neural network consisting of two fully connected layers, connected by a nonlinear GELU activation function.

[0138] Again, another downsampling residual network block is applied to further reduce the resolution of the BEV features to

[0139] Z 2 =ResNetDownBlock(Z 1 )

[0140] Finally, several global self-attention blocks are used to capture and model the global dependencies between features. The calculation process is as follows:

[0141]

[0142]

[0143] Among them, f G-MSA Represents global multi-head self-attention. Through the fusion processing of the BEV feature fusion module, the fused low-resolution BEV feature B can be obtained fused , and use it as the BEV feature input for the downstream behavior planning network.

[0144] Optionally, the step of adopting a behavior planning network based on a Transformer decoder, taking the fused BEV features as decision input of the behavior planning network, and finally outputting the predicted behavior specifically includes:

[0145] S31. Use local target points to initialize and generate feature vectors, which are used as query embeddings for the decoder. At the same time, the vehicle speed and high-level navigation commands are generated into state features through a fully connected layer, and these state features are superimposed on the query embedding together with the learnable position encoding embedding.

[0146] Specifically, the local target points are used to initialize and generate 5 learnable query embeddings, 4 of which are used as waypoint query embeddings and 1 is used as control signal query embedding, as shown in Figure 5 As shown in Figure 2. Superimposing with the learnable positional encoding embeddings enables the decoder to distinguish and understand the relationship between different query embeddings. High-level navigation instructions are generated using A * Algorithm-generated global waypoint generation, including straight ahead, left turn, right turn, and lane keeping.

[0147] S32. Using the cross-attention mechanism, the BEV features obtained by fusion of multimodal features are globally averaged and pooled, and the sinusoidal position encoding embedding is superimposed as the key and value of the cross-attention calculation, and the query embedding is used to query the spatial information in the BEV features.

[0148] Specifically, in each decoder layer, the query embedding obtained in step S31 is used to query the feature information in the BEV feature superimposed with the sinusoidal position encoding. By introducing position encoding in the BEV feature, the query embedding can extract the spatial information in the BEV feature using the attention mechanism in the decoder. After being processed by multiple layers of decoding layers, the query embedding can fully decode the information in the BEV feature.

[0149] S33. While using future waypoints as the main prediction output, control command prediction is introduced as an auxiliary task. The output of control command prediction is fused through a specific fusion strategy to improve the control performance of the autonomous driving vehicle.

[0150] Specifically, the present invention designs two prediction methods for the future behavior of the vehicle. One is to first predict multiple waypoints in the future, and then convert the waypoint trajectory into a control signal in combination with a controller (such as PID, LQR, etc.), while the other directly outputs a detailed control signal. Specifically, the present invention connects two prediction heads after the decoder output to predict multiple future waypoints and control instructions respectively. The waypoint prediction head uses a layer of GRU network, takes 4 waypoint query embeddings as input, and autoregressively predicts multiple future waypoints. Then two PID controllers perform lateral and longitudinal control to generate the vehicle's steering value steer wp ∈[-1,1], throttle value wp ∈[-1,1] and brake value brake wp ∈[-1,1]. The control command prediction head consists of three fully connected neural network layers, taking a control command query embedding as input and directly predicting the vehicle’s steering value steer ctr ∈[-1,1], throttle value throttle ctr ∈[-1,1] and brake value brake ctr ∈[-1,1]. At the same time, the control instruction query embedding is appended to the first waypoint query embedding and used together as the input of the GRU network to improve the prediction accuracy of the first waypoint, as shown in Figure 5 shown.

[0151] By effectively fusing the prediction outputs of the two prediction methods, the inertia problem of the model during large turns can be effectively solved. The present invention designs a strategy for fusing the outputs of two different prediction strategies during the vehicle turning phase. The specific approach of the strategy is to determine whether the vehicle is in a turning state. If so, the steering angle predicted by the control command is steered. ctr The steering angle steer output by the controller is fused with a certain weight coefficient based on trajectory prediction wp Go up, the calculation is as follows:

[0152] steer = (1-α) × steer wp +α×steer ctr

[0153] Among them, steer is the actual steering value of the vehicle, α is steer wp and steer ctr When the vehicle is moving straight, only trajectory prediction combined with PID controller is used as the control output.

[0154] S34. A multi-task learning framework is adopted and the loss of auxiliary tasks is introduced to guide the multimodal feature learning process and enhance the performance and robustness of the model.

[0155] Specifically, the loss function L of the entire network model is composed of the depth estimation loss function L of the depth estimation network depth , waypoint prediction network loss function L wp And the control instruction prediction network loss function L ctr The overall loss function L can be expressed as follows:

[0156] L=λ depth L depth +λ wp L wp +λ ctr L ctr

[0157] Among them, λ depth , wp and λ ctr are the weight coefficients of the corresponding tasks respectively.

[0158] For the depth estimation network loss function L depth , the present invention adopts the binary cross entropy loss function, and the calculation formula is as follows:

[0159]

[0160] Where N is the number of pixels in the depth map, D is the number of discrete depths, is the depth value of the corresponding pixel in the dense depth map projected by LiDAR, d ij is the predicted depth value.

[0161] For the waypoint prediction network loss function L wp , the present invention adopts L1 loss function and uses the real waypoints of expert driving data set To supervise the predicted waypoints, the calculation formula is as follows:

[0162]

[0163] Where T represents the number of predicted waypoints, wp t is the predicted waypoint.

[0164] For the control command prediction network loss function L ctr , can be divided into three parts: steering value prediction loss L steer , Throttle value prediction loss L throttle and the predicted braking loss L brake , the three losses all use the L1 loss function, and the control instruction prediction loss function can be expressed as follows:

[0165] L ctr =λ steer L steer +λ throttle L throttle +λ brake L brake

[0166] L steer =‖steer-steer gt ‖ 1

[0167] L throttle =‖throttle-throttle gt ‖ 1

[0168] L brake =‖brake-brake gt ‖ 1

[0169] Among them, λ steer , throttle and λ brake is the weight coefficient of the corresponding task.

[0170] In summary, the present invention optimizes trajectory prediction by integrating control command prediction with the control output of the controller, effectively improving the overall planning performance of the end-to-end model, fully exploring the feature complementarity of different modal data, and realizing efficient feature fusion of multimodal data, which has strong practical application value.

[0171] Another aspect of the embodiment of the present invention further provides a multimodal end-to-end autonomous driving system based on a unified bird's-eye view representation, including:

[0172] The first module is used to obtain the RGB camera image and LiDAR point cloud data of the target vehicle object, extract the image features and LiDAR point cloud features using a dual-branch network, and transform the image features and LiDAR point cloud features into the bird's-eye view space to obtain multimodal bird's-eye view features;

[0173] The second module is used to align the multimodal bird's-eye view features in the channel dimension, and fuse the multimodal bird's-eye view features through the convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features;

[0174] The third module is used to adopt a behavior planning network based on a Transformer decoder, use the fused bird's-eye view features as the decision input of the behavior planning network, and output the predicted autonomous driving behavior of the target vehicle object.

[0175] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0176] The embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the multimodal end-to-end autonomous driving method based on unified bird's-eye view representation when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0177] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0178] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0179] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0180] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0181] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0182] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0183] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0184] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0185] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0186] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0187] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0188] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0189] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A multimodal end-to-end autonomous driving method based on a unified bird's-eye view representation, characterized in that: include: Obtain the RGB camera image and LiDAR point cloud data of the target vehicle object, use a dual-branch network to extract image features and LiDAR point cloud features, and transform the image features and LiDAR point cloud features into the bird's-eye view space to obtain multimodal bird's-eye view features; Align the multimodal bird's-eye view features in the channel dimension, and fuse the multimodal bird's-eye view features through the convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features; A behavior planning network based on a Transformer decoder is adopted, and the fused bird's-eye view features are used as the decision input of the behavior planning network to output the autonomous driving predicted behavior of the target vehicle object.

2. The multimodal end-to-end autonomous driving method based on unified bird's-eye view representation according to claim 1, characterized in that: The method obtains the RGB camera image and LiDAR point cloud data of the target vehicle object, extracts the image features and the LiDAR point cloud features using a dual-branch network, and transforms the image features and the LiDAR point cloud features into a bird's-eye view space to obtain multimodal bird's-eye view features, including: ResNet-34 is used as the backbone network for image feature extraction to perform preliminary feature extraction; Based on high-resolution image features, a feature pyramid network containing a dilated spatial pyramid pooling node is used to extract multi-scale resolution features. Predicting discrete depth distributions corresponding to image features through a depth estimation network; The frustum feature can be obtained by performing an outer product operation on the image feature and the corresponding depth distribution; Each voxel in the frustum feature is assigned to a corresponding BEV grid through the camera's internal and external parameter matrix, and the features in the same grid are summed and pooled to obtain the camera BEV feature; Use several convolutional layers and downsampling layers to reduce the dimension of the original point cloud BEV features to obtain low-resolution features; The multi-view image features from the image feature extraction network are concatenated along the width direction to form new image features, the query vector, key vector and value vector are configured, and a multi-head attention mechanism and a convolutional layer function are applied to obtain the aggregated features; The aggregated features are concatenated with the low-resolution features after dimensionality reduction in the channel dimension, and then the convolution layer function and the upsampling layer function are applied to obtain the enhanced point cloud BEV features.

3. The multimodal end-to-end autonomous driving method based on unified bird's-eye view representation according to claim 2, characterized in that: The multimodal bird's-eye view features are aligned in the channel dimension, and the multimodal bird's-eye view features are fused through the convolution self-attention mechanism fusion module to obtain the fused bird's-eye view features, including: The obtained image BEV features and point cloud BEV features are concatenated in the channel dimension to fuse the two modal features in the BEV space; The residual neural network is used to enhance the local feature extraction capability of the network, and the global perception capability based on the self-attention mechanism is used to strengthen the fused BEV features, realize the deep fusion of the two modal features in the BEV space, and obtain the fused bird's-eye view features.

4. The multimodal end-to-end autonomous driving method based on unified bird's-eye view representation according to claim 3, characterized in that: The residual neural network is used to enhance the local feature extraction capability of the network, and the global perception capability based on the self-attention mechanism is used to strengthen the fused BEV features, so as to achieve the deep fusion of the two modal features in the BEV space, and obtain the fused bird's-eye view features, including: A residual neural network block is used to fuse the concatenated BEV features to reduce the resolution of BEV features; By using several layers of window self-attention blocks, the feature map is divided into four independent windows, and the window self-attention calculation is performed inside each window to enhance the feature map after dimensionality reduction; Apply another downsampled residual neural network block to reduce the resolution of the BEV features again; Several global self-attention blocks are used to capture the global dependencies between BEV features and modeling features, and deep fusion of the two modal features in the BEV space is achieved to obtain the fused bird's-eye view features.

5. The multimodal end-to-end autonomous driving method based on unified bird's-eye view representation according to claim 1, characterized in that: The behavior planning network based on the Transformer decoder is adopted, the fused bird's-eye view features are used as the decision input of the behavior planning network, and the automatic driving predicted behavior of the target vehicle object is output, including: Initialize a feature vector using a local target point, use the feature vector as a query embedding for a decoder, generate state features using vehicle feature information of the target vehicle object through a fully connected layer, and superimpose the state features and a learnable position encoding embedding onto the query embedding; Using the cross-attention mechanism, the BEV features obtained by fusion of multimodal features are globally averaged and pooled, and the sinusoidal position encoding embedding is superimposed as the key and value of the cross-attention calculation, and the query embedding is used to query the spatial information in the BEV features; While taking future waypoints as the main prediction output, control command prediction is introduced as an auxiliary task, and the output of control command prediction is fused through a specific fusion strategy; A multi-task learning framework is adopted and the loss of auxiliary tasks is introduced to guide the multimodal feature learning process and enhance the performance and robustness of the model.

6. The multimodal end-to-end autonomous driving method based on unified bird's-eye view representation according to claim 5, characterized in that: While taking the future waypoints as the main prediction output, the control command prediction is introduced as an auxiliary task, and the output of the control command prediction is fused through a specific fusion strategy, including: Two prediction heads are connected after the decoder output to predict multiple future waypoints and control commands respectively; The waypoint prediction head uses a layer of GRU network, takes 4 waypoint query embeddings as input, and autoregressively predicts multiple future waypoints. Then two PID controllers perform lateral and longitudinal control to generate the steering value, throttle value, and brake value of the target vehicle object. The control command prediction head consists of three fully connected neural network layers, taking a control command query embedding as input and directly predicting the vehicle's steering value, throttle value, and brake value; The query embedding of the control instruction is appended to the query embedding of the waypoint and used together as the input of the GRU network to improve the prediction accuracy of the waypoint; During the vehicle turning phase, it is determined whether the target vehicle object is in a turning state. If it is in a turning state, the steering angle predicted by the control instruction is integrated into the steering angle output by the trajectory prediction combined with the PID controller with a specific weight coefficient as the control output; When the target vehicle object is in the straight-moving stage, trajectory prediction combined with PID controller is used as the control output.

7. The multimodal end-to-end autonomous driving method based on unified bird's-eye view representation according to claim 5, characterized in that: In the step of adopting a multi-task learning framework and introducing the loss of auxiliary tasks to guide the multimodal feature learning process, the expression of the overall loss function is: L=λ depth L depth +λ wp L wp +λ ctr L ctr Among them, L depth represents the loss function of the depth estimation network; L wp represents the waypoint prediction network loss function; L ctr represents the control instruction prediction network loss function; λ depth , wp and λ ctr are the weight coefficients of the corresponding tasks respectively.

8. A multimodal end-to-end autonomous driving system based on a unified bird's-eye view representation, characterized in that: include: The first module is used to obtain the RGB camera image and LiDAR point cloud data of the target vehicle object, extract the image features and LiDAR point cloud features using a dual-branch network, and transform the image features and LiDAR point cloud features into the bird's-eye view space to obtain multimodal bird's-eye view features; The second module is used to align the multimodal bird's-eye view features in the channel dimension, and fuse the multimodal bird's-eye view features through the convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features; The third module is used to adopt a behavior planning network based on a Transformer decoder, use the fused bird's-eye view features as the decision input of the behavior planning network, and output the predicted autonomous driving behavior of the target vehicle object.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Bird-eye view semantic segmentation prediction system and method fused with multi-modal sensor

    CN118172560A

  • Methods and systems for the fast estimation of three-dimensional bounding boxes and drivable surfaces using lidar point clouds

    US20200134372A1

Cited By

  • Video-memory-efficient three-dimensional target detection method and device based on aerial view

    CN119810818A

  • Infrared image generation method and device based on multi-mode self-attention mechanism generative adversarial network

    CN120070213A

  • Track prediction method and device for surrounding vehicles in automatic driving scene and medium

    CN120071281A

  • Trajectory prediction method, device and medium for surrounding vehicles in autonomous driving scenario

    CN120071281B

  • End-to-end automatic driving prediction and planning method, system and device and medium

    CN120071286A