Multimodal end-to-end autonomous driving method and system based on unified bird's-eye view representation

Abstract

The application discloses a kind of multi-modal end-to-end automatic driving method and system based on unified bird's eye view characterization, method includes: obtaining the RGB camera image and LiDAR point cloud data of target vehicle object, using double branch network to extract features, and image features and LiDAR point cloud features are converted to bird's eye view space, obtain multi-modal bird's eye view features;Multi-modal bird's eye view features are aligned in channel dimension, and multi-modal bird's eye view feature fusion is carried out by convolution self-attention mechanism fusion module;Adopt the behavior planning network based on Transformer decoder, the fused bird's eye view feature is regarded as the decision input of behavior planning network, and the automatic driving prediction behavior of target vehicle object is output.The application effectively improves the environmental scene understanding ability and planning performance of end-to-end automatic driving system, improves the accuracy of automatic driving trajectory prediction, and can be widely applied in the field of automatic driving technology.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a multimodal end-to-end autonomous driving method and system based on unified bird's-eye view representation. Background Art

[0002] Traditional autonomous driving systems construct the intermediate process from sensor input to actuator output into multiple independent sub-modules, such as perception, prediction, planning, and control. However, this type of modular solution naturally has the problem of error accumulation during task execution, and the planning output is completely dependent on the results of perception and prediction. Once perception and prediction are wrong, planning will be seriously affected. To reduce the accumulated errors during the execution process, researchers have proposed the concept of end-to-end autonomous driving. This solution directly maps raw sensor data to future driving trajectories or control instructions through a single neural network model. Compared with traditional modular solutions, end-to-end solutions not only simplify the decision-making process, but also reduce the problems caused by error transmission between modules.

[0003] Early end-to-end autonomous driving systems achieved significant progress using monocular cameras. However, environmental representations derived solely from RGB images have significant limitations in capturing spatial geometry. Consequently, existing autonomous vehicles are typically equipped with multiple complementary sensors, such as RGB cameras, LiDAR, and millimeter-wave radar. To effectively fuse information from these different sensors, existing research has mostly projected features from one modality onto another for modality enhancement. However, this approach can result in information loss from the projected modality. Other studies have attempted to associate features from different sensors using attention mechanisms, but this type of attention learning, which associates features based on feature similarity across different views, is inefficient. Compared to these approaches, using a bird's-eye view (BEV) representation for sensor information fusion results in less information loss. By establishing a unified coordinate system and scale for all sensors, it facilitates data association between different sensors, significantly reducing system design complexity and facilitating integration with other sensors. Furthermore, existing end-to-end autonomous driving models mostly employ complex feature extraction and fusion networks, while the behavior planning network architecture consists of a few simple multi-layer perceptrons (MLPs) and gated recurrent units (GRUs). This imbalance between resources and tasks limits the learning efficiency of the model and affects the overall learning ability of the model. Even if the feature extraction and fusion network can provide high-quality feature representation, due to the limited capabilities of the behavior planning network, the model may not be able to effectively learn the mapping relationship from input to output. Summary of the Invention

[0004] In view of this, an embodiment of the present invention provides a multimodal end-to-end autonomous driving method and system based on a unified bird's-eye view representation, so as to effectively improve the environmental scene understanding capability and planning performance of the end-to-end autonomous driving system.

[0005] An aspect of an embodiment of the present invention provides a multimodal end-to-end autonomous driving method based on a unified bird's-eye view representation, including:

[0006] Obtain RGB camera images and LiDAR point cloud data of the target vehicle object, use a two-branch network to extract image features and LiDAR point cloud features, and transform the image features and LiDAR point cloud features into the bird's-eye view space to obtain multimodal bird's-eye view features;

[0007] Align the multimodal bird's-eye view features in the channel dimension and fuse them through the convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features.

[0008] A behavior planning network based on a Transformer decoder is adopted, and the fused bird's-eye view features are used as the decision input of the behavior planning network to output the autonomous driving predicted behavior of the target vehicle object.

[0009] Optionally, the step of acquiring an RGB camera image and LiDAR point cloud data of a target vehicle object, extracting image features and LiDAR point cloud features using a dual-branch network, and converting the image features and LiDAR point cloud features into a bird's-eye view space to obtain multimodal bird's-eye view features includes:

[0010] ResNet-34 is used as the backbone network for image feature extraction to perform preliminary feature extraction;

[0011] Based on high-resolution image features, a feature pyramid network containing a dilated spatial pyramid pooling node is used to extract multi-scale resolution features.

[0012] Predicting discrete depth distributions corresponding to image features through a depth estimation network;

[0013] The frustum feature can be obtained by performing outer product operation on the image feature and the corresponding depth distribution;

[0014] Using the camera's internal and external parameter matrices, each voxel in the frustum feature is assigned to a corresponding BEV grid, and the features in the same grid are summed and pooled to obtain the camera BEV feature;

[0015] Use several convolutional layers and downsampling layers to reduce the dimensionality of the original point cloud BEV features to obtain low-resolution features;

[0016] Concatenate the multi-view image features from the image feature extraction network along the width direction to form a new image feature, configure the query vector, key vector and value vector, and apply the multi-head attention mechanism and a convolution layer function to obtain the aggregated feature;

[0017] The aggregated features are concatenated with the low-resolution features after dimensionality reduction in the channel dimension, and then the convolution layer function and the upsampling layer function are applied to obtain the enhanced point cloud BEV features.

[0018] Optionally, aligning the multimodal bird's-eye view features in the channel dimension and fusing the multimodal bird's-eye view features through a convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features includes:

[0019] The obtained image BEV features and point cloud BEV features are spliced ​​in the channel dimension to fuse the two modal features in the BEV space;

[0020] The residual neural network is used to enhance the local feature extraction capability of the network, and the global perception capability based on the self-attention mechanism is used to strengthen the fused BEV features, realize the deep fusion of the two modal features in the BEV space, and obtain the fused bird's-eye view features.

[0021] Optionally, the residual neural network is used to enhance the local feature extraction capability of the network, and the global perception capability based on the self-attention mechanism is used to strengthen the fused BEV features, thereby achieving deep fusion of the two modal features in the BEV space, and obtaining the fused bird's-eye view features, including:

[0022] A residual neural network block is used to fuse the spliced ​​BEV features to reduce the resolution of BEV features;

[0023] By using several layers of window self-attention blocks, the feature map is divided into four independent windows, and the window self-attention calculation is performed within each window to enhance the feature map after dimensionality reduction;

[0024] Apply another downsampling residual neural network block to reduce the resolution of the BEV features again;

[0025] Several global self-attention blocks are used to capture the global dependency between BEV features and modeling features, achieving deep fusion of the two modal features in the BEV space and obtaining the fused bird's-eye view features.

[0026] Optionally, the behavior planning network based on the Transformer decoder is adopted, the fused bird's-eye view features are used as decision inputs of the behavior planning network, and the predicted autonomous driving behavior of the target vehicle object is output, including:

[0027] Initialize a feature vector using a local target point, use the feature vector as the query embedding for the decoder, pass the vehicle feature information of the target vehicle object through a fully connected layer to generate state features, and superimpose the state features and a learnable position encoding embedding onto the query embedding;

[0028] Using a cross-attention mechanism, the BEV features obtained by fusing multimodal features are globally average pooled and superimposed with sinusoidal position encoding embeddings as keys and values ​​for cross-attention calculations. The query embeddings are used to query the spatial information in the BEV features.

[0029] While using future waypoints as the main prediction output, control command prediction is introduced as an auxiliary task, and the output of control command prediction is fused through a specific fusion strategy;

[0030] A multi-task learning framework is adopted and the loss of auxiliary tasks is introduced to guide the multimodal feature learning process and enhance the performance and robustness of the model.

[0031] Optionally, while using future waypoints as the main prediction output, control instruction prediction is introduced as an auxiliary task, and the output of the control instruction prediction is fused through a specific fusion strategy, including:

[0032] Two prediction heads are connected after the decoder output to predict multiple future waypoints and control commands respectively;

[0033] The waypoint prediction head uses a one-layer GRU network to take four waypoint query embeddings as input and autoregressively predict multiple future waypoints. Two PID controllers then perform lateral and longitudinal control to generate steering, throttle, and brake values ​​for the target vehicle object.

[0034] The control command prediction head consists of three fully connected neural network layers, which takes a control command query embedding as input and directly predicts the vehicle's steering value, throttle value, and brake value;

[0035] The query embedding of the control instruction is appended to the query embedding of the waypoint and used together as the input of the GRU network to improve the prediction accuracy of the waypoint;

[0036] During the vehicle turning phase, it is determined whether the target vehicle object is in a turning state. If it is in a turning state, the steering angle predicted by the control command is integrated with the steering angle output by the trajectory prediction combined with the PID controller with a specific weight coefficient as the control output;

[0037] When the target vehicle object is in the straight-moving stage, trajectory prediction combined with PID controller is used as the control output.

[0038] Optionally, in the step of adopting a multi-task learning framework and introducing the loss of auxiliary tasks to guide the multimodal feature learning process, the expression of the overall loss function is:

[0039] L=λ depth L depth +λ wp L wp +λ ctr L ctr

[0040] Among them, L depth represents the depth estimation network loss function; L wp represents the waypoint prediction network loss function; L ctr represents the control instruction prediction network loss function; λ depth ,λ wp and λ ctr are the weight coefficients of the corresponding tasks respectively.

[0041] Another aspect of the present invention provides a multimodal end-to-end autonomous driving system based on a unified bird's-eye view representation, including:

[0042] The first module is used to obtain the RGB camera image and LiDAR point cloud data of the target vehicle object, use a two-branch network to extract image features and LiDAR point cloud features, and transform the image features and LiDAR point cloud features into the bird's-eye view space to obtain multimodal bird's-eye view features;

[0043] The second module is used to align the multimodal bird's-eye view features in the channel dimension and fuse the multimodal bird's-eye view features through the convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features;

[0044] The third module is used to adopt a behavior planning network based on a Transformer decoder, use the fused bird's-eye view features as the decision input of the behavior planning network, and output the autonomous driving predicted behavior of the target vehicle object.

[0045] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0046] The memory is used to store programs;

[0047] The processor executes the program to implement the method described above.

[0048] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0049] The present invention also discloses a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the above method.

[0050] The embodiment of the present invention obtains RGB camera images and LiDAR point cloud data of a target vehicle object, uses a dual-branch network to extract image features and LiDAR point cloud features, and converts the image features and LiDAR point cloud features into a bird's-eye view space to obtain multimodal bird's-eye view features; aligns the multimodal bird's-eye view features in the channel dimension, and fuses the multimodal bird's-eye view features through a convolutional self-attention mechanism fusion module to obtain fused bird's-eye view features; adopts a behavior planning network based on a Transformer decoder, uses the fused bird's-eye view features as the decision input of the behavior planning network, and outputs the autonomous driving predicted behavior of the target vehicle object. The present invention effectively improves the environmental scene understanding capability and planning performance of the end-to-end autonomous driving system, and improves the accuracy of autonomous driving trajectory prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 An overall step flow chart provided for an embodiment of the present invention;

[0053] Figure 2 An architectural diagram of a multimodal end-to-end autonomous driving system provided by an embodiment of the present invention;

[0054] Figure 3 A schematic diagram of a point cloud BEV feature enhancement module provided in an embodiment of the present invention;

[0055] Figure 4 A schematic diagram of a multimodal BEV feature fusion module provided in an embodiment of the present invention;

[0056] Figure 5 A schematic diagram of a behavior planning network provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to make the objects, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0058] It will be understood that the terms "first," "second," and the like used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are merely used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of," "when," or "in response to a determination."

[0059] The terms "at least one", "plurality", "each", "any", etc. used in the present invention include at least one, two or more, multiple, two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0061] The multimodal, end-to-end autonomous driving method based on a unified bird's-eye view representation provided in embodiments of the present invention relates to the field of autonomous driving technology. The multimodal, end-to-end autonomous driving method based on a unified bird's-eye view representation provided in embodiments of the present invention can be applied to a terminal or a server, or can be software running on a terminal or server. In some embodiments, the terminal can be, but is not limited to, a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, or in-vehicle terminal. The server can be configured as an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application that implements the multimodal, end-to-end autonomous driving method based on a unified bird's-eye view representation, but is not limited to the above forms.

[0062] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0063] Specifically, an embodiment of the present invention provides a multimodal end-to-end autonomous driving method based on a unified bird's-eye view representation, including:

[0064] Obtain RGB camera images and LiDAR point cloud data of the target vehicle object, use a two-branch network to extract image features and LiDAR point cloud features, and transform the image features and LiDAR point cloud features into the bird's-eye view space to obtain multimodal bird's-eye view features;

[0065] Align the multimodal bird's-eye view features in the channel dimension and fuse them through the convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features.

[0066] A behavior planning network based on a Transformer decoder is adopted, and the fused bird's-eye view features are used as the decision input of the behavior planning network to output the autonomous driving predicted behavior of the target vehicle object.

[0067] Optionally, the step of acquiring an RGB camera image and LiDAR point cloud data of a target vehicle object, extracting image features and LiDAR point cloud features using a dual-branch network, and converting the image features and LiDAR point cloud features into a bird's-eye view space to obtain multimodal bird's-eye view features includes:

[0068] ResNet-34 is used as the backbone network for image feature extraction to perform preliminary feature extraction;

[0069] Based on high-resolution image features, a feature pyramid network containing a dilated spatial pyramid pooling node is used to extract multi-scale resolution features.

[0070] Predicting discrete depth distributions corresponding to image features through a depth estimation network;

[0071] The frustum feature can be obtained by performing outer product operation on the image feature and the corresponding depth distribution;

[0072] Using the camera's internal and external parameter matrices, each voxel in the frustum feature is assigned to a corresponding BEV grid, and the features in the same grid are summed and pooled to obtain the camera BEV feature;

[0073] Use several convolutional layers and downsampling layers to reduce the dimensionality of the original point cloud BEV features to obtain low-resolution features;

[0074] Concatenate the multi-view image features from the image feature extraction network along the width direction to form a new image feature, configure the query vector, key vector and value vector, and apply the multi-head attention mechanism and a convolution layer function to obtain the aggregated feature;

[0075] The aggregated features are concatenated with the low-resolution features after dimensionality reduction in the channel dimension, and then the convolution layer function and the upsampling layer function are applied to obtain the enhanced point cloud BEV features.

[0076] Optionally, aligning the multimodal bird's-eye view features in the channel dimension and fusing the multimodal bird's-eye view features through a convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features includes:

[0077] The obtained image BEV features and point cloud BEV features are spliced ​​in the channel dimension to fuse the two modal features in the BEV space;

[0078] The residual neural network is used to enhance the local feature extraction capability of the network, and the global perception capability based on the self-attention mechanism is used to strengthen the fused BEV features, realize the deep fusion of the two modal features in the BEV space, and obtain the fused bird's-eye view features.

[0079] Optionally, the residual neural network is used to enhance the local feature extraction capability of the network, and the global perception capability based on the self-attention mechanism is used to strengthen the fused BEV features, thereby achieving deep fusion of the two modal features in the BEV space, and obtaining the fused bird's-eye view features, including:

[0080] A residual neural network block is used to fuse the spliced ​​BEV features to reduce the resolution of BEV features;

[0081] By using several layers of window self-attention blocks, the feature map is divided into four independent windows, and the window self-attention calculation is performed within each window to enhance the feature map after dimensionality reduction;

[0082] Apply another downsampling residual neural network block to reduce the resolution of the BEV features again;

[0083] Several global self-attention blocks are used to capture the global dependency between BEV features and modeling features, achieving deep fusion of the two modal features in the BEV space and obtaining the fused bird's-eye view features.

[0084] Optionally, the behavior planning network based on the Transformer decoder is adopted, the fused bird's-eye view features are used as decision inputs of the behavior planning network, and the predicted autonomous driving behavior of the target vehicle object is output, including:

[0085] Initialize a feature vector using a local target point, use the feature vector as the query embedding for the decoder, pass the vehicle feature information of the target vehicle object through a fully connected layer to generate state features, and superimpose the state features and a learnable position encoding embedding onto the query embedding;

[0086] Using a cross-attention mechanism, the BEV features obtained by fusing multimodal features are globally average pooled and superimposed with sinusoidal position encoding embeddings as keys and values ​​for cross-attention calculations. The query embeddings are used to query the spatial information in the BEV features.

[0087] While using future waypoints as the main prediction output, control command prediction is introduced as an auxiliary task, and the output of control command prediction is fused through a specific fusion strategy;

[0088] A multi-task learning framework is adopted and the loss of auxiliary tasks is introduced to guide the multimodal feature learning process and enhance the performance and robustness of the model.

[0089] Optionally, while taking the future waypoints as the main prediction output, control instruction prediction is introduced as an auxiliary task, and the output of the control instruction prediction is fused through a specific fusion strategy, including

[0090] Two prediction heads are connected after the decoder output to predict multiple future waypoints and control commands respectively;

[0091] The waypoint prediction head uses a one-layer GRU network to take four waypoint query embeddings as input and autoregressively predict multiple future waypoints. Two PID controllers then perform lateral and longitudinal control to generate steering, throttle, and brake values ​​for the target vehicle object.

[0092] The control command prediction head consists of three fully connected neural network layers, which takes a control command query embedding as input and directly predicts the vehicle's steering value, throttle value, and brake value;

[0093] The query embedding of the control instruction is appended to the query embedding of the waypoint and used together as the input of the GRU network to improve the prediction accuracy of the waypoint;

[0094] During the vehicle turning phase, it is determined whether the target vehicle object is in a turning state. If it is in a turning state, the steering angle predicted by the control command is integrated with the steering angle output by the trajectory prediction combined with the PID controller with a specific weight coefficient as the control output;

[0095] When the target vehicle object is in the straight-moving stage, trajectory prediction combined with PID controller is used as the control output.

[0096] Optionally, in the step of adopting a multi-task learning framework and introducing the loss of auxiliary tasks to guide the multimodal feature learning process, the expression of the overall loss function is:

[0097] L=λ depth L depth +λ wp L wp +λ ctr L ctr

[0098] Among them, L depth represents the depth estimation network loss function; L wp represents the waypoint prediction network loss function; L ctr represents the control instruction prediction network loss function; λ depth ,λ wp and λ ctr are the weight coefficients of the corresponding tasks respectively.

[0099] The following is an implementation process of the present invention in a specific application scenario with reference to the accompanying drawings:

[0100] Reference Figure 1 The present invention provides a multimodal end-to-end autonomous driving system based on a unified bird's-eye view representation, the method comprising the following steps:

[0101] S1, obtain RGB camera images and LiDAR point cloud data, use a two-branch network to extract image features and LiDAR point cloud features, and convert them into BEV space;

[0102] S2, align the multimodal BEV features in the channel dimension, and then perform multimodal BEV feature fusion through the convolutional self-attention mechanism fusion module;

[0103] S3. Use the behavior planning network based on Transformer decoder, use the fused BEV features as the decision input of the behavior planning network, and finally output the predicted behavior.

[0104] Optionally, the step of taking RGB camera images and LiDAR point cloud data, extracting image features and LiDAR point cloud features using a dual-branch network, and converting them into BEV space specifically includes:

[0105] S11. Collect RGB camera images from multiple perspectives of the left front, front, and right front of the vehicle, and convert the 2D image features of the multiple perspectives into 3D space to construct the camera BEV features.

[0106] Specifically, the image resolution captured by each camera is 300×400, and the horizontal viewing angle is 60°. The three cameras provide a total horizontal viewing angle of 180° for the autonomous vehicle. For RGB image inputs from multiple views, in order to construct camera BEV features, the present invention draws on the 2D to 3D view conversion method LSS proposed in the related art. This method promotes the multi-view camera image features to a predicted three-dimensional space, uses depth estimation to assign an assumed depth to each feature pixel, and then assigns these three-dimensional features to a predefined three-dimensional grid, so that the features from different cameras are unified into a common three-dimensional space, and then converts the features of this three-dimensional space into two-dimensional BEV features by performing a pooling operation. The construction process is as follows: Figure 2 shown.

[0107] First, ResNet-34 is used as the backbone network for image feature extraction to perform preliminary feature extraction. Then, a feature pyramid network containing a dilated spatial pyramid pooling node is used to extract multi-scale resolution features. Unlike the LSS method, this paper uses higher resolution image features. To retain more spatial information, N, H i , W i and C i Represents the number of cameras, the height and width of the i-th camera image, and the feature dimension. Secondly, the depth estimation network predicts the discrete depth distribution corresponding to the image features Where D represents the discrete depth value. Finally, the image feature F i The corresponding depth distribution D i The outer product operation can be performed to obtain the frustum feature Frum, which is calculated as follows:

[0108]

[0109] in, Through the camera's internal and external parameter matrices, each voxel in the frustum feature is assigned to the corresponding BEV grid, and the features in the same grid are summed and pooled to obtain the camera BEV feature B. camera ∈R H×W×C , where H, W, and C represent the size of the camera BEV feature and the feature dimension, respectively.

[0110] In order to improve the estimation accuracy of depth distribution, the present invention projects the LiDAR point cloud into a multi-view image, uses a depth completion method to densify the projected sparse depth map, and uses the dense depth map to supervise depth estimation.

[0111] S12. Use the PointPillars network to convert LiDAR point data into point cloud BEV features that match the image BEV features, and adopt a point cloud BEV feature enhancement module guided by image features to improve the learning efficiency of LiDAR point cloud features.

[0112] Specifically, for LiDAR point cloud input, the present invention uses the PointPillars network as the LiDAR point cloud feature extraction backbone network to generate point cloud BEV features B′∈R that match the image BEV features H×W×C However, due to the sparsity of LiDAR point cloud itself, relying solely on PointPillars network for feature extraction may result in the model being unable to effectively learn the feature information of sparse areas in the point cloud. To this end, the present invention designs a point cloud feature enhancement module guided by image features to improve the learning efficiency of LiDAR point cloud features. The specific details are as follows: Figure 3 shown.

[0113] First, several convolutional layers and downsampling layers are used to reduce the dimension of the original point cloud BEV feature B′ to obtain low-resolution features The calculation process is as follows:

[0114] N=f conv (f down (f conv (B′)))

[0115] Among them, f conv Represents a 3×3 convolutional layer function, f down Represents a downsampling layer function, which aims to reduce the dimension and complexity of the data to make subsequent processing more efficient.

[0116] Secondly, N is set as the query vector Q, and the multi-view image features F from the image feature extraction network are i Connect them in series along the width direction to form a new image feature i, which is used as the key vector K and value vector V. Apply the multi-head attention mechanism and a convolutional layer function to obtain the aggregated feature A. The calculation process is as follows:

[0117] A=f conv (MultiHead(Q,K,V))

[0118] Among them, the calculation process of the multi-head attention mechanism first undergoes h linear transformations on Q, V, and K to obtain h groups of Q i , K i , V i For each group Q i , K i , V i , the corresponding output Head is obtained through attention calculation i , splice all the Head i , and linearly map it to the final output. The calculation formula is as follows:

[0119] MultiHead(Q,K,V)=MLP(Concat(Head1,...,Head h ))

[0120]

[0121] This step achieves the effect of aggregating image features and point cloud features by considering the mutual relationship between image features and point cloud features.

[0122] Finally, the aggregated feature A is concatenated with the low-resolution feature N after dimensionality reduction in the channel dimension, and then the convolution layer function and the upsampling layer function are applied to obtain the enhanced point cloud BEV feature B. lidar ∈R H×W×C :

[0123] B lidar =f conv (f up (f conv(Concat[A,N])))

[0124] Among them, f up Represents an upsampling layer function, and Concat[,] represents the concatenation operation. This step enhances the learning of point cloud BEV features through image feature guidance, thereby effectively enhancing the expressive power of point cloud BEV features in sparse areas.

[0125] Optionally, the step of aligning the multimodal BEV features in the channel dimension and then fusing the multimodal BEV features through a convolutional self-attention mechanism fusion module specifically includes:

[0126] S21. Splice the obtained camera BEV features and point cloud BEV features in the channel dimension to achieve a simple fusion of the two modal features in the BEV space.

[0127] Specifically, when generating image BEV features and LiDAR point cloud BEV features After that, the two features are basically aligned in the BEV space, where H and W represent the feature resolution, and C c and C l The number of feature channels representing the image BEV features and the LiDAR point cloud BEV features respectively. By directly concatenating these two features in the channel dimension, a unified BEV representation can be synthesized:

[0128] Z=Concat[B camera ,B lidar ]

[0129] S22. Use the residual neural network to enhance the local feature extraction capability of the network. At the same time, with the help of the global perception capability of the self-attention mechanism, further strengthen the fused BEV features and realize the deep fusion of the two modal features in the BEV space.

[0130] Specifically, in order to solve the problem of spatial misalignment between certain features due to the limitation of depth estimation and further enhance the fused BEV features, the present invention designs a fusion module that combines convolution operation and self-attention mechanism, such as Figure 4As shown in the figure, (a) is a schematic diagram of the processing flow of the BEV feature fusion module, (b) is a schematic diagram of the processing flow of the window self-attention block, and (c) is a schematic diagram of the processing flow of the global self-attention block. This module cross-uses the residual network block (Residual Network Block) and the Transformer-based self-attention block (Self-Attention Block) to perform feature fusion on the BEV features obtained in step S21. This module combines the powerful local feature extraction capability of the residual neural network and the global dynamic feature extraction capability of the self-attention mechanism, compensates for the feature misalignment caused by inaccurate depth estimation, and also improves the entire model's understanding and processing capabilities of complex scenes through deep feature fusion. The specific implementation process is as follows:

[0131] First, a residual neural network block is used to fuse the BEV features concatenated in step S21 and reduce the feature resolution to

[0132] Z1=ResNetDownBlock(Z)

[0133] Among them, ResNetDownBlock represents a downsampled residual network block.

[0134] Secondly, the feature map after dimensionality reduction is further enhanced by using several layers of window self-attention blocks to split the feature map into four independent windows. The feature resolution of each window is Window self-attention calculation is performed separately within each window. This step can effectively reduce the overall computational load by decomposing the original self-attention calculation into each local window for self-attention calculation, while enhancing the module's ability to model local correlations. The calculation formula of the window self-attention block is as follows:

[0135]

[0136]

[0137] Among them, f W-MSA represents window multi-head self-attention, f LN Representative layer normalization, f FFN Represents a feedforward neural network consisting of two fully connected layers, connected by a nonlinear GELU activation function.

[0138] Again, another downsampling residual network block is applied to further reduce the resolution of the BEV features to

[0139] Z2=ResNetDownBlock(Z1)

[0140] Finally, several global self-attention blocks are used to capture and model the global dependencies between features. The calculation process is as follows:

[0141]

[0142]

[0143] Among them, f G-MSA Represents global multi-head self-attention. Through the fusion processing of the BEV feature fusion module, the fused low-resolution BEV feature B can be obtained. fused , and use it as the BEV feature input to the downstream behavior planning network.

[0144] Optionally, the step of adopting a behavior planning network based on a Transformer decoder, using the fused BEV features as the decision input of the behavior planning network, and finally outputting the predicted behavior specifically includes:

[0145] S31. Use local target points to initialize and generate feature vectors, which are used as query embeddings for the decoder. At the same time, vehicle speed and high-level navigation commands are generated through a fully connected layer to generate state features, and these state features are superimposed on the query embedding together with the learnable position encoding embedding.

[0146] Specifically, we use local target points to initialize and generate 5 learnable query embeddings, 4 of which are used as waypoint query embeddings and 1 is used as control signal query embedding, as shown in Figure 5 As shown. At the same time, superimposing with the learnable position encoding embedding enables the decoder to distinguish and understand the relationship between different query embeddings. High-level navigation instructions are generated by using A * Algorithm-generated global waypoint generation, including straight ahead, left turn, right turn, and lane keeping.

[0147] S32. Using the cross-attention mechanism, the BEV features obtained by fusion of multimodal features are globally average pooled, and the sinusoidal position encoding embedding is superimposed as the key and value of the cross-attention calculation, and the query embedding is used to query the spatial information in the BEV features.

[0148] Specifically, at each decoder layer, the query embedding obtained in step S31 is used to query the feature information within the BEV features superimposed with the sinusoidal position encoding. By introducing position encoding into the BEV features, the query embedding can leverage the attention mechanism within the decoder to extract the spatial information within the BEV features. After processing through multiple decoding layers, the query embedding can fully decode the information within the BEV features.

[0149] S33. While using future waypoints as the main prediction output, control command prediction is introduced as an auxiliary task. The output of control command prediction is fused through a specific fusion strategy to improve the control performance of the autonomous driving vehicle.

[0150] Specifically, the present invention designs two methods for predicting the future behavior of the vehicle. One is to first predict multiple waypoints in the future, and then convert the waypoint trajectory into a control signal in combination with a controller (such as PID, LQR, etc.), while the other directly outputs a detailed control signal. Specifically, the present invention connects two prediction heads after the decoder output to predict multiple future waypoints and control instructions respectively. The waypoint prediction head uses a layer of GRU network, takes 4 waypoint query embeddings as input, and autoregressively predicts multiple future waypoints. Then, two PID controllers perform lateral and longitudinal control to generate the vehicle's steering value steer. wp ∈[-1,1], throttle value throttle wp ∈[-1,1] and brake value brake wp ∈[-1,1]. The control command prediction head consists of three fully connected neural network layers, which takes a control command query embedding as input and directly predicts the vehicle’s steering value steer ctr ∈[-1,1], throttle value throttle ctr ∈[-1,1] and brake value brake ctr ∈[-1,1]. At the same time, the control instruction query embedding is appended to the first waypoint query embedding and used together as the input of the GRU network to improve the prediction accuracy of the first waypoint, as shown in Figure 5 shown.

[0151] By effectively fusing the prediction outputs of the two prediction methods, the inertia problem of the model during large turns can be effectively solved. The present invention designs a strategy for fusing the outputs of two different prediction strategies during the vehicle turning phase. The specific approach of this strategy is to determine whether the vehicle is in a turning state. If it is in a turning state, the steering angle predicted by the control command is steered. ctr The steering angle steer output by the trajectory prediction combined with the controller is fused with a certain weight coefficient wp Go up, the calculation is as follows:

[0152] steer = (1-α) × steer wp +α×steer ctr

[0153] Among them, steer is the actual steering value of the vehicle, α is steer wp and steer ctr When the vehicle is moving straight, only trajectory prediction combined with PID controller is used as the control output.

[0154] S34. A multi-task learning framework is adopted to introduce the loss of auxiliary tasks to guide the multimodal feature learning process and enhance the performance and robustness of the model.

[0155] Specifically, the loss function L of the entire network model is composed of the depth estimation loss function L of the depth estimation network depth , waypoint prediction network loss function L wp And the control instruction prediction network loss function L ctr The overall loss function L can be expressed as follows:

[0156] L=λ depth L depth +λ wp L wp +λ ctr L ctr

[0157] Among them, λ depth ,λ wp and λ ctr are the weight coefficients of the corresponding tasks respectively.

[0158] For the depth estimation network loss function L depth , the present invention adopts the binary cross entropy loss function, and the calculation formula is as follows:

[0159]

[0160] Where N is the number of pixels in the depth map, D is the number of discrete depths, is the depth value of the corresponding pixel on the dense depth map projected by LiDAR, d ij is the predicted depth value.

[0161] For the waypoint prediction network loss function L wp , this paper adopts L1 loss function and uses the real waypoints of expert driving dataset To supervise the predicted waypoints, the calculation formula is as follows:

[0162]

[0163] Wherein, T represents the predicted number of waypoints, wp t is a predicted waypoint.

[0164] For the control instruction prediction network loss function L ctr , can be divided into three parts: steering value prediction loss L steer , throttle value prediction loss L throttle and brake value prediction loss L brake , three losses are all adopted L1 loss function, control instruction prediction loss function can be expressed as follows:

[0165] L ctr =λ steer L steer +λ throttle L throttle +λ brake L brake

[0166] L steer =‖steer-steer gt ‖1

[0167] L throttle =‖throttle-throttle gt ‖1

[0168] L brake =‖brake-brake gt ‖1

[0169] Wherein, λ steer , λ throttle And λ brake It is the weight coefficient corresponding to the task.

[0170] In summary, the application optimizes the trajectory prediction by fusing the control instruction prediction to combine the control output of the controller, effectively improves the overall planning performance of the end-to-end model, can fully tap the feature complementarity of different modal data, realizes the efficient feature fusion of multi-modal data, and has strong practical application value.

[0171] Another aspect of the embodiment of the application also provides a multi-modal end-to-end automatic driving system based on unified bird's eye view representation, comprising:

[0172] The first module is used for acquiring the RGB camera image and LiDAR point cloud data of the target vehicle object, extracting image features and LiDAR point cloud features using a double-branch network, and converting the image features and LiDAR point cloud features to a bird's eye view space to obtain multi-modal bird's eye view features;

[0173] The second module is used to align the multimodal bird's-eye view features in the channel dimension and fuse the multimodal bird's-eye view features through the convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features;

[0174] The third module is used to adopt a behavior planning network based on a Transformer decoder, use the fused bird's-eye view features as the decision input of the behavior planning network, and output the autonomous driving predicted behavior of the target vehicle object.

[0175] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0176] An embodiment of the present invention further provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor, when executing the computer program, implements the aforementioned multimodal end-to-end autonomous driving method based on a unified bird's-eye view representation. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0177] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0178] Another aspect of the embodiments of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the method described above.

[0179] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0180] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0181] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0182] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0183] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0184] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0185] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0186] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0187] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0188] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0189] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A multimodal end-to-end autonomous driving method based on a unified bird's-eye view representation, characterized in that: include: Obtain RGB camera images and LiDAR point cloud data of the target vehicle object, use a two-branch network to extract image features and LiDAR point cloud features, and transform the image features and LiDAR point cloud features into the bird's-eye view space to obtain multimodal bird's-eye view features; Align the multimodal bird's-eye view features in the channel dimension and fuse them through the convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features. A behavior planning network based on a Transformer decoder is used, and the fused bird's-eye view features are used as the decision input of the behavior planning network to output the autonomous driving predicted behavior of the target vehicle object; The method obtains RGB camera images and LiDAR point cloud data of the target vehicle object, extracts image features and LiDAR point cloud features using a dual-branch network, and converts the image features and LiDAR point cloud features into a bird's-eye view space to obtain multimodal bird's-eye view features, including: ResNet-34 is used as the backbone network for image feature extraction to perform preliminary feature extraction; Based on high-resolution image features, a feature pyramid network containing a dilated spatial pyramid pooling node is used to extract multi-scale resolution features. Predicting discrete depth distributions corresponding to image features through a depth estimation network; The frustum feature can be obtained by performing outer product operation on the image feature and the corresponding depth distribution; Each voxel in the frustum feature is assigned to a corresponding BEV grid through the camera's internal and external parameter matrix, and the features in the same grid are summed and pooled to obtain the image BEV feature; Use several convolutional layers and downsampling layers to reduce the dimensionality of the original point cloud BEV features to obtain low-resolution features; Concatenate the multi-view image features from the image feature extraction network along the width direction to form a new image feature, configure the query vector, key vector and value vector, and apply the multi-head attention mechanism and a convolution layer function to obtain the aggregated feature; The aggregated features are concatenated with the low-resolution features after dimensionality reduction in the channel dimension, and then the convolution layer function and the upsampling layer function are applied to obtain the enhanced point cloud BEV features.

2. The multimodal end-to-end autonomous driving method based on unified bird's-eye view representation according to claim 1, characterized in that: The multimodal bird's-eye view features are aligned in the channel dimension, and the multimodal bird's-eye view features are fused through the convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features, including: The obtained image BEV features and point cloud BEV features are spliced ​​in the channel dimension to fuse the two modal features in the BEV space; The residual neural network is used to enhance the local feature extraction capability of the network, and the global perception capability based on the self-attention mechanism is used to strengthen the fused BEV features, realize the deep fusion of the two modal features in the BEV space, and obtain the fused bird's-eye view features.

3. The multimodal end-to-end autonomous driving method based on unified bird's-eye view representation according to claim 2, characterized in that: The residual neural network is used to enhance the local feature extraction capability of the network. Based on the global perception capability of the self-attention mechanism, the fused BEV features are strengthened, and the deep fusion of the two modal features in the BEV space is achieved. The fused bird's-eye view features are obtained, including: A residual neural network block is used to fuse the spliced ​​BEV features to reduce the resolution of BEV features; By using several layers of window self-attention blocks, the feature map is divided into four independent windows, and the window self-attention calculation is performed within each window to enhance the feature map after dimensionality reduction; Apply another downsampling residual neural network block to reduce the resolution of the BEV features again; Several global self-attention blocks are used to capture the global dependency between BEV features and modeling features, achieving deep fusion of the two modal features in the BEV space and obtaining the fused bird's-eye view features.

4. The multimodal end-to-end autonomous driving method based on unified bird's-eye view representation according to claim 1, characterized in that: The behavior planning network based on the Transformer decoder is used, and the fused bird's-eye view features are used as the decision input of the behavior planning network to output the autonomous driving predicted behavior of the target vehicle object, including: Initialize a feature vector using a local target point, use the feature vector as the query embedding for the decoder, and pass the vehicle feature information of the target vehicle object through a fully connected layer to generate state features, and superimpose the state features and the learnable position encoding embedding onto the query embedding; Using a cross-attention mechanism, the BEV features obtained by fusing multimodal features are globally average pooled and superimposed with sinusoidal position encoding embeddings as keys and values ​​for cross-attention calculations. The query embeddings are used to query the spatial information in the BEV features. While using future waypoints as the main prediction output, control command prediction is introduced as an auxiliary task, and the output of control command prediction is fused through a specific fusion strategy; A multi-task learning framework is adopted and the loss of auxiliary tasks is introduced to guide the multimodal feature learning process and enhance the performance and robustness of the model.

5. The multimodal end-to-end autonomous driving method based on unified bird's-eye view representation according to claim 4, characterized in that: While using future waypoints as the main prediction output, control command prediction is introduced as an auxiliary task. The output of control command prediction is fused through a specific fusion strategy, including: Two prediction heads are connected after the decoder output to predict multiple future waypoints and control commands respectively; The waypoint prediction head uses a one-layer GRU network to take four waypoint query embeddings as input and autoregressively predict multiple future waypoints. Two PID controllers then perform lateral and longitudinal control to generate steering, throttle, and brake values ​​for the target vehicle object. The control command prediction head consists of three fully connected neural network layers, which takes a control command query embedding as input and directly predicts the vehicle's steering value, throttle value, and brake value; The query embedding of the control instruction is appended to the query embedding of the waypoint and used together as the input of the GRU network to improve the prediction accuracy of the waypoint; During the vehicle turning phase, it is determined whether the target vehicle object is in a turning state. If it is in a turning state, the steering angle predicted by the control command is integrated with the steering angle output by the trajectory prediction combined with the PID controller with a specific weight coefficient as the control output; When the target vehicle object is in the straight-moving stage, trajectory prediction combined with PID controller is used as the control output.

6. The multimodal end-to-end autonomous driving method based on unified bird's-eye view representation according to claim 4, characterized in that: In the step of adopting the multi-task learning framework and introducing the loss of auxiliary tasks to guide the multimodal feature learning process, the expression of the overall loss function is: in, Represents the depth estimation network loss function; represents the waypoint prediction network loss function; Represents the control instruction prediction network loss function; 、 and are the weight coefficients of the corresponding tasks respectively.

7. A system for implementing the multimodal end-to-end autonomous driving method based on unified bird's-eye view representation according to any one of claims 1 to 6, characterized in that: include: The first module is used to obtain the RGB camera image and LiDAR point cloud data of the target vehicle object, use a two-branch network to extract image features and LiDAR point cloud features, and transform the image features and LiDAR point cloud features into the bird's-eye view space to obtain multimodal bird's-eye view features; The second module is used to align the multimodal bird's-eye view features in the channel dimension and fuse the multimodal bird's-eye view features through the convolutional self-attention mechanism fusion module to obtain the fused bird's-eye view features; The third module is used to adopt a behavior planning network based on a Transformer decoder, use the fused bird's-eye view features as the decision input of the behavior planning network, and output the autonomous driving predicted behavior of the target vehicle object.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 6.

Images