End-to-end automatic driving self-vehicle trajectory planning method, system, equipment and medium

Abstract

The embodiment of the invention provides an end-to-end automatic driving self-vehicle trajectory planning method, system and device and a medium, and relates to the technical field of automatic driving, and the method comprises the steps: extracting BEV features from perception data; inputting the BEV features into a trajectory prediction module based on an adapter to obtain a target trajectory prediction result, and adding first target related data as training data in the training process of the trajectory prediction module; inputting the BEV features into an adapter-based map detection module to obtain a detected local map, and adding an offline map as training data in the training process of the map detection module; and inputting the target object track prediction result and the detected local map into a planning module based on an adapter to obtain a self-vehicle track planning result, and adding an offline map and / or related data of the second target object as training data in the training process of the planning module. The scheme improves the precision of the planning result.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an end-to-end autonomous driving vehicle trajectory planning method, system, device and medium. Background Art

[0002] The field of autonomous driving is generally divided into three major modules: perception, planning, and control. The three modules form a serial structure from perception to planning and then to control modules, and transmit the information of each module step by step. For this loosely coupled serial autonomous driving architecture design, it is conducive to the parallel development and optimization of professional engineers in each module, but the final optimization goals between different modules may be different, and it is difficult for the entire system to find an optimal solution to balance each module. At the same time, the serial method may cause cumulative errors in the output of the entire system. Therefore, in recent times, academia and industry have increasingly focused on the research and development of end-to-end autonomous driving architectures, such as Figure 1 As shown in Figure 1, the tightly coupled end-to-end autonomous driving system is an evolution of the traditional loosely coupled driving system.

[0003] At present, a mainstream tightly coupled end-to-end autonomous driving system is designed based on the DNN (Deep Neural Network) / Transformer (attention mechanism) model. For such an architectural design, a unified goal can be used to optimize the entire system. At the same time, the tightly coupled design avoids the cumulative error between each module and the inconsistency of global optimization. Finally, model quantization, model distillation, and model pruning techniques can be applied to such a tightly coupled system to improve the real-time performance of the model.

[0004] Existing DNN / Transformer-based end-to-end algorithms often rely only on sensor input. During the deployment of the end-to-end algorithms, other factors are not considered to improve accuracy, which reduces the accuracy of the planning results. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides an adapter-based end-to-end autonomous driving vehicle trajectory planning method to solve the technical problem of low accuracy of planning results based on existing end-to-end algorithms in the prior art. The method includes:

[0006] Extract BEV features from the perception data;

[0007] Inputting the BEV feature into an adapter-based trajectory prediction module to obtain a target object trajectory prediction result, wherein the first target object related data is added as training data during the training process of the trajectory prediction module;

[0008] Inputting the BEV features into an adapter-based map detection module to obtain a detected local map, wherein an offline map is added as training data during the training process of the map detection module;

[0009] The target object trajectory prediction result and the detected local map are input into an adapter-based planning module to obtain the vehicle trajectory planning result, wherein the offline map and / or the second target object related data are added as training data during the training process of the planning module.

[0010] The embodiment of the present invention also provides an end-to-end autonomous driving architecture system based on an adapter to solve the technical problem of low accuracy of planning results based on existing end-to-end algorithms in the prior art. The system includes:

[0011] A feature extraction module is used to extract BEV features from the perception data;

[0012] A trajectory prediction module, used for inputting the BEV feature into an adapter-based trajectory prediction module to obtain a target object trajectory prediction result, wherein the first target object related data is added as training data during the training process of the trajectory prediction module;

[0013] a map detection module, configured to input the BEV features into an adapter-based map detection module to obtain a detected local map, wherein an offline map is added as training data during the training process of the map detection module;

[0014] A planning module is used to input the target object trajectory prediction result and the detected local map into an adapter-based planning module to obtain the vehicle trajectory planning result, wherein the offline map and / or the second target object related data are added as training data during the training process of the planning module.

[0015] An embodiment of the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, any of the above-mentioned adapter-based end-to-end autonomous driving vehicle trajectory planning methods is implemented to solve the technical problem of low accuracy of planning results based on existing end-to-end algorithms in the prior art.

[0016] An embodiment of the present invention also provides a computer-readable storage medium, which stores a computer program for executing any of the above-mentioned adapter-based end-to-end autonomous driving vehicle trajectory planning methods to solve the technical problem of low accuracy of planning results based on existing end-to-end algorithms in the prior art.

[0017] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least: proposing the use of an adapter-based trajectory prediction module, an adapter-based map detection module and an adapter-based planning module to plan the trajectory of the autonomous driving vehicle, thereby realizing a tightly coupled end-to-end autonomous driving architecture; at the same time, in the training process of the adapter-based trajectory prediction module, the adapter-based map detection module and the adapter-based planning module, other data besides sensor data are added as training data, which is beneficial to improving the accuracy of the output results of each module in the adapter-based trajectory prediction module, the adapter-based map detection module and the adapter-based planning module, thereby helping to improve the accuracy of the planning results. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 is a schematic diagram of iterative evolution of an autonomous driving system provided by an embodiment of the present invention;

[0020] Figure 2 is a flow chart of an adapter-based end-to-end autonomous driving vehicle trajectory planning method provided by an embodiment of the present invention;

[0021] Figure 3 It is a principle architecture diagram of a method for implementing an adapter-based end-to-end autonomous driving vehicle trajectory planning provided by an embodiment of the present invention;

[0022] Figure 4 It is a principle architecture diagram of a map detection module provided by an embodiment of the present invention;

[0023] Figure 5 is a principle architecture diagram of a trajectory prediction module provided by an embodiment of the present invention;

[0024] Figure 6 It is a principle architecture diagram of a planning module provided by an embodiment of the present invention;

[0025] Figure 7 is a structural block diagram of a computer device provided by an embodiment of the present invention;

[0026] Figure 8 It is a structural block diagram of an adapter-based end-to-end autonomous driving architecture system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0028] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0029] In an embodiment of the present invention, an adapter-based end-to-end autonomous driving vehicle trajectory planning method is provided, such as Figure 2 As shown, the method includes:

[0030] Step S201: extracting BEV features from the perception data;

[0031] Step S202: inputting the BEV feature into an adapter-based trajectory prediction module to obtain a target object trajectory prediction result, wherein the first target object related data is added as training data during the training process of the trajectory prediction module;

[0032] Step S203: inputting the BEV feature into an adapter-based map detection module to obtain a detected local map, wherein an offline map is added as training data during the training process of the map detection module;

[0033] Step S204: inputting the target object trajectory prediction result and the detected local map into an adapter-based planning module to obtain the vehicle trajectory planning result, wherein the offline map and / or the second target object related data are added as training data during the training process of the planning module.

[0034] Depend on Figure 2It can be seen from the process shown that in the embodiment of the present invention, it is proposed to use an adapter-based trajectory prediction module, an adapter-based map detection module and an adapter-based planning module to plan the trajectory of the autonomous driving vehicle, thereby realizing a tightly coupled end-to-end autonomous driving architecture; at the same time, in the training process of the adapter-based trajectory prediction module, the adapter-based map detection module and the adapter-based planning module, data other than sensor data are added as training data, which is beneficial to improving the accuracy of the output results of each module in the adapter-based trajectory prediction module, the adapter-based map detection module and the adapter-based planning module, and thus is beneficial to improving the accuracy of the planning results.

[0035] In specific implementation, the principle architecture of the end-to-end autonomous driving vehicle trajectory planning method based on adapter model integration in this application is as follows: Figure 3 As shown, first, the original sensor data input (i.e., the above-mentioned perception data, such as lidar, camera, and millimeter-wave radar, etc.) is passed through the BEV (Bird Eye View) feature extractor designed based on Transformer to obtain BEV features (the reason for selecting the BEV feature extractor here is that the BEV perception result representation form is easier to process and converge for the planning decision algorithm). The BEV features will be input into the adapter-based trajectory prediction module and the adapter-based map detection module respectively to obtain the target object trajectory prediction result and the detected local map respectively, wherein the target object trajectory prediction result includes the explicit result and implicit result (Q, K, V) of the target object trajectory prediction, and the detected local map includes the explicit result and implicit result (Q, K, V) of the detected local map. The target object trajectory prediction result and the detected local map will be used as the input of the adapter-based planning module to obtain the final self-vehicle trajectory planning result.

[0036] In specific implementation, most of the existing local map detectors only use offline maps as true values ​​for supervised training, which reduces the accuracy of the local map detector. This embodiment proposes an adapter-based map detection module to obtain a specific method for detecting a local map, so as to improve the accuracy of the adapter-based map detection module. For example,

[0037] generating a local map detection result through a local map detector according to the BEV feature;

[0038] The local map detection result is input into the first adapter to generate the detected local map, or the local map detection result and the offline map are input into the first adapter to generate a fused local map, and the fused local map is used as the detected local map (i.e., Figure 4The fused local map shown in the figure), wherein the first adapter has been trained with both historical local map detection results as samples and historical local map detection results and historical offline maps as samples.

[0039] When implementing it, Figure 4 As shown, since the first adapter of the map detection module has been trained with both historical local map detection results and historical local map detection results and historical offline maps as samples, the input of the adapter-based map detection module can be BEV features, or BEV features and offline maps, and the offline map is an optional input. This can improve the accuracy of the final local map detection results. At the same time, in the actual algorithm deployment process of the map detection module, the high-precision map (i.e., offline map) can be removed. At this time, the input of the adapter-based map detection module is BEV features, which can get rid of the module's dependence on offline maps (the "mapless" development iteration process advocated in the field of autonomous driving) while improving the real-time performance of this module.

[0040] In specific implementation, most of the existing trajectory prediction modules only use offline maps and / or target object true values ​​as true values ​​for supervised training, which reduces the accuracy of the trajectory prediction module. This embodiment proposes an adapter-based trajectory prediction module to obtain a specific method for the target object trajectory prediction result, so as to improve the accuracy of the adapter-based trajectory prediction module. For example,

[0041] Generate a surrounding 3D target detection result by using a BEV target detector according to the BEV feature;

[0042] Inputting the 3D object detection result into a second adapter to generate a fused object perception result, or inputting the 3D object detection result and the object true value annotation data (i.e., the first object related data) into a second adapter to generate a fused object perception result, wherein the second adapter has been trained with both historical 3D object detection results as samples and historical 3D object detection results and historical object true value annotation data as samples;

[0043] Inputting the fused target object perception result and the detected local map into a first trajectory predictor, and outputting a first target trajectory prediction result;

[0044] Inputting the fused target object perception result and the detected local map into a second trajectory predictor, and outputting a second target trajectory prediction result, wherein the first trajectory predictor adopts a Rule-based algorithm and the second trajectory predictor adopts a learning-based algorithm;

[0045] Inputting the BEV characteristic map into a third trajectory predictor to generate a third target trajectory prediction result;

[0046] Input any one or any combination of the first target trajectory prediction result, the second target trajectory prediction result and the third target trajectory prediction result into a third adapter to generate a target trajectory prediction result (i.e., Figure 5 The fused target trajectory prediction result is shown).

[0047] When implementing it, Figure 5 As shown, since the second adapter has been trained with both historical 3D target detection results and historical 3D target detection results and historical target true value annotation data as samples, that is, offline maps and / or target true values ​​can be added during the trajectory prediction model reasoning process to improve the accuracy of the final target trajectory prediction result, and the input of the second adapter can be 3D target detection results, and can also be 3D target detection results and target true value annotation data, that is, the process in which offline maps and target true value annotations participate ( Figure 5 The dotted part in the figure can remove / retain only the offline map input when the real vehicle is deployed (the target object true value annotation only participates in the model training process, and there is no target object true value annotation result input when the real vehicle is deployed). This can get rid of the module's dependence on offline maps (the "mapless" development iteration process advocated in the field of autonomous driving) while improving the overall real-time performance.

[0048] When implementing it, Figure 5 As shown, trajectory predictor 1, trajectory predictor 2 and trajectory predictor 3 can selectively remove 1 to 2 of them in order to improve the overall real-time performance. During the training process, 1 to 2 trajectory predictors will be randomly removed so that the adapter can adapt to different combinations of the above 3 trajectory predictors.

[0049] In a specific implementation, since the third adapter has been trained with any one of the target trajectory prediction results as samples and with any combination of the target trajectory prediction results as samples, the third adapter can adapt to different combinations of the above three trajectory predictors, so that when deployed on a real vehicle, the first trajectory predictor 1, the second trajectory predictor 2 and the third trajectory predictor 3 can be deployed at the same time, and the demand for improving overall real-time performance can also be considered, and 1 to 2 trajectory predictors can be selectively eliminated. For example, any one or any two trajectory predictors of the first trajectory predictor 1, the second trajectory predictor 2 and the third trajectory predictor 3 are deployed to improve the deployment feasibility and flexibility of this trajectory prediction module.

[0050] In a specific implementation, it is also proposed to input the fused target object perception result and the detected local map into a first trajectory predictor, and output a first target trajectory prediction result, including:

[0051] Inputting the fused target object perception result, the detected local map and the offline map into a first trajectory predictor, and outputting a first target trajectory prediction result, wherein the first trajectory predictor has been trained with both historical fused target object perception results and historical detected local maps as samples, and with historical fused target object perception results, historical detected local maps and historical offline maps as samples;

[0052] Inputting the fused target object perception result and the detected local map into a second trajectory predictor, and outputting a second target trajectory prediction result, including:

[0053] The fused target object perception result, the detected local map and the offline map are input into a second trajectory predictor, and a second target trajectory prediction result is output, wherein the second trajectory predictor is trained with both historical fused target object perception results and historical detected local maps as samples and historical fused target object perception results, historical detected local maps and historical offline maps as samples.

[0054] like Figure 5 As shown in FIG. 1 , since the first trajectory predictor and the second trajectory predictor have been trained with the historical fused target object perception results and the historical detected local map as samples, and also with the historical fused target object perception results, the historical detected local map and the historical offline map as samples, that is, the offline map can be added in the process of trajectory prediction model reasoning to improve the accuracy of the final target object trajectory prediction result, and the input of the first trajectory predictor and the second trajectory predictor can be the fused target object perception results and the detected local map, or the fused target object perception results, the detected local map and the offline map, that is, the process in which the offline map participates ( Figure 5 The dotted part in the figure can be removed when deploying on the actual vehicle. This can get rid of the module's dependence on offline maps (the "mapless" development iteration process advocated in the field of autonomous driving) while improving overall real-time performance.

[0055] In specific implementation, most of the existing planning modules are implemented based on learning-free local path planning solutions (such as MPC). There are very few local path planners based on Transformer / DNN network design. Even if there are, they only use offline maps, target object true value trajectories, ego vehicle postures or fused target trajectory prediction results as true values / inputs for supervised training, which reduces the accuracy of the planning module. This embodiment proposes a planning module based on an adapter and a specific method for obtaining the trajectory planning result of the ego vehicle to improve the accuracy of the planning module based on the adapter. For example,

[0056] Inputting the BEV characteristic graph into a first local path planner and outputting a first local path planning result;

[0057] Inputting the target object trajectory prediction result and the detected local map into a second local path planner, and outputting a second local path planning result;

[0058] Inputting the target object trajectory prediction result and the detected local map into a third local path planner, and outputting a third local path planning result, wherein the first local path planner and the second local path planner adopt a learning-based algorithm, and the third local path planner adopts a learning-free algorithm;

[0059] The first local path planning result, the second local path planning result and the third local path planning result are input into a fourth adapter to generate a fused local path planning result.

[0060] In a specific implementation, it is also proposed to input the target object trajectory prediction result and the detected local map into a second local path planner, and output a second local path planning result, including:

[0061] Input the target object trajectory prediction result, the detected local map and the ego vehicle posture (i.e., the second target object related data) into a second local path planner, and output a second local path planning result, wherein the second local path planner is trained with both historical target object trajectory prediction results and historical detected local maps as samples and historical target object trajectory prediction results, historical detected local maps and historical ego vehicle postures as samples;

[0062] Inputting the target object trajectory prediction result and the detected local map into a third local path planner, and outputting a third local path planning result, including:

[0063] The target object trajectory prediction result, the detected local map and the ego-vehicle posture are input into a third local path planner, and a third local path planning result is output, wherein the third local path planner is trained with both historical target object trajectory prediction results and historical detected local maps as samples and historical target object trajectory prediction results, historical detected local maps and historical ego-vehicle postures as samples.

[0064] When implementing it, Figure 6As shown in , since the second local path planner and the third local path planner have been trained with both the historical target trajectory prediction results and the local map of historical detection as samples, and the historical target trajectory prediction results, the local map of historical detection and the historical ego vehicle posture as samples, the ego vehicle posture can be added in the process of model reasoning, which improves the accuracy of the planning module. At the same time, the ego vehicle posture can be removed in the actual algorithm deployment process, as shown in Figure 6 ( Figure 6 The part shown by the dotted line about the self-vehicle posture in the figure is used to improve the overall real-time performance while getting rid of the dependence of this structure on the self-vehicle posture, thereby improving the deployment feasibility of this module.

[0065] In a specific implementation, it is also proposed to input the first local path planning result, the second local path planning result and the third local path planning result into a fourth adapter to generate a fused local path planning result, including:

[0066] Inputting the vehicle position into the fourth local path planner and outputting the fourth local path planning result;

[0067] The first local path planning result, the second local path planning result, the third local path planning result and the fourth local path planning result are input into the fourth adapter to generate the fused local path planning result, wherein the fourth adapter has been trained with the historical first local path planning result, the historical second local path planning result and the historical third local path planning result as samples, and has also been trained with the historical first local path planning result, the historical second local path planning result, the historical third local path planning result and the historical fourth local path planning result as samples.

[0068] When implementing it, Figure 6 As shown, since the fourth adapter has been trained with the historical first local path planning results, the historical second local path planning results and the historical third local path planning results as samples, and has also been trained with the historical first local path planning results, the historical second local path planning results, the historical third local path planning results and the historical fourth local path planning results as samples, when deployed on a real vehicle, four local path planners can be deployed at the same time, and the first local path planner 1, the second local path planner 2 and the third local path planner 3 can also be deployed to improve the deployment feasibility and flexibility of this module in consideration of the need to improve overall real-time performance. At the same time, local path planners 1, 2 and 4 all adopt learning-based methods, and the third local path planner 3 adopts a learning-free method (such as local path optimization and speed optimization methods).

[0069] In specific implementation, the adapters included in the trajectory prediction module, map detection module, and planning module may have the same structure or different structures. For example, the adapter may be implemented by a CNN (convolutional neural network) + MLP (multi-layer perceptron) structure.

[0070] In this embodiment, a computer device is provided, such as Figure 7 As shown, it includes a memory 701, a processor 702, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any of the above-mentioned adapter-based end-to-end autonomous driving vehicle trajectory planning methods is implemented.

[0071] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0072] In this embodiment, a computer-readable storage medium is provided, which stores a computer program for executing any of the above-mentioned adapter-based end-to-end autonomous driving vehicle trajectory planning methods.

[0073] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0074] Based on the same inventive concept, an adapter-based end-to-end autonomous driving architecture system is also provided in an embodiment of the present invention, as described in the following embodiments. Since the principle of solving the problem by the adapter-based end-to-end autonomous driving architecture system is similar to the adapter-based end-to-end autonomous driving self-vehicle trajectory planning method, the implementation of the adapter-based end-to-end autonomous driving architecture system can refer to the implementation of the adapter-based end-to-end autonomous driving self-vehicle trajectory planning method, and the repetitions will not be repeated. As used below, the terms "unit" or "module" may be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0075] Figure 8 is a structural block diagram of an end-to-end autonomous driving architecture system based on an adapter according to an embodiment of the present invention. Figure 8 As shown, including:

[0076] A feature extraction module 801 is used to extract BEV features from the perception data;

[0077] A trajectory prediction module 802, configured to input the BEV feature into an adapter-based trajectory prediction module to obtain a target object trajectory prediction result, wherein the first target object related data is added as training data during the training process of the trajectory prediction module;

[0078] A map detection module 803, configured to input the BEV features into an adapter-based map detection module to obtain a detected local map, wherein an offline map is added as training data during the training process of the map detection module;

[0079] The planning module 804 is used to input the target object trajectory prediction result and the detected local map into the planning module based on the adapter to obtain the vehicle trajectory planning result, wherein the offline map and / or the second target object related data are added as training data during the training process of the planning module.

[0080] In one embodiment, a map detection module is used to generate a local map detection result through a local map detector according to the BEV feature; input the local map detection result into a first adapter to generate the detected local map, or input the local map detection result and an offline map into the first adapter to generate a fused local map, and use the fused local map as the detected local map, wherein the first adapter has been trained with both historical local map detection results as samples and historical local map detection results and historical offline maps as samples.

[0081] In one embodiment, a trajectory prediction module is used to generate a surrounding 3D target detection result through a BEV target detector according to the BEV feature; input the 3D target detection result into a second adapter to generate a fused target perception result, or input the 3D target detection result and the target true value annotation data into a second adapter to generate a fused target perception result, wherein the second adapter is trained with both historical 3D target detection results as samples and historical 3D target detection results and historical target true value annotation data as samples; input the fused target perception result and the detected local map into a first trajectory predictor to output a first target trajectory prediction result; and input the fused target perception result into a second adapter to generate a fused target perception result. The result and the detected local map are input into a second trajectory predictor to output a second target trajectory prediction result, wherein the first trajectory predictor adopts a Rule-based algorithm and the second trajectory predictor adopts a learning-based algorithm; the BEV feature map is input into a third trajectory predictor to generate a third target trajectory prediction result; any one or any combination of the first target trajectory prediction result, the second target trajectory prediction result and the third target trajectory prediction result is input into a third adapter to generate a target trajectory prediction result, wherein the third adapter has been trained with any one of the target trajectory prediction results as a sample and with any combination of the target trajectory prediction results as a sample.

[0082] In one embodiment, a trajectory prediction module is used to input the fused target object perception result, the detected local map and the offline map into a first trajectory predictor, and output a first target trajectory prediction result, wherein the first trajectory predictor has been trained with both historical fused target object perception results and historical detected local maps as samples, and with historical fused target object perception results, historical detected local maps and historical offline maps as samples; input the fused target object perception result, the detected local map and the offline map into a second trajectory predictor, and output a second target trajectory prediction result, wherein the second trajectory predictor has been trained with both historical fused target object perception results and historical detected local maps as samples, and with historical fused target object perception results, historical detected local maps and historical offline maps as samples.

[0083] In one embodiment, a planning module is used to input the BEV feature map into a first local path planner and output a first local path planning result; input the target object trajectory prediction result and the detected local map into a second local path planner and output a second local path planning result; input the target object trajectory prediction result and the detected local map into a third local path planner and output a third local path planning result, wherein the first local path planner and the second local path planner adopt a learning-based algorithm, and the third local path planner adopts a learning-free algorithm; the first local path planning result, the second local path planning result and the third local path planning result are input into a fourth adapter to generate a fused local path planning result.

[0084] In one embodiment, a planning module is used to input the target object trajectory prediction result, the detected local map and the self-vehicle posture into a second local path planner, and output a second local path planning result, wherein the second local path planner has been trained with both historical target object trajectory prediction results and historical detected local maps as samples, and with historical target object trajectory prediction results, historical detected local maps and historical self-vehicle postures as samples; input the target object trajectory prediction result, the detected local map and the self-vehicle posture into a third local path planner, and output a third local path planning result, wherein the third local path planner has been trained with both historical target object trajectory prediction results and historical detected local maps as samples, and with historical target object trajectory prediction results, historical detected local maps and historical self-vehicle postures as samples.

[0085] In one embodiment, a planning module is used to input the vehicle posture into a fourth local path planner and output a fourth local path planning result; input the first local path planning result, the second local path planning result, the third local path planning result and the fourth local path planning result into the fourth adapter to generate the fused local path planning result, wherein the fourth adapter has been trained with the historical first local path planning result, the historical second local path planning result and the historical third local path planning result as samples, and has also been trained with the historical first local path planning result, the historical second local path planning result, the historical third local path planning result and the historical fourth local path planning result as samples.

[0086] The embodiments of the present invention achieve the following technical effects: it proposes to use an adapter-based trajectory prediction module, an adapter-based map detection module and an adapter-based planning module to plan the trajectory of an autonomous driving vehicle, thereby realizing a tightly coupled end-to-end autonomous driving architecture; at the same time, in the training process of the adapter-based trajectory prediction module, the adapter-based map detection module and the adapter-based planning module, other data except sensor data are added as training data, which is beneficial to improving the accuracy of the output results of each module in the adapter-based trajectory prediction module, the adapter-based map detection module and the adapter-based planning module, thereby helping to improve the accuracy of the planning results.

[0087] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0088] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An adapter-based end-to-end autonomous driving vehicle trajectory planning method, characterized in that: include: Extract BEV features from the perception data; Inputting the BEV feature into an adapter-based trajectory prediction module to obtain a target object trajectory prediction result, wherein the first target object related data is added as training data during the training process of the trajectory prediction module; Inputting the BEV features into an adapter-based map detection module to obtain a detected local map, wherein an offline map is added as training data during the training process of the map detection module; The target object trajectory prediction result and the detected local map are input into an adapter-based planning module to obtain the vehicle trajectory planning result, wherein the offline map and / or the second target object related data are added as training data during the training process of the planning module.

2. The adapter-based end-to-end autonomous driving vehicle trajectory planning method according to claim 1, characterized in that: The BEV features are input into an adapter-based map detection module to obtain a detected local map, including: generating a local map detection result through a local map detector according to the BEV feature; The local map detection result is input into a first adapter to generate the detected local map, or the local map detection result and the offline map are input into the first adapter to generate a fused local map, and the fused local map is used as the detected local map, wherein the first adapter has been trained with both historical local map detection results as samples and historical local map detection results and historical offline maps as samples.

3. The adapter-based end-to-end autonomous driving vehicle trajectory planning method according to claim 1, characterized in that: The BEV features are input into the adapter-based trajectory prediction module to obtain the target object trajectory prediction result, including: Generate a surrounding 3D target detection result by using a BEV target detector according to the BEV feature; Inputting the 3D object detection result into a second adapter to generate a fused object perception result, or inputting the 3D object detection result and the object true value annotation data into a second adapter to generate a fused object perception result, wherein the second adapter has been trained with both historical 3D object detection results as samples and historical 3D object detection results and historical object true value annotation data as samples; Inputting the fused target object perception result and the detected local map into a first trajectory predictor, and outputting a first target trajectory prediction result; Inputting the fused target object perception result and the detected local map into a second trajectory predictor, and outputting a second target trajectory prediction result, wherein the first trajectory predictor adopts a Rule-based algorithm and the second trajectory predictor adopts a learning-based algorithm; Inputting the BEV characteristic map into a third trajectory predictor to generate a third target trajectory prediction result; Input any one or any combination of the first target trajectory prediction result, the second target trajectory prediction result, and the third target trajectory prediction result into a third adapter to generate a target trajectory prediction result, wherein the third adapter has been trained using any one of the target trajectory prediction results as a sample and also has been trained using any combination of the target trajectory prediction results as a sample.

4. The adapter-based end-to-end autonomous driving vehicle trajectory planning method according to claim 3, characterized in that: Inputting the fused target object perception result and the detected local map into a first trajectory predictor, and outputting a first target trajectory prediction result, including: Inputting the fused target object perception result, the detected local map and the offline map into a first trajectory predictor, and outputting a first target trajectory prediction result, wherein the first trajectory predictor has been trained with both historical fused target object perception results and historical detected local maps as samples, and with historical fused target object perception results, historical detected local maps and historical offline maps as samples; Inputting the fused target object perception result and the detected local map into a second trajectory predictor, and outputting a second target trajectory prediction result, including: The fused target object perception result, the detected local map and the offline map are input into a second trajectory predictor, and a second target trajectory prediction result is output, wherein the second trajectory predictor is trained with both historical fused target object perception results and historical detected local maps as samples and historical fused target object perception results, historical detected local maps and historical offline maps as samples.

5. The adapter-based end-to-end autonomous driving vehicle trajectory planning method according to claim 1, characterized in that: The target object trajectory prediction result and the detected local map are input into the planning module based on the adapter to obtain the vehicle trajectory planning result, including: Inputting the BEV characteristic graph into a first local path planner and outputting a first local path planning result; Inputting the target object trajectory prediction result and the detected local map into a second local path planner, and outputting a second local path planning result; Inputting the target object trajectory prediction result and the detected local map into a third local path planner, and outputting a third local path planning result, wherein the first local path planner and the second local path planner adopt a learning-based algorithm, and the third local path planner adopts a learning-free algorithm; The first local path planning result, the second local path planning result and the third local path planning result are input into a fourth adapter to generate a fused local path planning result.

6. The adapter-based end-to-end autonomous driving vehicle trajectory planning method according to claim 5, characterized in that: Inputting the target object trajectory prediction result and the detected local map into a second local path planner, and outputting a second local path planning result, including: Inputting the target object trajectory prediction result, the detected local map and the ego vehicle posture into a second local path planner, and outputting a second local path planning result, wherein the second local path planner is trained with both historical target object trajectory prediction results and historical detected local maps as samples and historical target object trajectory prediction results, historical detected local maps and historical ego vehicle postures as samples; Inputting the target object trajectory prediction result and the detected local map into a third local path planner, and outputting a third local path planning result, including: The target object trajectory prediction result, the detected local map and the ego-vehicle posture are input into a third local path planner, and a third local path planning result is output, wherein the third local path planner is trained with both historical target object trajectory prediction results and historical detected local maps as samples and historical target object trajectory prediction results, historical detected local maps and historical ego-vehicle postures as samples.

7. The adapter-based end-to-end autonomous driving vehicle trajectory planning method according to claim 5, characterized in that: Inputting the first local path planning result, the second local path planning result and the third local path planning result into a fourth adapter to generate a fused local path planning result, including: Inputting the vehicle position into the fourth local path planner and outputting the fourth local path planning result; The first local path planning result, the second local path planning result, the third local path planning result and the fourth local path planning result are input into the fourth adapter to generate the fused local path planning result, wherein the fourth adapter has been trained with the historical first local path planning result, the historical second local path planning result and the historical third local path planning result as samples, and has also been trained with the historical first local path planning result, the historical second local path planning result, the historical third local path planning result and the historical fourth local path planning result as samples.

8. An adapter-based end-to-end autonomous driving architecture system, characterized in that: include: A feature extraction module is used to extract BEV features from the perception data; A trajectory prediction module, used for inputting the BEV feature into an adapter-based trajectory prediction module to obtain a target object trajectory prediction result, wherein the first target object related data is added as training data during the training process of the trajectory prediction module; a map detection module, configured to input the BEV features into an adapter-based map detection module to obtain a detected local map, wherein an offline map is added as training data during the training process of the map detection module; A planning module is used to input the target object trajectory prediction result and the detected local map into an adapter-based planning module to obtain the vehicle trajectory planning result, wherein the offline map and / or the second target object related data are added as training data during the training process of the planning module.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the adapter-based end-to-end autonomous driving vehicle trajectory planning method described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the adapter-based end-to-end autonomous driving vehicle trajectory planning method according to any one of claims 1 to 7.

Images