A method, device and equipment for training autonomous driving motion planning and its model
By introducing data cropping and interaction loss functions into the autonomous driving motion planning model, the processing of perceived data and the prediction of bicycle trajectory is solved, and the accuracy and safety of the planning network are improved. .
Patent Information
- Application Number
- CN202411232587.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-09-04
AI Technical Summary
The existing Mid2Mid-based autonomous driving planning scheme has poor performance in extreme scenarios, and fails to effectively consider the importance of input information and the interaction between the bicycle and the surrounding agent trajectory, resulting in a collision risk for motion planning.
A training method for autonomous driving motion planning model was designed. The impact of perceived data on bicycle trajectory is evaluated through the data cropping module, remove unimportant data, and generate bicycle and agent trajectories using the prediction module, and introduce interactive loss functions to optimize. Network training to ensure that the bike avoids collisions with surrounding agents.
It improves the accuracy and safety of the autonomous driving planning network, can pay more effectively to important information in the scenario, reduce collision risks in extreme scenarios, and improves performance in complex road structures and dynamic traffic environments.
Smart Images

Figure CN118734923B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure belongs to the field of autonomous driving, and specifically relates to a method, device and equipment for training autonomous driving motion planning and its model. Background Art
[0002] Data-driven autonomous driving planning methods can learn complex driving patterns directly from data, and can therefore more flexibly respond to the complex road structure and dynamic traffic environment in cities, thus becoming a hot topic in autonomous driving technology research.
[0003] End2End and Mid2Mid are two major data-driven autonomous driving planning schemes. The Mid2Mid method is usually divided into a perception network and a planning network. The planning network takes the perception results as input and outputs a drivable trajectory. Compared with the End2End method, the Mid2Mid method is more modular and flexible, with a simple structure and interpretability, making the system easier to debug and more generalizable, which is conducive to driving the performance improvement of the planning network through a closed data loop.
[0004] However, existing Mid2Mid methods predict trajectories after encoding input data. First, they do not consider the different impacts of the importance of input information on ego vehicle planning. Second, they do not consider the interaction between the ego vehicle's predicted trajectory and the predicted trajectory of surrounding agents during prediction, which can easily lead to neglect of more important information in extreme scenarios. At the same time, the motion planning made has the risk of collision with surrounding agents. Therefore, existing Mid2Mid-based autonomous driving planning solutions have poor performance in extreme scenarios. Summary of the invention
[0005] The disclosed embodiment proposes an autonomous driving motion planning solution to solve the problem that the existing Mid2Mid-based autonomous driving planning solution has poor planning performance in extreme scenarios.
[0006] A first aspect of an embodiment of the present disclosure provides a method for training an autonomous driving motion planning model, comprising:
[0007] Obtain autonomous driving perception data for a preset time period and determine the trajectory of the ego vehicle and the trajectory of the agent within a preset range around it;
[0008] Constructing a motion planning neural network, wherein the motion planning neural network includes a data clipping module and a prediction module, wherein the data clipping module is used to evaluate the influence of the perception data on the motion trajectory of the ego vehicle, and remove part of the perception data according to a preset rule based on the influence, and the prediction module is used to predict the ego vehicle trajectory and the agent trajectory of a preset number of future frames based on the retained perception data;
[0009] The autonomous driving perception data is used as a training sample, and the ego vehicle trajectory and the agent trajectory are used as true values respectively, to train the motion planning neural network and generate the autonomous driving motion planning model.
[0010] In some embodiments, the autonomous driving perception data is used as a training sample, the ego vehicle trajectory and the proxy trajectory are used as true values, and the motion planning neural network is trained, including:
[0011] Predicting the vehicle trajectory of a preset number of frames based on the training samples, and determining the vehicle trajectory planning loss according to the predicted vehicle trajectory and the true value of the vehicle trajectory;
[0012] Predicting a proxy trajectory of a preset number of frames based on the training sample, and determining a proxy trajectory prediction loss according to the proxy prediction trajectory and a true value of the proxy trajectory;
[0013] Determining an interaction loss based on the ego vehicle predicted trajectory and the agent predicted trajectory, wherein the interaction loss is used to describe a collision probability between the ego vehicle predicted trajectory and the agent predicted trajectory;
[0014] The total loss is composed of the weighted sum of the ego-vehicle trajectory planning loss, the agent trajectory prediction loss and the interaction loss, and the parameters of the motion planning neural network are adjusted based on the total loss to generate the autonomous driving motion planning model.
[0015] In some embodiments, determining the interaction loss based on the ego-vehicle predicted trajectory and the agent predicted trajectory includes:
[0016] At each time step, the Euclidean distance between the vehicle and neighboring vehicles within a preset range is calculated and the shortest distance between the vehicle and all the neighboring vehicles is determined, wherein the neighboring vehicles whose Euclidean distance at the time step is greater than a preset threshold are ignored;
[0017] Determine a potential risk factor for each of the time steps based on the closest distance, wherein the potential risk factor corresponding to the closest distance less than a preset threshold is weighted according to a preset rule;
[0018] The interaction loss consists of the mean of the sum of the latent risk factors over all time steps.
[0019] In some embodiments, the motion planning neural network also includes an encoder based on a transformer structure, which is used to encode the received perception data and input the encoding result into the data cropping module.
[0020] In some embodiments, evaluating the impact of the perception data on the motion trajectory of the vehicle and removing part of the perception data according to a preset rule based on the impact includes:
[0021] Based on the attention mechanism, the influence of the connection of the attention weights between the tags in the encoding result and the tags on the movement trajectory of the vehicle is sorted;
[0022] The connection between the tag and the attention weight with a lower ranking is removed according to a preset rule.
[0023] In some embodiments, predicting the ego-vehicle and agent trajectories for a preset number of future frames based on the retained perception data includes:
[0024] The prediction module is a decoder based on a transformer structure, and the decoder predicts the trajectory of the vehicle and the trajectory of the agent for a preset number of frames in the future at the current moment based on the encoding result output by the data clipping module.
[0025] A second aspect of an embodiment of the present disclosure provides an autonomous driving motion planning method, including:
[0026] Acquire autonomous driving perception data, wherein the autonomous driving perception data includes but is not limited to the traffic light detection result of the current frame, the global route of the vehicle, the crossroad perception result, the lane line perception result, the vehicle status, and the perception information of the past preset number of frames of agents within the surrounding preset range;
[0027] The autonomous driving perception data is input into the autonomous driving motion planning model described in the first aspect of the present disclosure to predict the vehicle trajectory for a preset number of frames in the future.
[0028] A third aspect of the embodiments of the present disclosure provides a training device for an autonomous driving motion planning model, including:
[0029] A data acquisition module, used to obtain autonomous driving perception data in a preset time period and determine the trajectory of the vehicle and the trajectory of the agent within a preset range around it;
[0030] A model construction module, used to construct a motion planning neural network, wherein the motion planning neural network includes a data clipping module and a prediction module, wherein the data clipping module is used to evaluate the influence of the perception data on the motion trajectory of the ego vehicle, and remove part of the perception data according to a preset rule based on the influence, and the prediction module is used to predict the ego vehicle trajectory and the agent trajectory of a preset number of frames in the future based on the retained perception data;
[0031] The model training module is used to train the motion planning neural network using the autonomous driving perception data as training samples, the ego vehicle trajectory and the agent trajectory as true values, and generate the autonomous driving motion planning model.
[0032] A fourth aspect of the embodiments of the present disclosure provides an automatic driving motion planning device, including:
[0033] A data acquisition module is used to acquire autonomous driving perception data, wherein the autonomous driving perception data includes but is not limited to the traffic light detection result of the current frame, the global route of the vehicle, the crossroad perception result, the lane line perception result, the vehicle status, and the perception information of the surrounding agents for a preset number of frames in the past;
[0034] The trajectory planning module is used to input the autonomous driving perception data into the autonomous driving motion planning model described in the first aspect of the present disclosure to predict the vehicle trajectory for a preset number of frames in the future.
[0035] A fifth aspect of an embodiment of the present disclosure provides an autonomous driving motion planning device, including a memory and a processor:
[0036] The memory is used to store computer programs;
[0037] The processor is configured to implement the method according to the first aspect or the second aspect of the present disclosure when executing the computer program.
[0038] In summary, the training method and apparatus of the autonomous driving motion planning model, the autonomous driving motion planning method and apparatus, and the autonomous driving motion planning device provided by the embodiments of the present disclosure optimize the input perception data by designing a sorting and sparseness strategy for the perception input data, and sparsely pruning the number of tokens after encoding the perception input data and the number of connections between different tokens in the attention weights, thereby removing data that is less relevant to the self-vehicle planning in a certain proportion, retaining those data that are highly relevant to the current driving situation, and improving the accuracy and effectiveness of the planning network; considering the different effects of different surrounding information on the self-vehicle planning, some information with lower effects is removed during the motion planning network processing, and the input information is sorted by importance, so as to encourage the network to pay more attention to important information in the scene when making plans, thereby improving the planning effect; at the same time, during the planning network training process, the interaction loss function with the surrounding agents is used to encourage the self-vehicle to avoid collision with the future trajectory development of the surrounding agents as much as possible, thereby improving the safety of interaction with the surrounding agents. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The features and advantages of the present disclosure will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:
[0040] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable;
[0041] Figure 2 is a flowchart of a method for training an autonomous driving motion planning model according to some embodiments of the present disclosure;
[0042] Figure 3 It is an autonomous driving motion planning framework shown in some embodiments of the present disclosure;
[0043] Figure 4 is a flowchart of an autonomous driving motion planning method according to some embodiments of the present disclosure;
[0044] Figure 5 yes Figure 4 The method shown demonstrates the effect of obstacle avoidance on straight roads in urban areas;
[0045] Figure 6 yes Figure 4 The method shown demonstrates the effect of obstacle avoidance on cross roads in urban areas;
[0046] Figure 7 yes Figure 4 The method shown is effective in avoiding obstacles on roundabout roads in urban areas;
[0047] Figure 8 is a schematic diagram of a training device for an autonomous driving motion planning model according to some embodiments of the present disclosure;
[0048] Fig. 9 is a schematic diagram of an autonomous driving motion planning device according to some embodiments of the present disclosure;
[0049] Fig.10 It is a schematic diagram of an autonomous driving motion planning device according to an embodiment of the present disclosure.
[0050] List of reference numerals:
[0051] 1: Vehicle position DETAILED DESCRIPTION
[0052] In the detailed description below, many specific details of the present disclosure are set forth by way of example in order to provide a thorough understanding of the relevant disclosure. However, it will be apparent to one of ordinary skill in the art that the present disclosure can be implemented without these details. It should be understood that the use of the terms "system," "device," "unit," and / or "module" in the present disclosure is a method for distinguishing different parts, elements, parts, or assemblies at different levels in a sequential arrangement. However, these terms may be replaced by other expressions if other expressions can achieve the same purpose.
[0053] It should be understood that when a device, unit or module is referred to as being "on," "connected to," or "coupled to" another device, unit or module, it may be directly on, connected to, coupled to, or communicating with other devices, units or modules, or there may be intermediate devices, units or modules, unless the context clearly indicates an exception. For example, the term "and / or" as used in this disclosure includes any and all combinations of one or more of the relevant listed items.
[0054] The terms used in this disclosure are only for describing specific embodiments and are not intended to limit the scope of this disclosure. As shown in the specification and claims of this disclosure, unless the context clearly indicates an exception, the words "one", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified features, wholes, steps, operations, elements and / or components, and such expressions do not constitute an exclusive list, and other features, wholes, steps, operations, elements and / or components may also be included.
[0055] These and other features and characteristics, methods of operation, functions of related elements of the structure, combinations of parts, and economies of manufacture of the present disclosure may be better understood with reference to the following description and drawings, which form a part of the specification. However, it is to be clearly understood that the drawings are for illustration and description purposes only and are not intended to limit the scope of protection of the present disclosure. It is to be understood that the drawings are not drawn to scale.
[0056] Various structural diagrams are used in this disclosure to illustrate various variations of the embodiments of the present disclosure. It should be understood that the above or below structures are not intended to limit the present disclosure. The scope of protection of the present disclosure shall be subject to the claims.
[0057] Figure 1 is a schematic diagram of a computer system to which the present disclosure is applicable. Figure 1 The system shown includes an autonomous driving motion planning server connected to sensors and autonomous driving vehicle (hereinafter referred to as ego vehicle) data, wherein the autonomous driving motion planning server predicts the trajectory of the ego vehicle based on the perception data of the ego vehicle and the surrounding environment obtained by the sensors.
[0058] The sensor is deployed in the vehicle or agent to obtain the perception data of the vehicle and its surrounding environment, and can also be deployed on the roadside or in the air to obtain the perception data of the vehicle and its surrounding environment. The sensor includes various image sensors and various distance sensors (such as lidar, microwave radar), and can also be other types of sensors. In the field of autonomous driving, agents specifically refer to objects that occupy the right of way and may interfere with the driving of the vehicle, such as other vehicles, motor vehicles, non-motor vehicles, pedestrians, etc. The autonomous driving motion planning server can be any of a stand-alone, cluster or distributed server.
[0059] Figure 1 The system shown belongs to a data-driven autonomous driving planning scheme based on Mid2Mid. Data-driven autonomous driving planning methods can learn complex driving patterns directly from data, so they can more flexibly cope with complex road structures and dynamic traffic environments in cities, thus becoming a hot topic in autonomous driving technology research.
[0060] End2End and Mid2Mid are two main data-driven autonomous driving planning methods. Among them, the End2End method generates control signals directly from sensor inputs (such as camera images and lidar point clouds) through a deep learning model to control vehicle motion. The advantage of this method is that the system architecture is simple and can learn complex driving patterns directly from data. However, this method has a high demand for a large amount of high-quality data, and its internal decision-making process lacks interpretability, is difficult to debug, and is difficult to ensure generalization. Most of them are currently in the scientific research stage. In contrast, the Mid2Mid method is usually divided into a perception network and a planning network, with explicit input and output. Among them, the planning network of Mid2Mid takes the perception results as input and outputs a drivable trajectory. This method is more modular and flexible, with a simple structure and interpretability, making the system easier to debug and highly generalizable, which is conducive to driving the performance improvement of the planning network through a data closed loop.
[0061] However, the existing Mid2Mid method uniformly encodes the input results and then performs corresponding processing to obtain the trajectory. It does not consider the different effects of different surrounding input information on the ego vehicle planning, and does not consider removing some unimportant information during the network processing, while retaining more important information, and sorting the input information before using it to output the trajectory. Therefore, the existing methods are prone to making poor plans in some special or extreme scenarios because they do not pay attention to the more important information in the scene. At the same time, most of the existing methods will use the estimated loss of the motion prediction of the surrounding agents as an auxiliary function to participate in the training during training when considering the game with the surrounding agents, without considering the interaction between the ego vehicle trajectory and the predicted trajectory of the surrounding agents.
[0062] Figure 2is a flowchart of a method for training an autonomous driving motion planning model according to some embodiments of the present disclosure. In some embodiments, the method for training an autonomous driving motion planning model is composed of Figure 1 The autonomous driving motion planning server is shown to execute the training method of the autonomous driving motion planning model, which includes the following steps:
[0063] S210, obtaining autonomous driving perception data for a preset time period and determining the trajectory of the vehicle and the trajectory of the proxy within a preset surrounding range.
[0064] Specifically, the autonomous driving perception data includes but is not limited to the traffic light detection results of the current frame, the global route of the vehicle, the crossroad perception results, the lane line perception results, the state of the vehicle (including position, heading angle, speed, acceleration information), and the past 20 frames of information of surrounding agents (including the position, heading angle, speed, acceleration, angular velocity, and bounding box length and width information of each agent).
[0065] Determine the ego vehicle trajectory and adjacent agent trajectories based on the autonomous driving perception data or from other sources.
[0066] S220, constructing a motion planning neural network, wherein the motion planning neural network includes a data clipping module and a prediction module, the data clipping module is used to evaluate the impact of the perception data on the motion trajectory of the ego vehicle, and remove part of the perception data according to a preset rule based on the impact, and the prediction module is used to predict the ego vehicle trajectory and the agent trajectory of a preset number of frames in the future based on the retained perception data.
[0067] Figure 3 is an autonomous driving motion planning framework according to some embodiments of the present disclosure. Figure 3 As shown, the autonomous driving motion planning framework includes an encoder, a data clipping module and a decoder.
[0068] First, various perception data obtained in s210 are input into a transformer-based encoder structure for data encoding.
[0069] The encoded perception results are then input into the data clipping module. The data clipping module uses the network's attention map to sort the impact of the connection between each token and the attention weight between tokens in the input encoding results on the vehicle's motion trajectory, and removes the connection between tokens and attention weights with lower rankings at a fixed ratio. This method reduces the amount of data in the encoding results, while retaining the encoded data that is highly relevant to the current driving situation, which is conducive to the motion planning neural network focusing on more important scene information.
[0070] Specifically, assuming is the label input to the data clipping module, where This is the token for the vehicle status. There is only one token. is the token corresponding to the surrounding agents, including N tokens, where N is the number of input agent information; is the token corresponding to the map element, which contains L tokens, so It contains M=1+N+L tokens in total. In the self-attention process of transformer, the input After three linear layers, query (Q), key (K), and value (V) matrices are generated respectively, and the attention operation is performed as follows:
[0071]
[0072] Q and K can be seen as the concatenation of M token vectors. , K = For the i-th token, the attention probability Represents each key With query The output can be viewed as a linear combination of all value vectors, where is the coefficient of the combination. Therefore, it can be assumed that Represents the importance score of all tokens. A larger attention value indicates that the corresponding token is important to the query token.
[0073] Therefore, the attention map of the ego-car state token is obtained by Indicates the contribution of the token to the planning result. The first element of As a criterion for selecting tokens. In multi-head self-attention, the average attention probability of all heads is used The value of is used as the basis for sorting and selecting tokens, where .
[0074] Finally, the sparsely pruned encoding result is input into a transformer-based decoder structure, which outputs the motion trajectory of the current vehicle for the next 80 frames and the predicted trajectory of the surrounding agents for the next 80 frames.
[0075] S230, using the autonomous driving perception data as training samples, the ego vehicle trajectory and the agent trajectory as true values, respectively, to train the motion planning neural network and generate the autonomous driving motion planning model.
[0076] Specifically, in some embodiments of the present disclosure, the main loss function for training the motion planning neural network is the ego vehicle trajectory planning loss, and also includes an auxiliary loss function consisting of the agent trajectory prediction loss and the interaction loss. Among them:
[0077] Based on the training samples, the ego vehicle trajectory of a preset number of frames is predicted, and the ego vehicle trajectory planning loss is determined according to the ego vehicle predicted trajectory and the ego vehicle trajectory true value. The calculation formula of the ego vehicle trajectory planning loss is as follows:
[0078]
[0079] in: Planning loss for the ego vehicle trajectory. represents the total planning time, represents the position of the vehicle's predicted trajectory at time t, represents the position of the true value of the vehicle trajectory at the tth time point, Indicates calculation of Manhattan distance.
[0080] Based on the training sample, a proxy trajectory of a preset number of frames is predicted, and the proxy trajectory prediction loss is determined according to the proxy prediction trajectory and the true value of the proxy trajectory. The calculation formula of the proxy trajectory prediction loss is as follows:
[0081]
[0082] in, is the proxy trajectory prediction loss, represents the total number of predicted agents, represents the total planning time, represents the position of the trajectory predicted by the nth agent at time t, represents the position of the true value of the nth agent trajectory at the tth time point, Indicates calculation of Manhattan distance.
[0083] The interaction loss is used to describe the collision probability between the ego vehicle’s predicted trajectory and the agent’s predicted trajectory.
[0084] make represents the position of the vehicle at time step t, represents the position of the kth neighbor vehicle at time step t. The trajectory of the vehicle is recorded as , the trajectory of the neighbor vehicle is recorded as . Define the validity matrix of neighbor vehicles in the future time step ,in indicates that the neighbor vehicle is valid at time step t, Indicates invalid (invalid means that at this time step, the distance between the neighbor vehicle and the vehicle is greater than the preset threshold, that is, the neighbor vehicle will not affect the trajectory of the vehicle).
[0085] The calculation method of interaction loss is as follows:
[0086] First, the Euclidean distance between the ego vehicle and all neighboring vehicles is calculated at each time step t
[0087]
[0088] in, is the time step The Euclidean distance between the vehicle and its neighbor vehicles, is the position matrix of all neighbor vehicles.
[0089] For invalid neighbor vehicles, the distance is set to a large value (such as 1000) to ignore them, so the final Euclidean distance between the ego vehicle and its neighbor vehicles is:
[0090]
[0091] in, is the Euclidean distance between the ego vehicle and its neighbor vehicles at time step t, is the position matrix of all neighbor vehicles, is the neighbor vehicle at time step The effectiveness matrix, is the default value.
[0092] By time step The Euclidean distance between the ego vehicle and all neighboring vehicles can be obtained by time step The shortest distance between the vehicle and all neighboring vehicles:
[0093]
[0094] The potential risk factor is then calculated for each time step:
[0095]
[0096] And for distance less than the safety threshold The weighted parts are:
[0097]
[0098] in, is the time step The shortest distance between the vehicle and all neighboring vehicles, is the time step potential risk factors, is the safety threshold, is the distance weight.
[0099] Finally, the interaction loss is obtained by averaging the sum of the potential risk factors at all time steps:
[0100]
[0101] in, is the interaction loss, is the batch size, represents the total planning time, is the time step potential risk factors.
[0102] The overall loss function is:
[0103]
[0104] in: is the total loss function, is the ego-vehicle trajectory planning loss, is the ego-vehicle trajectory planning loss weight, is the proxy trajectory prediction loss, is the proxy trajectory prediction loss weight, is the interaction loss, is the interaction loss weight.
[0105] Train the motion planning neural network based on the total loss function to generate an autonomous driving motion planning model
[0106] Figure 4 is a flowchart of an autonomous driving motion planning method according to some embodiments of the present disclosure. In some embodiments, the autonomous driving motion planning method is composed of Figure 1 The autonomous driving motion planning server is shown to execute the autonomous driving motion planning method, which includes the following steps:
[0107] S410, obtaining autonomous driving perception data, wherein the autonomous driving perception data includes but is not limited to a traffic light detection result of a current frame, a global route of the vehicle, a crossroads perception result, a lane line perception result, a self-vehicle state, and perception information of a preset number of past frames of agents within a preset range of the surroundings;
[0108] S420, inputting the autonomous driving perception data into the Figure 2 The autonomous driving motion planning model trained by the method predicts the vehicle trajectory for a preset number of frames in the future.
[0109] The present disclosure optimizes the input perception data by sparsely pruning the number of tokens after encoding the perception input data and the number of connections between different tokens in the attention weight, removes data that is less relevant to the vehicle planning at a certain ratio, retains data that is highly relevant to the current driving situation, and improves the accuracy and effectiveness of the planning network. At the same time, by designing an interaction loss function with surrounding agents, it is achieved that during the planning network training process, the vehicle is encouraged to avoid collisions with the future trajectory development of surrounding agents as much as possible, thereby improving the safety of interaction with surrounding agents.
[0110] The Mid2Mid-based autonomous driving planning method proposed in this paper was tested on the nuplan public dataset and its corresponding simulator. Some of the experimental results are visualized as follows:
[0111] Figure 5 yes Figure 4 The method shown is effective in avoiding obstacles on straight roads in urban areas. Figure 5 The trajectory segment in front of the vehicle position 1 in the left figure represents the current trajectory of the vehicle. If it continues to run along this trajectory, there will be a risk of collision with the vehicle in front that is stationary waiting for the traffic light. As shown in the right figure, the trajectory of the vehicle is greatly shortened, which means that measures have been taken to slow down and stop, avoiding a collision with the stationary vehicle in front.
[0112] Figure 6 yes Figure 4 The method shown is effective in avoiding obstacles at a crossroad in an urban area. Figure 6 The trajectory segment in front of the vehicle position 1 in the middle represents the current trajectory of the vehicle. If it continues to run along this trajectory, there will be a risk of collision with the vehicle in front that is stationary waiting for the traffic light. As shown in the left figure, the trajectory of the vehicle is short, which means that deceleration measures are taken. As shown in the right figure, there is almost no trajectory, which means that the vehicle stops and avoids a collision with the stationary vehicle in front.
[0113] Figure 7 yes Figure 4 The method shown in the figure demonstrates the obstacle avoidance effect on the urban roundabout road. The trajectory line segment in front of the vehicle position 1 represents the current trajectory of the vehicle. The trajectory on the left shows that the vehicle is preparing to change lanes and start, but a vehicle in the next lane is driving next to it. As shown in the right figure, the trajectory of the vehicle almost disappears, which means that the vehicle has taken measures to stop and avoid collision with the vehicle next to it.
[0114] As can be seen from the above, the Mid2Mid-based autonomous driving planning method proposed in the present disclosure can easily cope with the complex road structure and dynamic traffic environment in the city, reduce the risk of collision, and achieve better planning performance in extreme environments.
[0115] Figure 8is a schematic diagram of a training device for an autonomous driving motion planning model according to some embodiments of the present disclosure. Figure 8 As shown, the training device 800 of the autonomous driving motion planning model includes a data acquisition module 810, a model framework module 820 and a model training module 830. In some embodiments of the present disclosure, the training function of the autonomous driving motion planning model can be performed by Figure 1 The autonomous driving motion planning server is shown as follows.
[0116] The data acquisition module 810 is used to acquire the autonomous driving perception data in a preset time period and determine the trajectory of the vehicle and the trajectory of the agent within a preset range around it;
[0117] A model building module 820 is used to construct a motion planning neural network, wherein the motion planning neural network includes a data clipping module and a prediction module, wherein the data clipping module is used to evaluate the influence of the perception data on the motion trajectory of the ego vehicle, and remove part of the perception data according to a preset rule based on the influence, and the prediction module is used to predict the ego vehicle trajectory and the agent trajectory of a preset number of future frames based on the retained perception data;
[0118] The model training module 830 is used to train the motion planning neural network using the autonomous driving perception data as training samples, the ego vehicle trajectory and the agent trajectory as true values, and generate the autonomous driving motion planning model.
[0119] Fig. 9 is a schematic diagram of an automatic driving motion planning device according to some embodiments of the present disclosure. Fig. 9 As shown, the autonomous driving motion planning device 900 includes a data acquisition module 910 and a trajectory planning module 920. In some embodiments of the present disclosure, the autonomous driving motion planning function may be provided by Figure 1 The autonomous driving motion planning server is shown as follows.
[0120] The data acquisition module 910 is used to acquire autonomous driving perception data, wherein the autonomous driving perception data includes but is not limited to the traffic light detection result of the current frame, the global route of the vehicle, the crossroad perception result, the lane line perception result, the vehicle status, and the perception information of the surrounding agents for the past preset number of frames;
[0121] Trajectory planning module 920, used to input the autonomous driving perception data into Figure 2 The autonomous driving motion planning model trained by the method predicts the vehicle trajectory for a preset number of frames in the future.
[0122] One embodiment of the present disclosure provides an autonomous driving motion planning device. Fig.10As shown, the autonomous driving motion planning device 1000 includes a memory 1020 and a processor 1010, wherein the memory 1020 is used to store a computer program; and the processor 1010 is used to implement when executing the computer program. Figure 2 S210-S230 or Figure 4 The method described in S410-S420.
[0123] In summary, the training method and apparatus of the autonomous driving motion planning model, the autonomous driving motion planning method and apparatus, and the autonomous driving motion planning device provided by the embodiments of the present disclosure optimize the input perception data by designing a sorting and sparseness strategy for the perception input data, and sparsely pruning the number of tokens after encoding the perception input data and the number of connections between different tokens in the attention weights, thereby removing data that is less relevant to the self-vehicle planning in a certain proportion, retaining those data that are highly relevant to the current driving situation, and improving the accuracy and effectiveness of the planning network; considering the different effects of different surrounding information on the self-vehicle planning, some information with lower effects is removed during the motion planning network processing, and the input information is sorted by importance, so as to encourage the network to pay more attention to important information in the scene when making plans, thereby improving the planning effect; at the same time, during the planning network training process, the interaction loss function with the surrounding agents is used to encourage the self-vehicle to avoid collision with the future trajectory development of the surrounding agents as much as possible, thereby improving the safety of interaction with the surrounding agents.
[0124] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding descriptions in the aforementioned device embodiments and will not be repeated here.
[0125] Although the subject matter described herein is provided in the general context of being executed in conjunction with the execution of operating systems and application programs on computer systems, it will be appreciated by those skilled in the art that other implementations may also be performed in conjunction with other types of program modules. In general, program modules include routines, programs, components, data structures, and other types of structures that perform specific tasks or implement specific abstract data types. It will be appreciated by those skilled in the art that the subject matter described herein may be practiced using other computer system configurations, including handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc., and may also be used in a distributed computing environment in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.
[0126] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.
[0127] It should be understood that the above specific embodiments of the present disclosure are only used to illustrate or explain the principles of the present disclosure, and do not constitute a limitation of the present disclosure. Therefore, any modifications, equivalent substitutions, improvements, etc. made without departing from the spirit and scope of the present disclosure should be included in the protection scope of the present disclosure. In addition, the claims attached to the present disclosure are intended to cover all changes and modifications that fall within the scope and boundaries of the attached claims, or the equivalent forms of such scope and boundaries.
Claims
1. A training method for an autonomous driving motion planning model, characterized in that: include: Obtain autonomous driving perception data for a preset time period and determine the trajectory of the ego vehicle and the trajectory of the agent within a preset range around it; Constructing a motion planning neural network, wherein the motion planning neural network includes a data clipping module and a prediction module, wherein the data clipping module is used to evaluate the influence of the perception data on the trajectory of the ego vehicle, and remove part of the perception data according to a preset rule based on the influence, and the prediction module is used to predict the trajectory of the ego vehicle and the trajectory of the agent for a preset number of frames in the future based on the retained perception data; Using the autonomous driving perception data as training samples, the ego vehicle trajectory and the agent trajectory as true values, respectively, to train the motion planning neural network and generate the autonomous driving motion planning model; The step of using the autonomous driving perception data as training samples, the ego vehicle trajectory and the agent trajectory as true values, and training the motion planning neural network includes: Predicting the vehicle trajectory of a preset number of frames based on the training samples, and determining the vehicle trajectory planning loss according to the predicted vehicle trajectory and the true value of the vehicle trajectory; Predicting a proxy trajectory of a preset number of frames based on the training sample, and determining a proxy trajectory prediction loss according to the proxy prediction trajectory and a true value of the proxy trajectory; Determining an interaction loss based on the ego vehicle predicted trajectory and the agent predicted trajectory, wherein the interaction loss is used to describe a collision probability between the ego vehicle predicted trajectory and the agent predicted trajectory; The total loss is composed of the weighted sum of the ego-vehicle trajectory planning loss, the agent trajectory prediction loss and the interaction loss, and the parameters of the motion planning neural network are adjusted based on the total loss to generate the autonomous driving motion planning model.
2. The method according to claim 1, characterized in that: The determining of the interaction loss based on the ego vehicle predicted trajectory and the agent predicted trajectory comprises: At each time step, the Euclidean distance between the vehicle and neighboring vehicles within a preset range is calculated and the shortest distance between the vehicle and all the neighboring vehicles is determined, wherein the neighboring vehicles whose Euclidean distance at the time step is greater than a preset threshold are ignored; Determine a potential risk factor for each of the time steps based on the closest distance, wherein the potential risk factor corresponding to the closest distance less than a preset threshold is weighted according to a preset rule; The interaction loss consists of the mean of the sum of the latent risk factors over all time steps.
3. The method according to claim 1, characterized in that: The motion planning neural network also includes an encoder based on a transformer structure, which is used to encode the received perception data and input the encoding result into the data clipping module.
4. The method according to claim 3, characterized in that: The evaluating the influence of the perception data on the trajectory of the vehicle and removing part of the perception data according to a preset rule based on the influence comprises: Based on the attention mechanism, sorting the influence of the connection of the attention weights between the tags in the encoding result and the ego-vehicle trajectory; The connection between the tag and the attention weight with a lower ranking is removed according to a preset rule.
5. The method according to claim 4, characterized in that: The predicting of the ego-vehicle and agent trajectories for a preset number of frames in the future based on the retained perception data includes: The prediction module is a decoder based on a transformer structure, and the decoder predicts the trajectory of the vehicle and the trajectory of the agent for a preset number of frames in the future at the current moment based on the encoding result output by the data clipping module.
6. A method for autonomous driving motion planning, characterized in that: include: Acquire autonomous driving perception data, wherein the autonomous driving perception data includes but is not limited to the traffic light detection result of the current frame, the global route of the vehicle, the crossroad perception result, the lane line perception result, the vehicle status, and the perception information of the past preset number of frames of agents within the surrounding preset range; The autonomous driving perception data is input into the autonomous driving motion planning model trained according to the method according to any one of claims 1-5 to predict the trajectory of the vehicle for a preset number of frames in the future.
7. A training device for an autonomous driving motion planning model, characterized in that: include: Data acquisition module, used to acquire autonomous driving perception data for a preset time period and determine the trajectory of the vehicle and the trajectory of the agent within a preset range around it; A model construction module, used to construct a motion planning neural network, wherein the motion planning neural network includes a data clipping module and a prediction module, wherein the data clipping module is used to evaluate the influence of the perception data on the trajectory of the ego vehicle, and remove part of the perception data according to a preset rule based on the influence, and the prediction module is used to predict the trajectory of the ego vehicle and the trajectory of the agent for a preset number of frames in the future based on the retained perception data; A model training module, configured to train the motion planning neural network using the autonomous driving perception data as training samples, the ego vehicle trajectory and the proxy trajectory as true values, and generate the autonomous driving motion planning model; The step of using the autonomous driving perception data as training samples, the ego vehicle trajectory and the agent trajectory as true values, and training the motion planning neural network includes: Predicting the vehicle trajectory of a preset number of frames based on the training samples, and determining the vehicle trajectory planning loss according to the predicted vehicle trajectory and the true value of the vehicle trajectory; Predicting a proxy trajectory of a preset number of frames based on the training sample, and determining a proxy trajectory prediction loss according to the proxy prediction trajectory and a true value of the proxy trajectory; Determining an interaction loss based on the ego vehicle predicted trajectory and the agent predicted trajectory, wherein the interaction loss is used to describe a collision probability between the ego vehicle predicted trajectory and the agent predicted trajectory; The total loss is composed of the weighted sum of the ego-vehicle trajectory planning loss, the agent trajectory prediction loss and the interaction loss, and the parameters of the motion planning neural network are adjusted based on the total loss to generate the autonomous driving motion planning model.
8. An automatic driving motion planning device, characterized in that: include: A data acquisition module is used to acquire autonomous driving perception data, wherein the autonomous driving perception data includes but is not limited to the traffic light detection result of the current frame, the global route of the vehicle, the crossroad perception result, the lane line perception result, the vehicle status, and the perception information of the surrounding agents for a preset number of frames in the past; The trajectory planning module is used to input the autonomous driving perception data into the autonomous driving motion planning model trained according to the method according to any one of claims 1-5 to predict the trajectory of the vehicle for a preset number of frames in the future.
9. An automatic driving motion planning device, characterized in that: Including memory and processor: The memory is used to store computer programs; The processor is configured to implement the method according to any one of claims 1 to 5 or claim 6 when executing the computer program.
