Vehicle motion planning method, system and equipment based on deep learning and medium

Abstract

The invention belongs to the technical field of automatic driving, and discloses a vehicle motion planning method, system and device based on deep learning and a medium, and the method comprises the steps: obtaining real-time vehicle sensing data of a vehicle in a vehicle driving process, the real-time vehicle sensing data comprising vehicle-mounted camera shooting data, radar measurement data and GPS / IMU sensor data; extracting aerial view features of the real-time vehicle sensing data; inputting the aerial view features into a trajectory generation model for curve fitting, and outputting a plurality of curve trajectories; wherein the trajectory generation model is constructed based on a deep learning model; performing collision detection and dynamic violation detection on each curve track to obtain a plurality of candidate tracks; and selecting an optimal trajectory from the candidate trajectories based on a preset comprehensive optimal selection index as an execution trajectory of the vehicle. According to the method, a smooth and stable track can be generated in a complex scene, and meanwhile, the method has high efficiency, flexibility and expandability.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving, and particularly relates to a vehicle motion planning method, system, device and medium based on deep learning. Background Art

[0002] With the rapid development of autonomous driving technology, the trajectory planning of vehicles has become one of the key links to ensure the safety and reliability of autonomous driving systems. Traditional trajectory planning methods are mostly based on rules or optimization algorithms. Although they can achieve path planning to a certain extent, they often lack sufficient flexibility and adaptability in complex dynamic environments.

[0003] In recent years, significant progress has been made in the application of deep learning technology in the field of autonomous driving, especially in environmental perception and object detection. However, how to effectively apply deep learning methods to trajectory planning and combine the dynamic characteristics of vehicles remains a challenging research topic. Summary of the Invention

[0004] The purpose of the present invention is to provide a vehicle motion planning method, system, device and medium based on deep learning to solve the problems existing in the above-mentioned prior art.

[0005] To achieve the above purpose, the present invention provides a vehicle motion planning method based on deep learning, including:

[0006] Obtaining real-time vehicle sensing data of the vehicle during driving, where the real-time vehicle sensing data includes data captured by on-vehicle cameras, radar measurement data, and GPS / IMU sensor data;

[0007] Extracting the bird's-eye view features of the real-time vehicle sensing data;

[0008] Inputting the bird's-eye view features into a trajectory generation model for curve fitting to output multiple curve trajectories; wherein, the trajectory generation model is constructed based on a deep learning model;

[0009] Performing collision detection and dynamic violation detection on each of the curve trajectories to obtain a number of candidate trajectories;

[0010] Selecting an optimal trajectory from each of the candidate trajectories based on a preset comprehensive optimization index as the execution trajectory of the vehicle.

[0011] Optionally, the extracting the bird's-eye view features of the real-time vehicle sensing data specifically includes:

[0012] Aligning the real-time vehicle sensing data to a unified coordinate system, removing noise and redundancy to obtain preprocessed sensing data;

[0013] Project the preprocessed vehicle sensing data onto a two-dimensional grid coordinate system to form bird's-eye view features, and record the true values of the vehicle state and trajectory.

[0014] Optionally, the training process of the trajectory generation model specifically includes:

[0015] Obtain training data, where the training data includes vehicle sensing training data and corresponding true driving trajectories;

[0016] Input the training data into the trajectory generation model for curve fitting, and perform multi-stage training according to the target loss function to obtain a trained trajectory generation model.

[0017] Optionally, the design process of the target loss function specifically includes:

[0018] Design the target loss function based on dynamic constraints, where the dynamic constraints include speed constraints, acceleration constraints, friction circle constraints, and sideslip angle constraints.

[0019] Optionally, the processing process of the trajectory generation model specifically includes:

[0020] Based on a temporal deep learning network, extract the historical features of the bird's-eye view features, and generate multiple curve trajectories at once at the decoding end.

[0021] Optionally, the collision detection and dynamic violation detection for each of the curve trajectories specifically include:

[0022] Interpolate each of the curve trajectories to obtain a discrete point sequence;

[0023] Calculate the distance between the discrete points of each curve trajectory and the obstacles. If a collision is detected, the corresponding curve is removed;

[0024] Perform violation detection of speed, acceleration, friction circle, and sideslip angle on the remaining curve trajectories. If a violation occurs, discard them.

[0025] Optionally, the selection process of the optimal trajectory specifically includes:

[0026] Construct a comprehensive optimization index based on the curvature, steering cost, distance from the trajectory to the target point, or path length of each candidate trajectory;

[0027] Select the optimal trajectory as the execution trajectory of the vehicle based on the comprehensive optimization index.

[0028] A vehicle motion planning system based on deep learning, including:

[0029] A data acquisition module, configured to obtain real-time vehicle sensing data of a vehicle during driving, where the real-time vehicle sensing data includes data captured by an in-vehicle camera, radar measurement data, and GPS / IMU sensor data;

[0030] A trajectory generation and screening module, configured to extract bird's-eye view features of the real-time vehicle sensing data; input the bird's-eye view features into a trajectory generation model for curve fitting, and output multiple curve trajectories; wherein, the trajectory generation model is constructed based on a deep learning model; perform collision detection and dynamic violation detection on each of the curve trajectories to obtain a number of candidate trajectories;

[0031] A final trajectory optimization module, configured to select an optimal trajectory from each of the candidate trajectories as the execution trajectory of the vehicle according to a preset comprehensive optimization index.

[0032] An electronic device, including a memory and a processor, where the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the described vehicle motion planning method based on deep learning.

[0033] A computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the described vehicle motion planning method based on deep learning.

[0034] The technical effect of the present invention is:

[0035] By combining vehicle dynamics constraints and deep learning technology, the present invention realizes safe, feasible, and comfortable autonomous driving trajectory planning. It can not only effectively avoid collisions, but also generate smooth and stable trajectories in complex scenarios, and at the same time has high efficiency, flexibility, and scalability, and has important application value in the field of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0037] The drawings constituting a part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation to this application. In the drawings:

[0038] Figure 1 It is a flowchart in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0039] A detailed description of various exemplary embodiments of the present invention will now be given. This detailed description should not be considered as a limitation of the present invention, but rather as a more detailed description of certain aspects, features, and implementation manners of the present invention.

[0040] It should be understood that the terms described in the present invention are only for describing specific embodiments and are not used to limit the present invention. Additionally, for the numerical ranges in the present invention, it should be understood that each intermediate value between the upper and lower limits of the range is also specifically disclosed. Each intermediate value within any stated value or stated range, as well as each smaller range between any other stated value or intermediate value within the stated range, is also included in the present invention. The upper and lower limits of these smaller ranges can be independently included or excluded from the range.

[0041] Without departing from the scope or spirit of the present invention, various improvements and changes can be made to the specific embodiments of the present invention's specification, which are obvious to those skilled in the art. Other embodiments obtained from the specification of the present invention are obvious to those skilled in the art. The specification and embodiments of this application are merely exemplary.

[0042] Regarding the use of "comprising", "including", "having", "containing", etc. in this article, they are all open-ended terms, meaning including but not limited to.

[0043] It should be noted that, without conflict, the embodiments in this application and the features in the embodiments can be combined with each other. The following will refer to the accompanying drawings and combine with the embodiments to detail this application.

[0044] As Figure 1 shown, in this embodiment, a vehicle motion planning method based on deep learning is provided, including: obtaining real-time vehicle sensing data of the vehicle during driving, where the real-time vehicle sensing data includes data captured by an on-vehicle camera, radar measurement data, and GPS / IMU sensor data; extracting bird's-eye view features of the real-time vehicle sensing data; inputting the bird's-eye view features into a trajectory generation model for curve fitting to output multiple curve trajectories; where the trajectory generation model is constructed based on a deep learning model; performing collision detection and dynamic violation detection on each of the curve trajectories to obtain several candidate trajectories; and selecting an optimal trajectory from each of the candidate trajectories based on a preset comprehensive optimization index as the execution trajectory of the vehicle.

[0045] This embodiment provides a motion planning method based on deep learning that combines vehicle dynamics constraints. This method evaluates and selects an optimal trajectory by combining vehicle dynamics constraints, thereby achieving safe, feasible, and comfortable autonomous driving trajectory planning.

[0046] The specific implementation process of this embodiment includes:

[0047] Offline Training:

[0048] Step S1. Generate BEV features from multi-sensor data.

[0049] Step S1.1. Sensor data acquisition and synchronization. To generate high-precision BEV features, it is recommended to include vehicle-mounted camera, lidar, millimeter-wave radar, and IMU sensor data.

[0050] Vehicle-mounted camera: Acquire images of the front and surrounding environment for identifying lane lines, traffic signs, obstacle contours, etc.

[0051] Lidar: Provide high-precision 3D point clouds to determine the position and size of obstacles.

[0052] Millimeter-wave radar: Capture the relative speed and distance of surrounding vehicles / pedestrians.

[0053] GPS / IMU: Record key motion states of the vehicle body such as position, speed, acceleration, and heading angle.

[0054] Align the source data to a unified coordinate system, remove noise and redundancy, and obtain a structured perception result.

[0055] Step S1.2. BEV projection and map discretization: Project the fused perception information onto a two-dimensional grid coordinate system to form BEV features with a resolution of 0.2 meters. Mark elements such as static obstacles, dynamic obstacles, lane lines, and traffic signs on the BEV map to obtain F BEV ={C(x,y)|(x,y)∈Ω}, where Ω is the set of discrete grids, and F BEV represents the BEV features within the entire scene, and C(x,y) represents the environmental feature information at the position (x,y).

[0056] Step S1.3. Vehicle state and trajectory ground truth recording: During actual road tests or simulations, record the driving state of the vehicle VehicleState t =(x t ,y t ,v t ,θ t ,a t ,δ t ), and establish a deep learning training mapping by corresponding the time stamp with the BEV features:

[0057] ({F BEV,t-l+1 ,…,F BEV,t},{VehicleState t-l+1 ,…,VehicleState t )→True trajectory t:t+T

[0058] Among them, the BEV features and vehicle states at the last l historical moments are taken as inputs, and the output is the true trajectory points at the next T moments. It is used for imitation learning.

[0059] Taking the true driving trajectory as the ground truth for subsequent imitation learning ensures that the model can learn the actual operation patterns of humans in various scenarios. By introducing information from multiple historical moments, the network can perceive the continuity of vehicle motion trends and environmental changes, improving the accuracy of trajectory prediction.

[0060] Step S2: Multi-scheme output of deep learning fitting curves - basic dynamics constraint stage.

[0061] In this embodiment, the deep learning network generates multiple candidate trajectories at once for subsequent collision detection, dynamic violation assessment, and optimal output.

[0062] Step S2.1: From multi-historical inputs to fitting curve output. Define the sequential deep learning network as Φ, extract historical features, and generate multiple curve trajectories f k (s) at the decoding end. Here, a Transformer or LSTM structure can be adopted:

[0063] f k (s) = Φ({F BEV,t-l+1 , …, F BEV,t}, {VehicleState t-l+1 , …, VehicleState t})

[0064] Among them, f k (s) is determined by the following parameter α k :

[0065]

[0066] Thus, the curve C k is completely described by α k = (a0, a1, a2, a3, b0, b1, b2, b3), where s is the time parameter of the curve.

[0067] Among them, α k represents the polynomial coefficient vector of the k-th fitting curve, k = 1, …, K. The network outputs K groups of coefficients at the decoding end, which can be interpolated into a discrete point sequence of C k (s) offline or online. This method ensures that the discrete points fall on a smooth curve, making it easier to satisfy vehicle dynamics laws and facilitating collision detection.

[0068] Step S2.2, Basic Dynamics Constraints.

[0069] S2.21 Speed and Acceleration Limitations:

[0070] If curve C k (s) corresponds to time t + i, the speed and acceleration are derived. The requirement is

[0071]

[0072] The violation degree velAccViolation(α k ) is:

[0073]

[0074] S2.22 The loss function is designed as:

[0075]

[0076] Where: C k (s i ) represents the coordinates k of the k-th curve α at parameter s i (corresponding to time t + i), p t+i =(x t+i , y t+i ) being the true trajectory point; velAccViolation(α k ) corresponds to the speed and acceleration violation degree; λ basic is a fixed weight coefficient, set before training and not as a learnable parameter to ensure fast convergence.

[0077] Through the training of this stage, the network can learn the basic dynamics and output K curves, each of which can fit the true trajectory and does not violate the simple speed / acceleration limitations.

[0078] Step S3: Introduce Advanced Dynamics: Friction Circle, Side Slip Angle (Strengthened in Stages).

[0079] Step S3.1, Friction Circle:

[0080] It is required that the longitudinal and lateral accelerations of the vehicle satisfy for all times If it exceeds the limit, fricViolation(α k ) is accumulated to represent the violation degree:

[0081]

[0082] Among them, is the curve C k The lateral acceleration at time t+i; is the curve C k The longitudinal acceleration at time t+i; μ is the road surface friction coefficient; g is the acceleration due to gravity (about 9.81 m / s);

[0083] Step S3.2, Slip Angle:

[0084] The slip angle needs to be limited at high speeds or sharp turns or the sideslip angle Exceeding it will result in a penalty. The slip angle violation degree slipViolation(α k ):

[0085]

[0086] Step S3.3, Phased strengthening:

[0087] Phase II: Add a small weight λ to the loss function f , λ s Preheat the friction circle and slip angle violation;

[0088] Phase III: Set strict thresholds μ, α max , and greatly increase λ f , λ s Penalty, and sharply increase the loss for the ray curve violation situation.

[0089] New loss function design:

[0090]

[0091] The network strictly adheres to the physical limits during high-speed turning and rapid acceleration and deceleration in the later stage of training, and multiple output curves will not get out of control.

[0092] Step S4: Offline training is completed and the model is solidified;

[0093] Through the primary stage (Step S2) and advanced dynamics strengthening (Step S3), the network learns the mapping Φ from multi-history BEV+VehicleState to multiple curve parameters, ensuring that the output curves meet the physical limits of vehicle speed, acceleration, grip, and slip angle. At this time, the network parameters are solidified and the offline training is ended.

[0094] Online inference:

[0095] Step S5: Online fuse sensors and generate BEV.

[0096] Step S5.1, Real-time multi-sensor acquisition and BEV update:

[0097] During the actual driving of the vehicle, the camera, LiDAR, millimeter-wave radar, and GPS / IMU data are periodically collected and fused in the same method as in step S1;

[0098] Similarly, project it onto the ground to form the BEV feature F at the latest moment BEV,t and obtain the vehicle's nearest l frame states {VehicleState t-l+1 ,…,VehicleState t};

[0099] If the obstacle detection and recognition and the safety radius can be used in subsequent collision checks.

[0100] Step S6: Generation of multiple curves and final optimization of "removing when hitting an obstacle"

[0101] Step S6.1: The network outputs multiple curve formulas

[0102] S6.11 Network forward inference:

[0103] Input {F BEV,t-l+1 ,…,F BEV,t},{VehicleState t-l+1 ,…,VehicleState t} to the function Φ: ({F BEV ,…},{VehicleState,…}) → {α1,...,α K}, and the network generates K sets of curve parameters α k at one time. If represented by a cubic polynomial, each set of parameters α k contains (a0, a1, a2, a3, b0, b1, b2, b3) to describe the curve C k (s).

[0104] S6.12 Interpolate discrete points: For each set of α k , interpolate to obtain the discrete points at the position of time t + i:

[0105]

[0106] where s i is the time parameter t + i×Δt. In this way, K sequences of spatial curves are obtained in the online stage, and each curve is composed of discrete points .

[0107] Step S6.2: Once a curve hits an obstacle, it is immediately removed.

[0108] S6.21 Collision detection: If the center of the obstacle is detected and its safety radius The safety radius r of the host vehicle ego , then for curve C k the discrete points perform distance calculation:

[0109]

[0110] If a violation of this formula at any time t + i indicates a collision, then immediately remove C k to prevent execution of this dangerous curve.

[0111] S6.2.2 Kinematic violation re - detection: Check whether the speed, acceleration, friction circle, and sideslip angle of the remaining curves exceed the limits; if there are violations, discard the curve.

[0112] Step S6.3, Final curve optimization.

[0113] S6.31 Define the comprehensive optimization index:

[0114] Among the remaining legal curves, define the measurement index according to the driving requirements:

[0115] D(C k ): The distance from the trajectory to the target point or the path length

[0116] Q(C k ): Curvature, steering effort

[0117] Construct the comprehensive index C(C k ) = λ dist D(C k ) + λ curv Q(C k ).

[0118] Among them, λ dist , λ curv are balance coefficients.

[0119] Select as the final curve output for the vehicle to execute; if there is only one legal curve left, directly use it.

[0120] Through this process, multiple curves are first filtered for collisions and kinematic detection, and then the optimal curve C * is selected. This solution can meet both safety and driving comfort in complex scenarios.

[0121] A vehicle motion planning system based on deep learning, including:

[0122] A data acquisition module, configured to obtain real-time vehicle sensing data of a vehicle during vehicle driving, where the real-time vehicle sensing data includes vehicle-mounted camera shooting data, radar measurement data, and GPS / IMU sensor data;

[0123] A trajectory generation and screening module, configured to extract bird's-eye view features of the real-time vehicle sensing data; input the bird's-eye view features into a trajectory generation model for curve fitting, and output multiple curve trajectories; wherein, the trajectory generation model is constructed based on a deep learning model; perform collision detection and dynamic violation detection on each of the curve trajectories to obtain a number of candidate trajectories;

[0124] A final trajectory optimization module, configured to select an optimal trajectory from each of the candidate trajectories as the execution trajectory of the vehicle according to a preset comprehensive optimization index.

[0125] An electronic device, including a memory and a processor, where the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute the described vehicle motion planning method based on deep learning.

[0126] As mentioned above, the above is only a preferred specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A vehicle motion planning method based on deep learning, characterized in that, include: Acquire real-time vehicle sensor data of the vehicle during driving, wherein the real-time vehicle sensor data includes vehicle camera shooting data, radar measurement data and GPS / IMU sensor data; extracting bird's-eye view features of the real-time vehicle sensor data; Inputting the bird's-eye view features into a trajectory generation model for curve fitting, and outputting a plurality of curve trajectories; wherein the trajectory generation model is constructed based on a deep learning model; Performing collision detection and dynamic violation detection on each of the curved trajectories to obtain a number of candidate trajectories; The optimal trajectory is selected from each of the candidate trajectories based on a preset comprehensive optimization index as the execution trajectory of the vehicle.

2. The vehicle motion planning method based on deep learning according to claim 1, wherein, The extracting of the bird's-eye view features of the real-time vehicle sensor data specifically includes: Aligning the real-time vehicle sensor data to a unified coordinate system, and removing noise and redundancy to obtain preprocessed sensor data; The preprocessed vehicle sensor data is projected into a two-dimensional grid coordinate system to form a bird's-eye view feature, and the vehicle status and trajectory true value are recorded.

3. A vehicle motion planning method based on deep learning according to claim 1, characterized in that, The training process of the trajectory generation model specifically includes: Acquire training data, wherein the training data includes vehicle sensor training data and corresponding real driving trajectory; The training data is input into the trajectory generation model for curve fitting, and multi-stage training is performed according to the target loss function to obtain a trained trajectory generation model.

4. A vehicle motion planning method based on deep learning according to claim 3, characterized in that, The design process of the objective loss function specifically includes: The objective loss function is designed based on dynamic constraints, wherein the dynamic constraints include speed constraints, acceleration constraints, friction circle constraints and sideslip angle constraints.

5. A vehicle motion planning method based on deep learning according to claim 1, characterized in that, The processing process of the trajectory generation model specifically includes: Based on the time series deep learning network, the historical features of the bird's-eye view features are extracted, and multiple curve trajectories are generated at one time at the decoding end.

6. The vehicle motion planning method based on deep learning according to claim 1, characterized in that, The performing collision detection and dynamic violation detection on each of the curve trajectories specifically includes: Interpolating each of the curve trajectories to obtain a discrete point sequence; Calculate the distance between the discrete points of each curve trajectory and the obstacle, and remove the corresponding curve if a collision is detected; The remaining curve trajectories are checked for speed, acceleration, friction circle, and sideslip angle violations and discarded if any violation occurs.

7. A vehicle motion planning method based on deep learning according to claim 1, characterized in that, The process of selecting the optimal trajectory specifically includes: A comprehensive optimization index is constructed based on the curvature, steering cost, distance from the trajectory to the target point or path length of each candidate trajectory; The optimal trajectory is selected as the vehicle's execution trajectory based on the comprehensive optimization index.

8. A vehicle motion planning system based on deep learning, characterized in that, include: A data acquisition module is used to obtain real-time vehicle sensor data of the vehicle during driving, wherein the real-time vehicle sensor data includes vehicle camera shooting data, radar measurement data and GPS / IMU sensor data; A trajectory generation and screening module is used to extract the bird's-eye view features of the real-time vehicle sensor data; input the bird's-eye view features into a trajectory generation model for curve fitting, and output a plurality of curve trajectories; wherein the trajectory generation model is constructed based on a deep learning model; perform collision detection and dynamic violation detection on each of the curve trajectories to obtain a plurality of candidate trajectories; The final trajectory selection module is used to select the best trajectory from the candidate trajectories as the execution trajectory of the vehicle according to a preset comprehensive selection index.

9. An electronic device, characterized in that, It includes a memory and a processor. The memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute a vehicle motion planning method based on deep learning according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, It stores a computer program, and when the computer program is executed by a processor, it realizes a vehicle motion planning method based on deep learning according to any one of claims 1-7.

Images