Vehicle trajectory planning method and device, vehicle, medium and program

Through offline reinforcement learning network processing vehicle perception data, predict future trajectories and optimize planning, the problem of high scenario dependence in the existing technology is solved, and more accurate and widely applicable trajectory planning is achieved, which improves user experience.

CN118928472AInactive Publication Date: 2024-11-12大卓智能科技有限公司
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202411202719.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-29
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing vehicle trajectory planning methods have high dependence on scenarios, resulting in high requirements for scenario classification, difficulty in accurately identifying different scenarios, and difficulty in strategy modification, and poor applicability.

Method used

The offline reinforcement learning network is adopted to generate a state sequence by obtaining the vehicle's perceived data, and input it to the offline reinforcement learning model to predict the vehicle trajectory of the future duration. The reward function and policy network are used to continuously optimize the trajectory planning to achieve data-driven method updates.

Benefits of technology

It improves the accuracy and applicability of trajectory planning, can perform smarter in complex traffic flows, improve user experience without the need for expert labeling data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118928472A_ABST
    Figure CN118928472A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of vehicles, in particular to a trajectory planning method and device of a vehicle, the vehicle, a medium and a program.The method comprises the steps that perception data of the current vehicle are obtained, and the perception data comprise image data, point cloud data, vehicle state data and vehicle positioning data; a state sequence is generated according to the perception data, and the state sequence comprises high-dimensional features of the vehicle surrounding environment information, a track point set and a reward function; and inputting the state sequence into an off-line reinforcement learning model, predicting a driving track of the vehicle in the future duration by the off-line reinforcement learning model, and controlling the current vehicle according to the driving track of the vehicle in the future duration. Therefore, the problems of high dependence of vehicle trajectory planning on scenes, relatively low recognition accuracy for different scenes, relatively poor applicability and the like in related technologies are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a vehicle trajectory planning method, device, vehicle, medium and program. Background Art

[0002] Trajectory planning is an important part of the field of autonomous driving, which involves the optimal path of the vehicle from the starting position to the target position. This process must not only consider physical constraints, but also efficiency, safety, and dynamic environmental changes.

[0003] In the related technology, in the trajectory planning of the rule scheme, the actual traffic conditions are divided into different scenes such as highways, ramps, and intersections. In different scenes, the perception information such as cameras, radars, and navigation maps and the vehicle information such as vehicle speed and yaw angle are combined to generate a drivable area, and then evaluation functions such as trajectory smoothness are designed. Then, numerical solution tools such as convex optimization are used to generate a curve that best fits the trajectory evaluation function within the drivable area. This curve is the solved trajectory. However, due to the high dependence on the scene, the derivative problems brought about are high requirements for scene classification, difficulty in accurately identifying different scenes, and the difficulty of strategy modification will also increase rapidly with the richness of the scene, and the overall applicability is poor. Summary of the invention

[0004] The present application provides a vehicle trajectory planning method, device, vehicle, medium and program to solve the problems in the related art that vehicle trajectory planning has high scene dependence, low recognition accuracy for different scenes, and poor applicability.

[0005] The first aspect of the present application provides a vehicle trajectory planning method, comprising the following steps: acquiring perception data of a current vehicle, wherein the perception data includes image data, point cloud data, vehicle status data, and vehicle positioning data; generating a state sequence based on the perception data, wherein the state sequence includes high-dimensional features of the vehicle's surrounding environment information, a trajectory point set, and a reward function; inputting the state sequence into an offline reinforcement learning model, the offline reinforcement learning model predicts the vehicle's driving trajectory for a future duration, and controlling the current vehicle based on the vehicle's trajectory for the future duration.

[0006] Optionally, generating a state sequence based on the perception data includes: generating high-dimensional features of the vehicle's surrounding environment information at the current moment and future moments based on the image data and point cloud data in the perception data; fusing the vehicle state data and vehicle positioning data in the perception data to predict the current vehicle's driving trajectory; constructing a reward function based on the vehicle state data in the perception data at each moment and preset rules, and generating a state sequence based on the high-dimensional features of the vehicle's surrounding environment information, the current vehicle's driving trajectory and the reward function.

[0007] Optionally, the generating high-dimensional features of the vehicle's surrounding environment information at the current moment and the future moment based on the image data and the point cloud data in the perception data includes: inputting the image data and the point cloud data in the perception data into a perception network model, and the perception network model outputs an environment map around the vehicle, wherein the perception network model includes a neural network model for scene understanding and a neural network model for tracking targets; inputting the image data and the point cloud data in the perception data into a prediction network model, and the prediction network model outputs high-dimensional features of the vehicle's surrounding environment information at the current moment and the future moment.

[0008] Optionally, the offline reinforcement learning model includes a first model and a second model, wherein the first model and the second model both include a fully connected layer, a multilayer perceptron composed of an activation function, and an output layer.

[0009] Optionally, the training process of the offline reinforcement learning model: converting each of the state sequences into a parameter value of a corresponding policy network in the offline reinforcement learning model; calculating a corresponding advantage function according to the parameter value; updating a corresponding loss function according to the parameter value and the advantage function; iteratively updating the corresponding policy network and value function according to the loss function; training the offline reinforcement learning model according to the policy network and value function until the policy network and value function converge to an optimal solution.

[0010] Optionally, controlling the current vehicle according to the trajectory of the vehicle of the future duration includes: identifying a vehicle driving action corresponding to the trajectory of the vehicle of the future duration; inputting the vehicle driving action into a dual-loop PID controller, the dual-loop PID controller outputting corresponding vehicle control parameters; and controlling the current vehicle according to the vehicle control parameters.

[0011] The second aspect of the present application provides a vehicle trajectory planning device, including: an acquisition module for acquiring the perception data of the current vehicle, wherein the perception data includes image data, point cloud data, vehicle status data and vehicle positioning data; a generation module for generating a state sequence based on the perception data, wherein the state sequence includes high-dimensional features of the vehicle's surrounding environment information, a trajectory point set and a reward function; a control module for inputting the state sequence into an offline reinforcement learning model, wherein the offline reinforcement learning model predicts the vehicle's driving trajectory for a future duration, and controls the current vehicle according to the vehicle's trajectory for a future duration. Optionally,

[0012] A third aspect of the present application provides a vehicle, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to perform the vehicle trajectory planning method as described in the above embodiment.

[0013] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to perform the vehicle trajectory planning method as described in the above embodiment.

[0014] The fifth aspect of the present application provides a computer program product, including a computer program or instructions, characterized in that when the computer program or instructions are executed, the vehicle trajectory planning method as described in the above embodiment is implemented.

[0015] Therefore, this application has at least the following beneficial effects:

[0016] The embodiment of the present application can generate a corresponding state sequence based on the perception data acquired in real time, input the state sequence into the offline reinforcement learning model, and the offline reinforcement learning model predicts the driving trajectory of the vehicle in the future, and controls the current vehicle according to the trajectory of the vehicle in the future. The offline reinforcement learning network is used to realize data-driven planning method updates, which has better coverage and growth. The corresponding state sequence can be generated according to the perception data to continuously improve the offline reinforcement learning model, so that the trajectory planning is more accurate, more applicable, and the user experience is improved.

[0017] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0019] Figure 1 A flowchart of a vehicle trajectory planning method provided according to an embodiment of the present application;

[0020] Figure 2 A schematic diagram of a method for processing perception data according to an embodiment of the present application;

[0021] Figure 3 A schematic diagram of the structure of an offline reinforcement learning model provided according to an embodiment of the present application;

[0022] Figure 4 A schematic diagram of a trajectory planning system based on offline reinforcement learning provided according to an embodiment of the present application;

[0023] Figure 5 A block diagram of a vehicle trajectory planning device provided according to an embodiment of the present application;

[0024] Figure 6 It is a schematic diagram of the structure of a vehicle provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0025] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0026] In the related art (1), the existing technical solutions focus on the trajectory planning of the rule-based solution, which divides the actual traffic conditions into different scenarios such as highways, ramps, and intersections, and performs corresponding logic design in different scenarios. The logic design specifically refers to generating a drivable area in different scenarios by combining sensory information such as cameras, radars, and navigation maps with vehicle information such as vehicle speed and yaw angle, and then designing evaluation functions such as trajectory smoothness, and then using numerical solution tools such as convex optimization to generate a curve that best meets the trajectory evaluation function within the drivable area. This curve is the solved trajectory.

[0027] The main disadvantage of rule-based trajectory planning is its high dependence on scenarios. The derivative problems brought about by high scenario dependence are high requirements for scene classification, difficulty in accurately identifying different scenarios, and the difficulty of strategy modification will also increase rapidly with the enrichment of scenarios.

[0028] In the related technology (2), data-driven planning technology has gradually emerged in recent years. Existing data-driven technology is mainly concentrated in the field of supervised learning. Supervised learning is a technology that calculates network loss and updates network gradients by comparing with manual calibration. In the field of planning, the main representative of supervised learning is imitation learning, which uses expert drivers to drive expert trajectories on autonomous driving vehicles and manually annotates the trajectories to generate labeled expert trajectories that can be used for network training. This trajectory is used as the true value and is passed into the training network of imitation learning to make the network converge in the direction of the true value, so that in similar road environments, autonomous driving vehicles can plan routes close to the expert trajectory, thereby achieving high-level autonomous driving.

[0029] The main disadvantages of technical solutions represented by imitation learning are that it is very difficult to obtain training data and that they are highly dependent on data scenarios, because the training data must be driven by professional drivers and require a large amount of annotation later. After the network is trained with a large amount of expert data, it can only replicate similar scenarios to the expert data, which is difficult to meet the complex driving conditions in actual driving.

[0030] In the related technology (3), the trajectory planning based on reinforcement learning is explored and learned through trial and error, and is mostly concentrated in the design of simulation systems. It cannot be directly explored and trained on the real vehicle. It needs to be trained in the simulation environment and then transferred to the real vehicle. However, because the simulation environment is ultimately difficult to accurately simulate the real environment, the output of the reinforcement learning network cannot be directly applied to the real vehicle.

[0031] As a research hotspot in the field of deep learning, deep reinforcement learning is making continuous progress in its basic theory and network construction technology. As a decision-making network, deep reinforcement learning can continuously improve the decision-making level of intelligent agents, greatly promoting the development of strong strategy fields such as autonomous driving and robots. In the field of autonomous driving, vehicle trajectory pre-planning is a key technology. According to the current driving scene and the state of the vehicle, the trajectory within a certain period of time in the future is pre-planned, and the actuator can control the vehicle movement safely and smoothly according to the trajectory.

[0032] In response to the problems existing in the prior art, this application designs a trajectory planning system and method based on offline reinforcement learning. The trajectory planning system is simple and easy to use and does not require expert data annotation. The method adopts an offline reinforcement learning network to realize data-driven planning method updates, which has better coverage and growth than existing traditional rule methods and supervised learning methods. The method learns people's processing under various traffic flows from the vehicle's perception data, and continuously improves the strategy through reward functions and offline reinforcement learning methods. The trajectory planning model obtained by this method is more in line with people's operating habits and will have more intelligent performance in complex traffic flows.

[0033] Therefore, this application mainly uses reinforcement learning networks to optimize long-sequence decision-making problems such as trajectory planning, and thus to a large extent optimizes the problem that trajectory planning has many limitations in applicable scenarios and long-tail scenarios are difficult to solve efficiently.

[0034] The following describes the vehicle trajectory planning method, device, vehicle, storage medium and program of the embodiments of the present application with reference to the accompanying drawings.

[0035] Specifically, Figure 1 A schematic flow chart of a vehicle trajectory planning method provided in an embodiment of the present application.

[0036] like Figure 1 As shown, the vehicle trajectory planning method includes the following steps:

[0037] In step S101, the perception data of the current vehicle is acquired, wherein the perception data includes image data, point cloud data, vehicle status data and vehicle positioning data.

[0038] It can be understood that the embodiments of the present application can obtain the perception data of the current vehicle through perception devices, where the perception devices include but are not limited to cameras, lidars, IMUs (Inertial Measurement Units) and GPS (Global Positioning System).

[0039] It should be noted that the camera and lidar map the surrounding environment, and their raw data are images and point clouds, which represent the relative relationship between the vehicle and the driving environment; IMU mainly maps acceleration information, and the time integration of acceleration can obtain accurate information such as the vehicle's speed, movement distance and direction deflection in a short period of time; GPS mainly maps the longitude and latitude information of the vehicle under the earth, and after converting to the geodetic coordinate system, it can obtain the driving trajectory of the vehicle. The perception data is directly uploaded to the original database via the network interface device.

[0040] In step S102, a state sequence is generated according to the perception data, wherein the state sequence includes high-dimensional features of the vehicle's surrounding environment information, a trajectory point set, and a reward function.

[0041] It can be understood that the embodiment of the present application generates a state sequence based on the perception data to facilitate the subsequent prediction of the vehicle's driving trajectory in the future.

[0042] In an embodiment of the present application, a state sequence is generated based on the perception data, including: generating high-dimensional features of the vehicle's surrounding environment information at the current moment and future moments based on the image data and point cloud data in the perception data; fusing the vehicle state data and the vehicle positioning data in the perception data to predict the current vehicle's driving trajectory; constructing a reward function based on the vehicle's state data in the perception data at each moment and preset rules, and generating a state sequence based on the high-dimensional features of the vehicle's surrounding environment information, the current vehicle's driving trajectory and the reward function.

[0043] It is understandable that the embodiments of the present application can process the perception data into the action values, state quantities and reward values ​​required for offline reinforcement, so as to facilitate the subsequent prediction of the vehicle's driving trajectory in the future.

[0044] It should be noted that after the perception data is initially screened, the state quantity is generated by the pre-trained perception network and prediction network; the short-term accuracy of GPS is relatively poor, and the long-term accuracy is relatively high. On the contrary, the IMU is the opposite. Therefore, the Kalman filter method is used to fuse the data to obtain the self-vehicle coordinates with higher accuracy; the reward function determines the optimization direction of offline reinforcement learning, and the comfort and termination conditions are designed through experience to guide the update of the policy function of the offline reinforcement learning network.

[0045] Specifically, if Figure 2As shown in the figure, preprocessing mainly includes three aspects: data screening, action value processing and state quantity processing:

[0046] 1. Data screening includes but is not limited to outlier removal, point cloud downsampling, sensor external parameter compensation, etc. After data screening, the camera data, lidar data, IMU data, and GPS data at time t are recorded as c respectively. t ,l t ,i t ,g t .

[0047] 2. The process of state quantity processing is to pass the pre-processed camera data and lidar data into the perception network and prediction network. The perception network is Map Former and Track Former, and the prediction network is Motion Former. The high-dimensional features of the surrounding environment information are obtained through the above networks, which are recorded as s t .

[0048] 3. The purpose of action value processing is to restore the driving trajectory of the vehicle to facilitate the gradient update of the planning network. The location point information of the trajectory can be recorded by methods including but not limited to wheel odometers, visual odometers, etc. However, the odometer method is prone to accumulated errors, and the long-term recording error is large. In order to improve the accuracy of the trajectory and reduce the difficulty of planning network training, global positioning information including but not limited to GPS can be integrated. Taking the classic Kalman filter as an example, after fusing the IMU and GPS signals, the position point at time t is recorded as pt. The trajectory is a collection of position points at a certain time step. The time step of the trajectory is K, then the trajectory point set is a t =[p t , p t+1 ,…,p t+k-1 ].

[0049] 4. After the state quantity and action value are processed, design and calculate the reward r at each time t t , the design of the reward function mainly considers comfort and termination conditions. Specifically, it can be designed as follows Among them, the acceleration k jerk and acceleration k acc The reward coefficient can be calibrated as needed.

[0050] (1) Comfort

[0051] Jerk: r0 = k jerk *Δa i , i=1,2,…

[0052] Acc:

[0053] (2) Termination conditions

[0054] 1) Collision r2 = -100, ifincollision

[0055] 2) Arrive at the target point r3 = 100, if reached

[0056] 3) Record the high-dimensional features, trajectory points and reward function of the surrounding environment at time t respectively, store them in preprocessed data, and record them as a state sequence (s t , a t , r t ,s t+1 ).

[0057] In an embodiment of the present application, high-dimensional features of the vehicle's surrounding environment information at the current moment and in the future are generated based on the image data and point cloud data in the perception data, including: inputting the image data and point cloud data in the perception data into a perception network model, and the perception network model outputs an environment map around the vehicle, wherein the perception network model includes a neural network model for scene understanding and a neural network model for tracking targets; inputting the image data and point cloud data in the perception data into a prediction network model, and the prediction network model outputs high-dimensional features of the vehicle's surrounding environment information at the current moment and in the future.

[0058] Among them, high-dimensional features are meaningful information extracted from the original data, which may include the driving trajectory of other vehicles, vehicle surrounding environment information, lane line information, traffic light information, etc., without specific limitation.

[0059] It is understandable that the embodiments of the present application can convert the perception data into high-dimensional features of the vehicle's surrounding environment information at the current moment and in the future, so as to facilitate the subsequent prediction of the vehicle's driving trajectory in the future.

[0060] It should be noted that the goal of the perception network is to correctly understand the sensor data and convert it into useful feature information to facilitate the use of the prediction network and planning network. It specifically includes the neural network model Track Former for tracking targets and the neural network model Map Former for scene understanding; the prediction network is Motion Former.

[0061] Specifically, Map Former is a neural network module used to build an environmental map. It is used to receive preprocessed lidar data and / or camera data, extract environmental features from them, and generate an environmental map containing static and dynamic objects. This process may involve semantic segmentation, target detection and other technologies.

[0062] Track Former is the neural network module responsible for tracking functions; it mainly uses the object information extracted from Map Former to perform target tracking, that is, to identify and predict the location and movement trends of objects; this helps to identify the historical trajectory of a specific object and predict its future behavior.

[0063] In step S103, the state sequence is input into an offline reinforcement learning model, and the offline reinforcement learning model predicts the driving trajectory of the vehicle in the future, and controls the current vehicle according to the trajectory of the vehicle in the future.

[0064] It can be understood that the embodiment of the present application can input the state sequence into the offline reinforcement learning model, the offline reinforcement learning model predicts the driving trajectory of the vehicle in the future, controls the current vehicle according to the trajectory of the vehicle in the future, and adopts the offline reinforcement learning network to realize the update of data-driven planning method, which has better coverage and growth. It can generate the corresponding state sequence according to the perception data and continuously improve the offline reinforcement learning model, so as to make the trajectory planning more accurate and more applicable, and enhance the user experience.

[0065] In the embodiments of the present application, Figure 3 As shown, the offline reinforcement learning model includes a first model and a second model, wherein the first model and the second model both include a fully connected layer, a multilayer perceptron composed of an activation function, and an output layer.

[0066] Among them, the first model is the Actor network, and the second model is the Critic network.

[0067] Specifically, the above offline reinforcement learning model is elaborated in detail as follows:

[0068] (1) Actor Network:

[0069] Input: The input layer receives state s or state-action pairs (s, a); Fully connected layer: Converts the input into a higher dimensional space in order to capture more complex relationships; Activation function: ReLU, tanh or other nonlinear functions are usually used to introduce nonlinear characteristics, enabling the network to learn more complex patterns; Output layer: The output layer produces an action a or the probability distribution of the action p(a|s).

[0070] (2)Critic Network:

[0071] Input: The input layer receives the state s and action a; Fully connected layer: Similar to the Actor network, it converts the input into a higher dimensional space; Activation function: Non-linear functions are also used to introduce non-linear characteristics; Output layer: The output layer produces a scalar value Q(s,a), which represents the value of the state-action pair.

[0072] (3) Training process

[0073] Actor Network: Updates parameters using the policy gradient method to maximize the expected return.

[0074] Critic network: Updates parameters using Q-learning or similar methods with the goal of minimizing the gap between predicted value and actual reward.

[0075] In an embodiment of the present application, the training process of the offline reinforcement learning model is as follows: converting each state sequence into a parameter value of a corresponding policy network in the offline reinforcement learning model; calculating a corresponding advantage function according to the parameter value; updating a corresponding loss function according to the parameter value and the advantage function; iteratively updating the corresponding policy network and value function according to the loss function; training the offline reinforcement learning model according to the policy network and value function until the policy network and value function converge to the optimal solution.

[0076] Specifically, the trajectory planning network is an offline reinforcement learning network. By injecting state sequences one by one, the gradient update of the policy network and the value function is realized. The specific process is as follows:

[0077] 1. Update process

[0078] Randomly initialize the initial value π(a|s;θ0)

[0079] 2. Calculate the advantage function

[0080] A π (s t , a t )=E A [E[U t |S t =s t , A t =a t ]]-E[U t |S t =s t , A t =a t ],

[0081]

[0082] 3. Update the loss function

[0083]

[0084] 4. Update strategy

[0085]

[0086] Update value function

[0087] in

[0088] In an embodiment of the present application, controlling the current vehicle according to the trajectory of the vehicle at a future time length includes: identifying the vehicle driving action corresponding to the trajectory of the vehicle at a future time length; inputting the vehicle driving action into a dual-loop PID controller, and the dual-loop PID controller outputs corresponding vehicle control parameters; and controlling the current vehicle according to the vehicle control parameters.

[0089] It can be understood that the embodiments of the present application can identify the vehicle driving action corresponding to the trajectory of the vehicle in the future; input the vehicle driving action into the dual-loop PID controller, and the dual-loop PID controller outputs the corresponding vehicle control parameters; control the current vehicle according to the vehicle control parameters to achieve autonomous driving verification.

[0090] It should be noted that the control program generates the control amount of the throttle and steering angle of the vehicle's wire-controlled system based on the action value through programs including but not limited to dual-loop PID, thereby realizing automatic driving verification.

[0091] Specifically, the control program is deployed in the verification vehicle at the same time, and control methods including but not limited to dual-loop PID can be used to convert the trajectory into control quantities such as throttle and steering wheel angle, thereby controlling the operation of the actual vehicle.

[0092] According to the vehicle trajectory planning method proposed in the embodiment of the present application, a corresponding state sequence is generated according to the perception data acquired in real time, and the state sequence is input into the offline reinforcement learning model. The offline reinforcement learning model predicts the driving trajectory of the vehicle in the future, and the current vehicle is controlled according to the trajectory of the vehicle in the future. An offline reinforcement learning network is used to realize data-driven planning method updates, which has better coverage and growth. The corresponding state sequence can be generated according to the perception data to continuously improve the offline reinforcement learning model, so that the trajectory planning is more accurate, more applicable, and the user experience is improved.

[0093] The following will be combined Figure 3-4 The vehicle trajectory planning method of the present application is described in detail. The present method is implemented by a trajectory planning system based on offline reinforcement learning. Therefore, the four levels of the trajectory planning system based on offline reinforcement learning, namely, the infrastructure layer, the execution layer, the network layer and the functional layer, are described first. Specifically:

[0094] (1) The infrastructure layer includes two parts: mobile vehicles and data centers. The function of mobile vehicles is to collect data, upload data and practice training results. It needs to include sensing devices such as cameras that perceive environmental information, control devices such as wire control systems that control vehicle movement, and network interface devices that upload / download data. The data center is mainly used to store data. The data mainly includes three types: raw data directly collected, data preprocessing such as filtering and screening, and network layer parameters that are constantly updated.

[0095] (2) The execution layer includes the acquisition unit and the execution unit. The acquisition unit refers to the vehicle used for learning that runs in the environment and uploads actual driving data for the training system to continuously learn. The execution unit refers to the self-driving vehicle used to verify the training results. The trajectory of the self-driving vehicle is input by the network, and the control system relies on this input to complete the control output.

[0096] (3) The network layer includes a perception network, a prediction network, and a planning network. The goal of the perception network is to correctly understand the sensor data and convert it into useful feature information for the use of the prediction network and the planning network. Specifically, it includes Track Former for tracking targets and Map Former for scene understanding. In this application, the perception network can use pre-trained transformers and their deformations and other pre-trained network models. This training model does not update the perception network. The goal of the prediction network is to correctly predict the driving trajectory of surrounding vehicles in a certain time step in the future, so that the planning network can output an effective trajectory that will not conflict with other vehicles. The specific form of the network is Motion Former. In this application, the prediction network can use pre-trained MTR and its deformations and other pre-trained network models. This application does not update the prediction network. The goal of the planning network is to plan a guiding trajectory that can meet the vehicle's kinematic and dynamic constraints and can be correctly pulled to the target position. In this application, the planning network is trained using offline reinforcement learning. Based on the original action distribution, the offline reinforcement learning network continuously learns from the collected offline data through the design of the reward function and continuously optimizes the action distribution.

[0097] (4) The functional layer includes four aspects: data collection, data processing, network training, and network output. Data collection refers to the ability to collect and formulate data from mobile vehicles. The collected data is used to plan network training. Data processing refers to the processing of raw data, including but not limited to invalid data screening, sensor signal compensation, action value sorting, and termination condition determination. Network training refers to the update of the imitation learning network and offline reinforcement learning network in the planning network. After the real vehicle data is input into the network output value, it is output using the trained network parameters and network layer structure. The output header information is the trajectory of a certain time step in the future, which is specifically manifested as a set of arrays containing time and position point information.

[0098] In addition, in a typical configuration of the present application, the service network includes equipment, computing equipment and storage equipment, data input / output interfaces, network interfaces and mobile vehicles. The service network equipment is used to maintain mobile vehicle data and ensure the security of data flow throughout the process, and may include server terminals and their corresponding supporting equipment. The computing equipment is used for neural network gradient calculation and network update, and may include processors such as CPU and GPU. The storage device is used to store data sent back by the mobile vehicle and record the updated data after the neural network calculation, and may include non-permanent memory in computer-readable media, random access memory and non-volatile memory and other forms of memory. Mobile vehicles are used for the collection of self-driving data and the practice of network output, and may include sensors such as cameras, radars, IMUs, GPS, control devices such as engines, steering wheels, and wire control systems, and network interface devices such as data collection, clipping, and uploading.

[0099] The specific steps of the vehicle trajectory planning method are as follows:

[0100] (1) Data collection phase: Deploy corresponding sensing devices such as cameras, radars, IMUs, and GPS, control devices such as wire control systems, and network interface devices such as data transmission systems on mobile vehicles. Multiple mobile vehicles transmit data to the data center in real time. The camera data, lidar data, IMU data, and GPS data at time t are c respectively. t ,l t ,i t ,g t .

[0101] The effective data of self-driving vehicles is mainly measured and mapped by the sensing devices at the infrastructure layer, and the surveying data is transmitted back to the data center via the network interface device. In this application, the sensing devices include but are not limited to cameras, lidars, IMUs and GPS. The cameras and lidars survey the surrounding environment, and their original data are images and point clouds, respectively, which characterize the relative relationship between the self-driving vehicle and the driving environment. The IMU mainly measures acceleration information, and the time integration of the acceleration can obtain accurate information such as the speed, movement distance and direction deflection of the vehicle in a short time. GPS mainly measures the longitude and latitude information of the self-driving vehicle under the earth. After converting to the geodetic coordinate system, the driving trajectory of the self-driving vehicle can be obtained. The perception data and trajectory data of the mobile vehicle will be transmitted to the data center at a high frequency.

[0102] (2) Data processing stage: In the data center, the raw data undergoes a preprocessing process in sequence to be converted into usable data for neural network training. The preprocessing methods include data screening, action value processing, state quantity processing, and reward function design.

[0103] The purpose of data preprocessing is to process the perception data into the action values, state quantities and reward values ​​required for offline reinforcement. After the perception data is initially screened, the state quantities are generated by the pre-trained perception network and prediction network; GPS has relatively poor short-term accuracy and relatively high long-term accuracy, while IMU is the opposite. For this reason, the Kalman filter method is used to fuse data to obtain more accurate self-vehicle coordinates; the reward function determines the optimization direction of offline reinforcement learning, and the comfort and termination conditions are designed through experience to guide the update of the policy function of the offline reinforcement learning network.

[0104] 1) Data screening includes abnormal data processing and downsampling. Abnormal data removal is to filter out the values ​​that exceed the measurement range in the laser and camera point cloud data and delete the Nan values. Downsampling is uniform downsampling. First, the point cloud is divided into 1 cm voxels, and the nearest point is selected from the voxel to represent the entire voxel. The other point clouds are deleted to achieve point cloud downsampling.

[0105] 2) Action value processing is to use the Kalman filter algorithm to pass the variance and measurement value of the IMU and GPS sensors into the filter function respectively, and the fused value is used as the coordinate of the vehicle at the current moment. The coordinates and corresponding time of a certain time step are selected as the trajectory point combination, that is, the action value.

[0106] 3) State quantity processing is to pass the perception data to the perception network and prediction network to obtain a high-dimensional feature representation of the environmental information, which is used as the state value s for training. The perception network is divided into pre-trained Track Former and MapFormer. BEVFormer can be used to initially extract it as BEV features, and then it is passed to the MOTRv2 target tracking network and the Mask2Former static map segmentation network respectively. The obtained features are passed to the Motion Transformer prediction network to obtain the driving trajectory of the other vehicle. The state quantity s is the driving trajectory of the other vehicle and the static map features.

[0107] 4) The reward function is mainly based on comfort and termination conditions:

[0108] Acceleration and its change as the basis for comfort evaluation r0 = 0.05 * Δa i , i = 1, 2, ..., r1 = 0.05*a, -3 <= a <= 4. The termination conditions are r2 = -100, ifincollision and r3 = 100, ifreached.

[0109] (3) Network training phase:

[0110] Reinforcement learning is an online training network that requires constant interaction with the environment. This interaction mode is not suitable for data-driven structures. To this end, offline reinforcement learning technology is introduced to learn the distribution of action values ​​under different state quantities offline, and continuously optimize through the reward function, so that better action values ​​can be learned under the premise of only non-expert data. The action value is the optimization target and is controlled by the policy function. In this patent application, the specific form of the action value is the trajectory point of a certain time step.

[0111] The main body of neural network training is the planning network, which is trained through offline reinforcement learning. Offline reinforcement learning is an offline network learning architecture, which mainly evaluates the incoming sample data through the design of the reward function, so that the gradient update direction is in the direction of higher reward function, so as to achieve the action distribution of the strategy function more in line with the design expectations of the reward function. In this application, the rewards for comfort and termination conditions are designed based on engineering experience, so that the action value is more in line with the public's travel expectations. Offline reinforcement learning is usually divided into Actor network and Critic network, both of which are MLP networks, specifically:

[0112] 1. Update process

[0113] Randomly initialize the initial value π(a|s;θ0)

[0114] 2. Calculate the advantage function

[0115] A π (s t , a t )=E A [E[U t |S t =s t , A t =a t ]]-E[U t |S t =s t , A t =a t ],

[0116]

[0117] 3. Update the loss function

[0118]

[0119] 4. Update strategy

[0120]

[0121] 5. Update the value function

[0122] in

[0123] Specifically: The network size of Actor is obs_dim+hidden_sizes+act_dim, the network size of Critic is obs_dim+hidden_sizes+1, and the activation function is Tanh function.

[0124] Among them, obs_dim is the dimension of the state quantity, specifically 256; hidden_size is the dimension of the hidden layer, specifically 128, act_dim is the size of the state quantity, specifically 512, and the activation function is the Tanh function.

[0125] (4) Network output stage: After the real vehicle receives the sensor device data and performs the same data processing operation, the same perception network, prediction network, and offline reinforcement learning policy function are transplanted to the verification vehicle. After processing, the real state quantity at time t is obtained, which is recorded as The strategy function generates action values ​​based on the state quantity. By transferring the action value to the control device, the vehicle can be controlled to drive automatically and the validity of the network output can be verified.

[0126] In summary, the trajectory planning system of the present application is simple and easy to use, and does not require expert labeled data; the method adopts an offline reinforcement learning network to realize the update of data-driven planning methods, which has better coverage and growth than the existing traditional rule methods and supervised learning methods; the method learns the processing of people under various traffic flows from the perception data of the vehicle, and continuously improves the strategy through the reward function and offline reinforcement learning method. The trajectory planning model obtained by this method is more in line with people's operating habits and will have more intelligent performance in complex traffic flows.

[0127] Next, a vehicle trajectory planning device according to an embodiment of the present application is described with reference to the accompanying drawings.

[0128] Figure 5 It is a block diagram of a trajectory planning device for a vehicle according to an embodiment of the present application.

[0129] like Figure 5 As shown, the vehicle trajectory planning device 10 includes: an acquisition module 100 , a generation module 200 and a control module 300 .

[0130] Among them, the acquisition module 100 is used to acquire the perception data of the current vehicle, wherein the perception data includes image data, point cloud data, vehicle status data and vehicle positioning data; the generation module 200 is used to generate a state sequence based on the perception data, wherein the state sequence includes high-dimensional features of the vehicle's surrounding environment information, trajectory point sets and reward functions; the control module 300 is used to input the state sequence into an offline reinforcement learning model, and the offline reinforcement learning model predicts the vehicle's driving trajectory in the future and controls the current vehicle according to the vehicle's trajectory in the future.

[0131] It should be noted that the above explanation of the vehicle trajectory planning method embodiment is also applicable to the vehicle trajectory planning device of this embodiment, and will not be repeated here.

[0132] According to the vehicle trajectory planning device proposed in the embodiment of the present application, a corresponding state sequence is generated according to the perception data acquired in real time, and the state sequence is input into the offline reinforcement learning model. The offline reinforcement learning model predicts the driving trajectory of the vehicle in the future, and the current vehicle is controlled according to the trajectory of the vehicle in the future. An offline reinforcement learning network is used to realize data-driven planning method updates, which has better coverage and growth. The corresponding state sequence can be generated according to the perception data to continuously improve the offline reinforcement learning model, so that the trajectory planning is more accurate, more applicable, and the user experience is improved.

[0133] Figure 6 A schematic diagram of the structure of a vehicle provided in an embodiment of the present application. The vehicle may include:

[0134] A memory 601 , a processor 602 , and a computer program stored in the memory 601 and executable on the processor 602 .

[0135] When the processor 602 executes the program, the vehicle trajectory planning method provided in the above embodiment is implemented.

[0136] Furthermore, the vehicle also includes:

[0137] The communication interface 603 is used for communication between the memory 601 and the processor 602 .

[0138] The memory 601 is used to store computer programs that can be executed on the processor 602 .

[0139] The memory 601 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0140] If the memory 601, the processor 602 and the communication interface 603 are implemented independently, the communication interface 603, the memory 601 and the processor 602 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0141] Optionally, in a specific implementation, if the memory 601, the processor 602 and the communication interface 603 are integrated on a chip, the memory 601, the processor 602 and the communication interface 603 can communicate with each other through an internal interface.

[0142] The processor 602 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0143] An embodiment of the present application also provides a computer-readable storage medium having a computer program or instructions stored thereon. When the computer program or instructions are executed by a processor, the vehicle trajectory planning method as described above is implemented.

[0144] An embodiment of the present application also provides a computer program product, including a computer program or instructions, characterized in that when the computer program or instructions are executed, the vehicle trajectory planning method as described above is implemented.

[0145] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0146] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0147] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0148] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one or a combination of multiple of the following technologies known in the art: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0149] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

Claims

1. A vehicle trajectory planning method, characterized in that: The following steps are involved: Acquire perception data of the current vehicle, wherein the perception data includes image data, point cloud data, vehicle status data, and vehicle positioning data; Generate a state sequence according to the perception data, wherein the state sequence includes high-dimensional features of vehicle surrounding environment information, a trajectory point set, and a reward function; The state sequence is input into an offline reinforcement learning model, the offline reinforcement learning model predicts a driving trajectory of a vehicle of a future duration, and the current vehicle is controlled according to the trajectory of the vehicle of the future duration.

2. The vehicle trajectory planning method according to claim 1, characterized in that: The generating a state sequence according to the perception data comprises: Generate high-dimensional features of vehicle surrounding environment information at current and future times based on the image data and point cloud data in the perception data; Fusion of vehicle status data and vehicle positioning data in the perception data to predict the current vehicle's driving trajectory; A reward function is constructed according to the state data of the vehicle in the perception data at each moment and preset rules, and a state sequence is generated according to the high-dimensional features of the vehicle's surrounding environment information, the current vehicle's driving trajectory and the reward function.

3. The vehicle trajectory planning method according to claim 2, characterized in that: The generating high-dimensional features of the vehicle surrounding environment information at the current moment and the future moment according to the image data and the point cloud data in the perception data includes: Inputting the image data and the point cloud data in the perception data into a perception network model, the perception network model outputting an environment map around the vehicle, wherein the perception network model includes a neural network model for scene understanding and a neural network model for tracking targets; The image data and point cloud data in the perception data are input into a prediction network model, and the prediction network model outputs high-dimensional features of the vehicle's surrounding environment information at the current moment and the future moment.

4. The vehicle trajectory planning method according to claim 1, characterized in that: The offline reinforcement learning model includes a first model and a second model, wherein the first model and the second model both include a fully connected layer, a multilayer perceptron composed of activation functions, and an output layer.

5. The vehicle trajectory planning method according to claim 4, characterized in that: The training process of the offline reinforcement learning model: Convert each of the state sequences into parameter values ​​of a corresponding policy network in an offline reinforcement learning model; Calculate the corresponding advantage function according to the parameter value; Update the corresponding loss function according to the parameter value and the advantage function; Iteratively update the corresponding policy network and value function according to the loss function; The offline reinforcement learning model is trained according to the policy network and the value function until the policy network and the value function converge to an optimal solution.

6. The vehicle trajectory planning method according to claim 1, characterized in that: The controlling the current vehicle according to the trajectory of the vehicle at the future duration includes: Identify the vehicle's driving action corresponding to the vehicle's trajectory at a future time; The vehicle driving action is input into a double-loop PID controller, and the double-loop PID controller outputs corresponding vehicle control parameters; The current vehicle is controlled according to the vehicle control parameters.

7. A vehicle trajectory planning device, characterized in that: include: An acquisition module is used to acquire the perception data of the current vehicle, wherein the perception data includes image data, point cloud data, vehicle status data and vehicle positioning data; A generation module, configured to generate a state sequence according to the perception data, wherein the state sequence includes high-dimensional features of the vehicle's surrounding environment information, a trajectory point set, and a reward function; A control module is used to input the state sequence into an offline reinforcement learning model, the offline reinforcement learning model predicts the driving trajectory of the vehicle in the future, and controls the current vehicle according to the trajectory of the vehicle in the future.

8. A vehicle, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the vehicle trajectory planning method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, it is used to implement the vehicle trajectory planning method as described in any one of claims 1-6.

10. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed, the vehicle trajectory planning method as described in any one of claims 1-6 is implemented.

Citation Information

Patent Citations

  • Offline agent using reinforcement learning to speedup trajectory planning for autonomous vehicles

    CN112034834A

  • Vehicle driving track planning method and device, vehicle and storage medium

    CN117601904A

  • Point cloud image fusion method and system for vehicle navigation

    CN117911829A

  • Model-based offline-to-online reinforcement learning method in automatic driving

    CN118313484A

  • Online agent using reinforcement learning to plan an open space trajectory for autonomous vehicles

    US20200363813A1