Intelligent vehicle path planning method, device and equipment and storage medium
By dynamically adjusting the sampling point interval using a pre-trained reinforcement learning model in the intelligent driving system, the problem that the path planning method in the prior art is unable to adapt to the complex traffic environment, and the quality and efficiency of path planning are improved.
Patent Information
- Application Number
- CN202410936733.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-08
AI Technical Summary
The existing intelligent driving path planning method cannot dynamically adjust the sampling point interval according to real-time environmental information, resulting in insufficient sampling points in areas with dense obstacles and too many sampling points in areas with sparse obstacles, affecting the quality and efficiency of path planning, making it difficult to adapt to complexity. Changeable traffic environment.
The pre-trained reinforcement learning model is used to determine the optimal scaling factor based on the obstacle state and motion state of the target vehicle's surrounding environment, and dynamically adjust the sampling point interval. By increasing the sampling point density in the obstacle dense area, the sampling in the sparse obstacle areas is reduced. point density, thereby optimizing path planning.
It realizes more accurate and detailed path planning in areas with dense obstacles, simplifies computing resource consumption in areas with sparse obstacles, improves the quality and efficiency of path planning, and adapts to complex traffic environments.
Smart Images

Figure CN118913299A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent driving technology, and in particular to an intelligent vehicle path planning method, device, equipment and storage medium. Background Art
[0002] In intelligent driving systems, vehicle path planning is a key link to ensure safe and efficient driving of vehicles. A common method is to collect multiple sampling points in a certain range of search space around the vehicle, and connect the sampling points to form the driving path of the vehicle in the future.
[0003] However, existing driving path planning methods often use fixed sampling point intervals and cannot be dynamically adjusted according to real-time environmental information, resulting in insufficient sampling points in areas with dense obstacles and too many sampling points in areas with sparse obstacles, which affects the quality and efficiency of path planning and makes it difficult to adapt to complex and changing traffic environments. Summary of the invention
[0004] The present application provides an intelligent vehicle path planning method, device, equipment and storage medium to solve the problem that existing driving path planning methods often use fixed sampling point intervals and cannot be dynamically adjusted according to real-time environmental information, resulting in insufficient sampling points in areas with dense obstacles and too many sampling points in areas with sparse obstacles, affecting the quality and efficiency of path planning and making it difficult to adapt to complex and changing traffic environments.
[0005] In a first aspect, the present application provides a method for intelligent vehicle path planning, the method comprising:
[0006] Determine, according to state information of obstacles in the surrounding environment of the target vehicle at a first moment and motion state information of the target vehicle, a first state of the target vehicle at the first moment, wherein the first state includes a relative distance and relative speed between the target vehicle and obstacles in the surrounding environment of the target vehicle at the first moment, a speed of the target vehicle, and a driving direction of the target vehicle;
[0007] Determining, by a pre-trained reinforcement learning model, an optimal scaling factor corresponding to the first state according to the first state;
[0008] Calculating the sampling point interval according to the optimal scaling factor;
[0009] In the environment model of the target vehicle's surrounding environment, a plurality of sampling points are collected at the sampling point interval, and spatial coordinate data of the plurality of sampling points are acquired;
[0010] Determining a target driving path equation of the target vehicle according to the spatial coordinate data of the plurality of sampling points;
[0011] The target vehicle is controlled to travel along the target driving path represented by the target driving path equation.
[0012] In a possible design, determining, by the pre-trained reinforcement learning model according to the first state, an optimal scaling factor corresponding to the first state includes:
[0013] The pre-trained reinforcement learning model determines, according to the first state, a first state-scaling factor pair having the highest corresponding value among first state-scaling factor pairs consisting of the first state and the scaling factor corresponding to the first state as a first target state-scaling factor pair;
[0014] The scaling factor in the first target state-scaling factor pair is determined as the optimal scaling factor.
[0015] In a possible design, determining the target driving path equation of the target vehicle according to the spatial coordinate data of the plurality of sampling points includes:
[0016] Generating a plurality of candidate driving path equations by curve fitting according to the spatial coordinate data of the plurality of sampling points;
[0017] According to the target cost function, the target driving path is determined among the multiple candidate driving paths so that the target cost function value is the lowest when the target vehicle travels along the target driving path. The target cost function includes a penalty factor for the risk of collision between the target vehicle and the obstacle, a penalty factor for the target vehicle's driving time, a penalty factor for the target vehicle's acceleration rate of change, and a penalty factor for the target driving path curvature.
[0018] In a possible design, after controlling the target vehicle to travel along the target driving path represented by the target driving path equation, the method further includes:
[0019] Determine a second state of the target vehicle at the second moment according to state information of obstacles in the surrounding environment of the target vehicle and motion state information of the target vehicle, wherein the second state includes a relative distance and relative speed between the target vehicle and obstacles in the surrounding environment of the target vehicle at the second moment, a speed of the target vehicle, and a driving direction of the target vehicle;
[0020] According to the second state, the value corresponding to the first target state-scaling factor pair is updated.
[0021] In a possible design, updating the value corresponding to the first target state-scaling factor pair according to the second state includes:
[0022] Determining a reward value of the optimal scaling factor according to a reward function;
[0023] The pre-trained reinforcement learning model determines, according to the second state, a second state-scaling factor pair having the highest corresponding value among second state-scaling factor pairs consisting of the second state and the scaling factor corresponding to the second state as a second target state-scaling factor pair;
[0024] According to the reward value and the value corresponding to the second target state-scaling factor pair, the value corresponding to the first target state-scaling factor pair is updated by the Bellman equation, and the Bellman equation is:
[0025] Q(s,a)←Q(s,a)+α[r+γMax a‘ Q(s',a')-Q(s,a)]
[0026] Wherein, Q(s, a) is the value corresponding to the first target state-scaling factor pair, s is the first state, a is the optimal scaling factor, α is the learning rate of the pre-trained reinforcement learning model, r is the reward value, γ is the discount rate of the pre-trained reinforcement learning model, Max a‘ Q(s', a') is the value corresponding to the second target state-scaling factor pair, s' is the second state, and a' is the scaling factor in the second state-scaling factor.
[0027] In one possible design, determining the reward value of the optimal scaling factor according to the reward function includes:
[0028] Determining, according to the first state, a scaling factor threshold corresponding to the first state;
[0029] If the optimal scaling factor is less than or equal to the scaling factor threshold, the reward function uses a target positive number as the reward value; if the optimal scaling factor is greater than the scaling factor threshold, the reward function uses a target negative number as the reward value.
[0030] In a possible design, before determining the first state of the target vehicle at the first moment based on the state information of obstacles in the surrounding environment of the target vehicle at the first moment and the motion state information of the target vehicle, the method further includes:
[0031] According to the prediction information of the motion state of the dynamic obstacle in the surrounding environment of the target vehicle by the filter at the previous moment and the detection information of the motion state of the dynamic obstacle at the first moment, the motion state information of the dynamic obstacle at the first moment is calculated by the filter, and the motion state information of the dynamic obstacle at the first moment is used as the state information of the obstacle in the surrounding environment of the target vehicle at the first moment.
[0032] In a second aspect, the present application provides an intelligent vehicle path planning device, the device comprising:
[0033] A state determination module, for determining a first state of the target vehicle at the first moment according to state information of obstacles in the surrounding environment of the target vehicle at the first moment and motion state information of the target vehicle, wherein the first state includes a relative distance and relative speed between the target vehicle and obstacles in the surrounding environment of the target vehicle at the first moment, a speed of the target vehicle, and a driving direction of the target vehicle;
[0034] Sampling point interval calculation module, used for:
[0035] Determining, by a pre-trained reinforcement learning model, an optimal scaling factor corresponding to the first state according to the first state;
[0036] Calculating the sampling point interval according to the optimal scaling factor;
[0037] A sampling module, used for collecting a plurality of sampling points at the sampling point interval in the environment model of the surrounding environment of the target vehicle, and obtaining spatial coordinate data of the plurality of sampling points;
[0038] A path planning module, used to determine a target driving path equation of the target vehicle according to the spatial coordinate data of the plurality of sampling points;
[0039] A control module is used to control the target vehicle to travel along the target driving path represented by the target driving path equation.
[0040] In a third aspect, the present application provides an intelligent vehicle path planning device, the device comprising: a processor, and a memory communicatively connected to the processor;
[0041] The memory stores computer-executable instructions;
[0042] The processor executes the computer-executable instructions stored in the memory to implement the intelligent vehicle path planning method as described in the first aspect.
[0043] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the intelligent vehicle path planning method as described in the first aspect.
[0044] The intelligent vehicle path planning method, device, equipment and storage medium provided by the present application achieve the following technical effects:
[0045] Through training, the reinforcement learning model can determine the current optimal scaling factor based on the relative distance and relative speed between the target vehicle and the obstacle, the current speed and driving direction of the target vehicle itself, so that when the target vehicle travels to an obstacle-dense area, a smaller scaling factor can be determined to increase the density of sampling points, thereby making the driving path planning more accurate and precise; when the target vehicle travels to an obstacle-sparse area, a larger scaling factor can be determined to reduce the density of sampling points, thereby simplifying the path planning process and reducing the consumption of computing resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0047] Figure 1 A method for intelligent vehicle path planning provided in this application embodiment Figure 1 ;
[0048] Figure 2 A schematic diagram of a smart vehicle path planning method provided in an embodiment of the present application Figure 2 ;
[0049] Figure 3 A schematic diagram of the structure of an intelligent vehicle path planning device provided in an embodiment of the present application;
[0050] Figure 4 A schematic diagram of the structure of an intelligent vehicle path planning device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0051] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation methods described in the following exemplary embodiments do not represent all implementation methods consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims, rather than all embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0052] First, the relevant concepts or nouns involved in this application are explained:
[0053] Reinforcement Learning: Reinforcement Learning (RL) is a machine learning method that learns how to take actions to maximize cumulative rewards by interacting with the environment. The core idea of reinforcement learning is to learn the best strategy through trial and error.
[0054] Greedy Policy: Greedy Policy is a commonly used strategy selection method in Q-learning. In Q-learning, the agent learns a Q-value function, Q(s,a), by interacting with the environment, where s represents the state and a represents the action. The Q-value function represents the expected reward that can be obtained by selecting action a in state s. The core idea of the greedy strategy is to select the action with the highest current Q value every time, that is, to select the action that is considered to be optimal in the current state. Specifically, if the agent is in state s, the greedy strategy will select action a to maximize Q(s,a). However, a purely greedy strategy may cause the agent to fall into a local optimal solution prematurely, because it always chooses the action that looks best at the moment without exploring other actions that may be better. To solve this problem, the ε-Greedy Policy is usually combined, where ε is a small positive number (0<=ε<=1). The ε-Greedy strategy works as follows:
[0055] 1. Select the action with the highest current Q value with probability 1-ε (greedy choice).
[0056] 2. Randomly select an action with probability ε (exploration).
[0057] In vehicle path planning, the vehicle needs to sample in the search space around itself to find a safe and efficient path. In areas with dense obstacles, the sampling points need to be dense to improve the precision of path planning and avoid collisions between vehicles and obstacles; in open areas, the density of sampling points can be reduced to reduce the occupation of computing resources. However, the existing driving path planning methods often use a fixed sampling point interval, which cannot be dynamically adjusted according to real-time environmental information, resulting in insufficient sampling points in areas with dense obstacles and too many sampling points in areas with sparse obstacles, which affects the quality and efficiency of path planning and is difficult to adapt to complex and changing traffic environments. In response to this problem, the technical concept of this application is: according to the state information of obstacles in the surrounding environment of the target vehicle, the current state of the target vehicle is determined, and the current state includes the relative distance and relative speed between the target vehicle and the obstacle at the current moment, the current speed and driving direction of the target vehicle itself, and the pre-trained reinforcement learning model determines the optimal scaling factor corresponding to the current state according to the current state of the target vehicle, and the scaling factor is used to adjust the interval between sampling points when sampling. Next, the interval between sampling points during the current sampling is calculated according to the optimal scaling factor, a plurality of sampling points are collected in the surrounding environment of the target vehicle according to the sampling point interval, and the driving path of the target vehicle is generated according to the sampling points.
[0058] Embodiment 1
[0059] An embodiment of the present application provides an intelligent vehicle path planning method, which can be applied to a control unit of a vehicle or a server interacting with the vehicle. Figure 1 A method for intelligent vehicle path planning provided in this application embodiment Figure 1 .like Figure 1 As shown, the method includes:
[0060] S101, determining a first state of the target vehicle at a first moment according to state information of obstacles in the surrounding environment of the target vehicle at a first moment and motion state information of the target vehicle, the first state including a relative distance and relative speed between the target vehicle and obstacles in the surrounding environment of the target vehicle at the first moment, a speed of the target vehicle, and a driving direction of the target vehicle;
[0061] Specifically, the state information of obstacles in the surrounding environment of the target vehicle at the first moment and the motion state information of the target vehicle can be obtained through the sensors carried by the target vehicle. For example, the distance and speed of obstacles around the target vehicle can be detected by a millimeter wave radar, the distance and speed of close-range obstacles can be detected by an ultrasonic sensor, and the motion state information of the target vehicle can be detected by the inertial navigation unit of the target vehicle. The motion state information of the target vehicle can be the speed and driving direction of the target vehicle. Through the data such as the distance and speed of obstacles around the target vehicle and the speed and driving direction of the target vehicle itself, the relative distance and relative speed between the target vehicle and obstacles in the surrounding environment of the target vehicle at the first moment, the speed of the target vehicle, and the driving direction of the target vehicle can be determined, that is, the first state of the target vehicle at the first moment.
[0062] S102, determining, by a pre-trained reinforcement learning model, an optimal scaling factor corresponding to the first state according to the first state;
[0063] First of all, it should be noted that there is a concept of state-action in reinforcement learning. The state is the current state of the target vehicle, which in this application includes the relative distance and relative speed between the target vehicle and obstacles in the surrounding environment, the speed of the target vehicle, and the driving direction of the target vehicle. In this application, the action refers to the scaling factor corresponding to the current state of the vehicle.
[0064] During the training phase, the vehicle is placed in a simulated environment or an actual environment for training, and the reinforcement learning model learns how to select the optimal action, i.e., the optimal scaling factor, according to the current state of the vehicle. In this embodiment, the reinforcement learning model can be based on the policy gradient method, the Q-Learning method, the Actor-Critic method, etc. In the policy gradient method, the policy network directly outputs an action (i.e., the optimal scaling factor) according to the current state of the vehicle; in the Q-learning method (such as DQN), the Q network evaluates the value of each possible action (scaling factor), and the current state of the vehicle is used as input. The Q network outputs the Q value of different actions and selects the action with the largest Q value as the optimal scaling factor; the Actor-Critic method combines the policy network and the value network. The policy network (Actor) generates an action (scaling factor), and the value network (Critic) evaluates the value of this action. By continuously optimizing these two networks, the model can output the optimal scaling factor under a given state.
[0065] S103, calculating the sampling point interval according to the optimal scaling factor;
[0066] In this step, a basic sampling interval can be first defined, which is the default sampling point interval when there is no scaling factor. After obtaining the current optimal scaling factor from the pre-trained reinforcement learning model, the basic sampling interval can be multiplied by the optimal scaling factor to calculate the current sampling point interval.
[0067] S104, collecting a plurality of sampling points at sampling point intervals in an environment model of the target vehicle's surrounding environment, and obtaining spatial coordinate data of the plurality of sampling points;
[0068] Specifically, multi-source sensors can be used to detect relevant data in the surrounding environment. For example, laser radar generates point cloud data of the surrounding environment through rotation or multi-beam scanning. Each point cloud data point contains spatial coordinates (x, y, z) and reflection intensity information; stereo vision (binocular camera) or structured light technology generates depth images, which are then converted into three-dimensional coordinates; radar emits electromagnetic waves and receives reflected signals, calculates the distance and speed of objects, and generates spatial coordinates; ultrasonic sensors emit sound waves and receive reflected signals, calculates the distance of objects, and generates spatial coordinates. The environment perceived by multi-source sensors is fused to generate an environmental model, and the spatial coordinate data of the sampling points are extracted according to the calculated sampling point intervals.
[0069] S105, determining a target driving path equation of the target vehicle according to the spatial coordinate data of the plurality of sampling points;
[0070] In this step, Bezier curves, spline curves or other smooth curve generation methods can be used to generate the local path. Commonly used algorithms include RRT (rapid random tree) and DWA (dynamic window method).
[0071] S106. Control the target vehicle to travel along the target driving path represented by the target driving path equation.
[0072] Specifically, the real-time position of the target vehicle can be determined through a high-precision GPS system, the acceleration and angular velocity information of the target vehicle can be obtained through the inertial measurement unit (IMU) carried by the target vehicle, and the surrounding environment can be sensed through the camera to detect obstacles and lane lines. Using algorithms such as the Kalman filter, the GPS, IMU and other sensor data are integrated to obtain the precise position and posture (position, speed, acceleration, direction, etc.) of the vehicle. Then, path tracking control algorithms such as PID control, model predictive control (MPC) or pure pursuit control can be used. These algorithms calculate the required steering angle, acceleration and deceleration based on the current state of the vehicle and the target driving path, and adjust the vehicle's control parameters (such as steering wheel angle, throttle and brake) in real time based on these parameters to make the vehicle travel along the target driving path.
[0073] The method provided in this embodiment achieves the following technical effects:
[0074] Through training, the reinforcement learning model can determine the current optimal scaling factor based on the relative distance and relative speed between the target vehicle and the obstacle, the current speed and driving direction of the target vehicle itself, so that when the target vehicle travels to an obstacle-dense area, a smaller scaling factor can be determined to increase the density of sampling points, thereby making the driving path planning more accurate and precise; when the target vehicle travels to an obstacle-sparse area, a larger scaling factor can be determined to reduce the density of sampling points, thereby simplifying the path planning process and reducing the consumption of computing resources.
[0075] The following uses a specific embodiment to explain in detail the intelligent vehicle path planning method of the present application.
[0076] Embodiment 2
[0077] Figure 2 A schematic diagram of a smart vehicle path planning method provided in an embodiment of the present application Figure 2 .like Figure 2 As shown, the method includes:
[0078] S201, Environmental Perception and Modeling;
[0079] Specifically, the sensors carried by the target vehicle may include laser radar, millimeter wave radar, high-definition camera, ultrasonic sensor, etc. Laser radar is used to obtain three-dimensional point cloud data of the surrounding environment, millimeter wave radar is used to detect the distance and speed of obstacles, high-definition camera is used to capture information such as traffic signals and road signs, and ultrasonic sensor is used to detect obstacles at close range.
[0080] Next, the collected raw data needs to go through preprocessing (such as denoising, filtering, etc.) and data fusion (such as spatiotemporal alignment of sensor information, data association, etc.) steps to form a consistent and accurate perception environment model. This process can use algorithms such as Kalman filtering, extended Kalman filtering, and particle filtering to estimate and predict the state of dynamic obstacles. Specifically, since there are more dynamic obstacles on the road (such as other vehicles and pedestrians), for dynamic obstacles, the prediction information of the motion state of dynamic obstacles in the environment around the target vehicle at the previous moment before the first moment and the detection information of the motion state of dynamic obstacles at the first moment can be used to calculate the motion state information of dynamic obstacles at the first moment through the filter. The detection information of the motion state of dynamic obstacles at the first moment can be obtained through the above-mentioned laser radar, millimeter wave radar, high-definition camera, ultrasonic sensor, etc.
[0081] For static obstacles, the relative distance between the static obstacle and the target vehicle at the first moment, the size of the static obstacle and other state information can be detected, and the state information of the static obstacle and the motion state information of the dynamic obstacle can be used as the state information of the obstacles around the target vehicle at the first moment.
[0082] The processed perception data is integrated with the road information provided by the high-precision map to build a more accurate environmental model, including road network model, obstacle model, traffic signal model, etc. These models can be represented in the form of raster map, topological map or semantic map to provide necessary spatial information for subsequent path planning.
[0083] S202, determining a first state of the target vehicle at the first moment according to state information of obstacles in the surrounding environment of the target vehicle at the first moment and motion state information of the target vehicle, the first state including a relative distance and relative speed between the target vehicle and obstacles in the surrounding environment of the target vehicle at the first moment, a speed of the target vehicle, and a driving direction of the target vehicle;
[0084] Specifically, the real-time position of the target vehicle can be determined by a high-precision GPS system, and the acceleration, angular velocity, heading and other information of the target vehicle can be obtained by an inertial measurement unit (IMU) carried by the target vehicle, and the real-time position, acceleration, angular velocity, heading and other information of the target vehicle at the first moment can be used as the motion state information of the target vehicle at the first moment. According to the state information of the obstacle at the first moment and the motion state information of the target vehicle at the first moment, the first state of the target vehicle at the first moment can be calculated, that is, the relative distance and relative speed between the target vehicle and the obstacles in the surrounding environment of the target vehicle at the first moment, the speed of the target vehicle, and the driving direction of the target vehicle.
[0085] S203, determining, by the pre-trained reinforcement learning model, an optimal scaling factor corresponding to the first state according to the first state;
[0086] Optionally, the pre-trained reinforcement learning model determines the first state-scaling factor pair with the highest value as the first target state-scaling factor pair among the first state-scaling factor pairs consisting of the first state and the scaling factor corresponding to the first state according to the first state, and determines the scaling factor in the first target state-scaling factor pair as the optimal scaling factor. The pre-trained reinforcement learning model can be based on the Q-learning method, taking the first state of the target vehicle as input, and the Q-learning method outputs the Q values of different actions (scaling factors), and selects the scaling factor with the largest Q value as the optimal scaling factor.
[0087] S204, calculating the sampling point interval according to the optimal scaling factor;
[0088] S205, collecting a plurality of sampling points at sampling point intervals in the environment model of the target vehicle's surrounding environment, and obtaining spatial coordinate data of the plurality of sampling points;
[0089] S206, determining a target driving path equation of the target vehicle according to the spatial coordinate data of the plurality of sampling points;
[0090] Specifically, according to the spatial coordinate data of the plurality of sampling points, a plurality of candidate driving path equations are generated by curve fitting, such as polynomial fitting;
[0091] According to the target cost function, the target driving path is determined from multiple candidate driving paths so that the target cost function value is the lowest when the target vehicle drives along the target driving path. The target cost function includes the target vehicle and obstacle collision risk penalty factor, the target vehicle driving time penalty factor, the target vehicle acceleration change rate penalty factor, and the target driving path curvature penalty factor. It should be noted that if the acceleration change rate of the vehicle is large, it means that the vehicle frequently accelerates and decelerates, which will inevitably cause the passengers to have a poor riding experience. The large curvature of the driving path will also affect the riding comfort. Through the target cost function, the comprehensive cost of collision risk, driving time, acceleration change rate, and path curvature is minimized when the vehicle drives along the target driving path, thereby ensuring the safety, comfort, and efficiency of the ride.
[0092] S207, controlling the target vehicle to travel along the target driving path represented by the target driving path equation;
[0093] S208, determining a second state of the target vehicle at the second moment according to the state information of the obstacles in the surrounding environment of the target vehicle at the second moment and the motion state information of the target vehicle, the second state including the relative distance and relative speed between the target vehicle and the obstacles in the surrounding environment of the target vehicle at the second moment, the speed of the target vehicle, and the driving direction of the target vehicle;
[0094] S209: Update the value corresponding to the first target state-scaling factor pair according to the second state.
[0095] S208 and S209 are similar to the pre-training stage of the reinforcement learning model. The difference is that S208 and S209 are the reinforcement learning model that is pre-trained and then transferred to the target vehicle and continues to be trained in the traffic scene actually faced by the target vehicle.
[0096] First, the pre-training stage of the reinforcement learning model based on the Q-learning method is explained.
[0097] In the pre-training phase, we first need to define the state of the vehicle. Specifically, we define the state space s of the vehicle, including the relative distance between the vehicle and the obstacle, the relative speed, and the current speed and direction of the vehicle. Then we define the action, and the action space a is defined as the scaling factor for adjusting the sampling point interval.
[0098] A reward function is also needed to evaluate the vehicle's behavior in different states. The reward function is designed to encourage the smart car to set more dense sampling points near obstacles and set more sparse sampling points when far away from obstacles. At the same time, the design of the reward function also comprehensively considers indicators such as the safety, comfort and efficiency of the planned path.
[0099] Specifically, the reward function can determine a scaling factor threshold according to the current state of the vehicle. For example, a smaller scaling factor threshold is set when the relative distance between the vehicle and the obstacle is small or the density of the surrounding obstacles is large, and a larger scaling factor threshold is set when the relative distance between the vehicle and the obstacle is large or the density of the surrounding obstacles is low. When the action (scaling factor) selected by the reinforcement learning model according to the current state of the vehicle is less than or equal to the scaling factor threshold, a positive reward (usually a positive number) is given, otherwise a negative reward (usually a negative number) is given to encourage the reinforcement learning model to set more dense sampling points near the obstacle and set sparser sampling points when far away from the obstacle. After the reinforcement learning model selects a certain scaling factor according to the current state of the vehicle and finally plans the vehicle's driving path according to the scaling factor, the reward function can evaluate the impact of the vehicle's selection of the scaling factor on the vehicle's driving safety, ride comfort, and driving efficiency in the current state in the following ways: evaluate the distance between the vehicle and the surrounding obstacles when traveling along the driving path, and give a negative reward if the distance is less than the distance threshold, otherwise a positive reward is given, thereby evaluating the safety of the vehicle using the scaling factor in the current state; evaluate the curvature of the driving path, and give a negative reward if the curvature of the driving path exceeds the curvature threshold, otherwise a positive reward is given, thereby evaluating the ride comfort of the vehicle using the scaling factor in the current state; evaluate the driving time of the driving path, and obtain the speed limit of the current road surface from the model obtained above, and then calculate the driving time according to the length of the driving path, and give a negative reward if the driving time exceeds the driving time threshold, otherwise a positive reward is given, thereby evaluating the driving efficiency of the vehicle using the scaling factor in the current state.
[0100] You also need to set the following parameters of the reinforcement learning model:
[0101] Learning rate α: used to control the step size of updating the Q value;
[0102] Discount factor γ: used to calculate the present value of future rewards;
[0103] Exploration rate ε: It is used to balance exploration and exploitation. It is set high at the beginning and gradually decreases during training.
[0104] After the above settings are made, the reinforcement learning model based on the Q-learning method is trained in a simulated environment or an actual environment to learn how to select the best scaling factor based on the current state. The Q value table is used to store the expected reward value (Q value) for each action (scaling factor) in each state.
[0105] In the model pre-training phase, the following steps are mainly included:
[0106] Environment configuration: At the beginning of each round, the vehicle is placed at a random position on the path and the obstacle information is initialized.
[0107] Select an action:
[0108] Step 1: Randomly select a scaling factor with probability ε (exploration), and select the scaling factor with the largest Q value in the current state with probability 1-ε (utilization);
[0109] Step 2: Execute the action: adjust the sampling point interval according to the selected action, generate the driving path, and then simulate the vehicle to drive one step according to the generated driving path;
[0110] Observe the reward and next state:
[0111] The reward function is used to evaluate the impact of the selected scaling factor and the generated driving path on vehicle driving safety, ride comfort, and driving efficiency, and the reward value is calculated. Then the Q value table is updated through the Bellman equation:
[0112] The Bellman equation is:
[0113] Q(s,a)←Q(s,a)+α[r+γMax a‘ Q(s',a')-Q(s,a)]
[0114] Q(s, a) is the Q value corresponding to the scaling factor selected in step 1 above; r is the reward value; after the driving path is generated, the simulated vehicle drives one step along the generated driving path, and the vehicle changes to a new state (after the vehicle drives, the relative distance between the vehicle and the surrounding obstacles, the relative speed, the vehicle's speed, the driving direction, etc. will inevitably change). This new state corresponds to different scaling factors, and different scaling factors correspond to different Q values. The largest Q value is Max a‘ Q(s', a'), in other words, Max a‘ Q(s', a') is the value of the action with the highest value (scaling factor) after the vehicle transitions to the new state.
[0115] In S209, according to the second state, the value corresponding to the first target state-scaling factor pair is updated, including:
[0116] According to the reward function, determine the reward value of the optimal scaling factor;
[0117] The pre-trained reinforcement learning model determines, according to the second state, a second state-scaling factor pair with the highest corresponding value among second state-scaling factor pairs consisting of the second state and the scaling factor corresponding to the second state as a second target state-scaling factor pair;
[0118] According to the reward value and the value corresponding to the second target state-scaling factor pair, the value corresponding to the first target state-scaling factor pair is updated through the Bellman equation.
[0119] Specifically, the principles of S208 and S209 are similar to those of the above pre-training stage, except that Q(s, a) is the value corresponding to the first target state-scaling factor pair, s is the first state, a is the optimal scaling factor, α is the learning rate of the pre-trained reinforcement learning model, r is the reward value, γ is the discount rate of the pre-trained reinforcement learning model, and Max a‘ Q(s', a') is the value corresponding to the second target state-scaling factor pair, s' is the second state, and a' is the scaling factor in the second state-scaling factor. The first state is the state of the target vehicle at the first moment in the real road environment. The future driving path is planned according to the first state, and the target vehicle changes to the second state after driving along the driving path. The role of the reward function is the same as that of the reward function in the pre-training stage, which will not be repeated here.
[0120] Since the pre-training stage cannot fully simulate the real road conditions, after the reinforcement learning model is trained, it is necessary to conduct subsequent training through real road conditions, that is, to update the Q value of the scaling factor of the target vehicle in each state through real road conditions. In other words, it is based on the pre-trained reinforcement learning model, and continues to be trained in the same way. This enables the reinforcement learning model to quickly adapt to new environments or tasks without retraining the entire model, which improves the flexibility and practicality of the reinforcement learning model. The trained model is deployed to the target vehicle and tested and adjusted in the actual road environment. As driving experience accumulates, the algorithm will continue to learn from actual data and optimize the policy model to adapt to changes in different environments and driving needs.
[0121] The embodiment of the present invention can divide the electronic device or the main control device into functional modules according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.
[0122] Figure 3 A schematic diagram of the structure of an intelligent vehicle path planning device provided in an embodiment of the present application.
[0123] like Figure 3 As shown, the device 30 includes:
[0124] A state determination module 301 is used to determine a first state of the target vehicle at a first moment according to state information of obstacles in the surrounding environment of the target vehicle at a first moment and motion state information of the target vehicle, wherein the first state includes a relative distance and relative speed between the target vehicle and obstacles in the surrounding environment of the target vehicle at the first moment, a speed of the target vehicle, and a driving direction of the target vehicle;
[0125] The sampling point interval calculation module 302 is used to:
[0126] Determining, by the pre-trained reinforcement learning model, an optimal scaling factor corresponding to the first state according to the first state;
[0127] According to the optimal scaling factor, the sampling point interval is calculated;
[0128] The sampling module 303 is used to collect a plurality of sampling points at sampling point intervals in the environment model of the surrounding environment of the target vehicle, and obtain spatial coordinate data of the plurality of sampling points;
[0129] A path planning module 304 is used to determine a target driving path equation of the target vehicle according to the spatial coordinate data of the plurality of sampling points;
[0130] The control module 305 is used to control the target vehicle to travel along the target driving path represented by the target driving path equation.
[0131] The present embodiment provides an intelligent vehicle path planning device, which can execute the intelligent vehicle path planning method of the above-mentioned embodiment. Its implementation principle and technical effect are similar, and will not be described in detail in this embodiment.
[0132] In the specific implementation of the aforementioned intelligent vehicle path planning method, each module can be implemented as a processor, and the processor can execute computer-executable instructions stored in the memory, so that the processor executes the aforementioned intelligent vehicle path planning method.
[0133] Figure 4 This is a schematic diagram of the structure of an intelligent vehicle path planning device provided in an embodiment of the present application. Figure 4 As shown, the device 40 includes: at least one processor 401 and a memory 402. The electronic device 40 also includes a communication component 403. The processor 401, the memory 402 and the communication component 403 are connected via a bus 404.
[0134] In the specific implementation process, at least one processor 501 executes the computer execution instructions stored in the memory 502, so that at least one processor 501 executes the intelligent vehicle path planning method executed by the electronic device side as above.
[0135] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0136] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0137] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.
[0138] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.
[0139] The above-mentioned functions implemented by the electronic device and the main control device introduce the scheme provided by the embodiment of the present invention. It can be understood that in order to implement the above-mentioned functions, the electronic device or the main control device includes a hardware structure and / or software module corresponding to the execution of each function. In combination with the units and algorithm steps of each example described in the embodiment disclosed in the embodiment of the present invention, the embodiment of the present invention can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present invention.
[0140] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above-mentioned intelligent vehicle path planning method is implemented.
[0141] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.
[0142] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in an electronic device or a main control device.
[0143] The present application also provides a computer program product, which includes: a computer program, which is stored in a readable storage medium. At least one processor of an electronic device can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the electronic device executes the solution provided by any of the above embodiments.
[0144] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.
[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for intelligent vehicle path planning, characterized in that: The method comprises: Determine, according to state information of obstacles in the surrounding environment of the target vehicle at a first moment and motion state information of the target vehicle, a first state of the target vehicle at the first moment, wherein the first state includes a relative distance and relative speed between the target vehicle and obstacles in the surrounding environment of the target vehicle at the first moment, a speed of the target vehicle, and a driving direction of the target vehicle; Determining, by a pre-trained reinforcement learning model, an optimal scaling factor corresponding to the first state according to the first state; Calculating the sampling point interval according to the optimal scaling factor; In the environment model of the target vehicle's surrounding environment, a plurality of sampling points are collected at the sampling point interval, and spatial coordinate data of the plurality of sampling points are acquired; Determining a target driving path equation of the target vehicle according to the spatial coordinate data of the plurality of sampling points; The target vehicle is controlled to travel along the target driving path represented by the target driving path equation.
2. The method according to claim 1, characterized in that: Determining, by the pre-trained reinforcement learning model according to the first state, an optimal scaling factor corresponding to the first state, includes: The pre-trained reinforcement learning model determines, according to the first state, a first state-scaling factor pair having the highest corresponding value among first state-scaling factor pairs consisting of the first state and the scaling factor corresponding to the first state as a first target state-scaling factor pair; The scaling factor in the first target state-scaling factor pair is determined as the optimal scaling factor.
3. The method according to claim 1, characterized in that Determining the target driving path equation of the target vehicle according to the spatial coordinate data of the plurality of sampling points includes: Generating a plurality of candidate driving path equations by curve fitting according to the spatial coordinate data of the plurality of sampling points; According to the target cost function, the target driving path is determined among the multiple candidate driving paths so that the target cost function value is the lowest when the target vehicle travels along the target driving path. The target cost function includes a penalty factor for the risk of collision between the target vehicle and the obstacle, a penalty factor for the target vehicle's driving time, a penalty factor for the target vehicle's acceleration rate of change, and a penalty factor for the target driving path curvature.
4. The method according to claim 2, characterized in that: After controlling the target vehicle to travel according to the target driving path represented by the target driving path equation, the method further includes: Determine a second state of the target vehicle at the second moment according to state information of obstacles in the surrounding environment of the target vehicle and motion state information of the target vehicle, wherein the second state includes a relative distance and relative speed between the target vehicle and obstacles in the surrounding environment of the target vehicle at the second moment, a speed of the target vehicle, and a driving direction of the target vehicle; According to the second state, the value corresponding to the first target state-scaling factor pair is updated.
5. The method according to claim 4, characterized in that The updating of the value corresponding to the first target state-scaling factor pair according to the second state includes: Determining a reward value of the optimal scaling factor according to a reward function; The pre-trained reinforcement learning model determines, according to the second state, a second state-scaling factor pair having the highest corresponding value among second state-scaling factor pairs consisting of the second state and the scaling factor corresponding to the second state as a second target state-scaling factor pair; According to the reward value and the value corresponding to the second target state-scaling factor pair, the value corresponding to the first target state-scaling factor pair is updated by the Bellman equation, and the Bellman equation is: Q(s,a)←Q(s,a)+α[r+γMax a‘ Q(s,,a,)-Q(s,a)] Wherein, Q(s, a) is the value corresponding to the first target state-scaling factor pair, s is the first state, a is the optimal scaling factor, α is the learning rate of the pre-trained reinforcement learning model, r is the reward value, γ is the discount rate of the pre-trained reinforcement learning model, Max a‘ Q(s,,a,) is the value corresponding to the second target state-scaling factor pair, s, is the second state, and a, is the scaling factor in the second state-scaling factor.
6. The method according to claim 5, characterized in that Determining the reward value of the optimal scaling factor according to the reward function includes: Determining, according to the first state, a scaling factor threshold corresponding to the first state; If the optimal scaling factor is less than or equal to the scaling factor threshold, the reward function uses a target positive number as the reward value; if the optimal scaling factor is greater than the scaling factor threshold, the reward function uses a target negative number as the reward value.
7. The method according to claim 6, characterized in that Before determining the first state of the target vehicle at the first moment according to the state information of obstacles in the surrounding environment of the target vehicle at the first moment and the motion state information of the target vehicle, the method further includes: According to the prediction information of the motion state of the dynamic obstacle in the surrounding environment of the target vehicle by the filter at the previous moment and the detection information of the motion state of the dynamic obstacle at the first moment, the motion state information of the dynamic obstacle at the first moment is calculated by the filter, and the motion state information of the dynamic obstacle at the first moment is used as the state information of the obstacle in the surrounding environment of the target vehicle at the first moment.
8. An intelligent vehicle path planning device, characterized in that: The device comprises: A state determination module, for determining a first state of the target vehicle at the first moment according to state information of obstacles in the surrounding environment of the target vehicle at the first moment and motion state information of the target vehicle, wherein the first state includes a relative distance and relative speed between the target vehicle and obstacles in the surrounding environment of the target vehicle at the first moment, a speed of the target vehicle, and a driving direction of the target vehicle; Sampling point interval calculation module, used for: Determining, by a pre-trained reinforcement learning model, an optimal scaling factor corresponding to the first state according to the first state; Calculating the sampling point interval according to the optimal scaling factor; A sampling module, used for collecting a plurality of sampling points at the sampling point interval in the environment model of the surrounding environment of the target vehicle, and obtaining spatial coordinate data of the plurality of sampling points; A path planning module, used to determine a target driving path equation of the target vehicle according to the spatial coordinate data of the plurality of sampling points; A control module is used to control the target vehicle to travel along the target driving path represented by the target driving path equation.
9. An intelligent vehicle path planning device, characterized in that: The device comprises: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the intelligent vehicle path planning method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the intelligent vehicle path planning method as described in any one of claims 1 to 7.
Citation Information
Cited By
Control method and system for automatic turning of automobile
CN121492930A
