A reinforcement learning-based automatic driving path planning method and system

Abstract

This invention provides a reinforcement learning-based method and system for autonomous driving path planning, relating to the field of autonomous driving technology. The method includes: constructing an environment model; constructing a path decision agent; defining the agent's state space, action space, and reward function for reinforcement learning; training the path decision agent until it converges; deploying the converged path decision agent to an autonomous vehicle, acquiring the vehicle's own state and the environment state in real time, generating a driving path by outputting the optimal path decision action through the agent, and adjusting the path in real time according to dynamic environmental changes to achieve adaptive path planning. This invention optimizes the reinforcement learning framework, designs a reasonable state space, action space, and reward function, and combines vehicle dynamics constraints and traffic rule constraints to achieve safe, efficient, and smooth path planning for autonomous vehicles in dynamic and complex environments, while improving the agent's training efficiency and reducing computational costs.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and specifically to an autonomous driving path planning method and system based on reinforcement learning. Background Technology

[0002] Autonomous driving path planning is one of the core technologies of autonomous driving systems. Its core task is to plan a driving path for autonomous vehicles that meets multiple objectives such as safety, comfort, and efficiency, based on the vehicle's own state, environmental perception information (such as obstacles, lane lines, and traffic signals), and navigation destination constraints. It is the prerequisite and foundation for vehicles to achieve autonomous driving.

[0003] Currently, autonomous driving path planning methods are mainly divided into two categories: traditional path planning methods and machine learning-based path planning methods. Traditional path planning methods include algorithms such as A*, Dijkstra, and RRT*. These methods are simple in principle and have a fast computation speed, and can meet basic path planning needs in static structured environments. However, they have obvious limitations: First, they are poorly adaptable to dynamic environments. When sudden obstacles appear in the traffic environment (such as pedestrians crossing or vehicles changing lanes illegally), it is difficult to quickly adjust the path, which can easily lead to path conflicts or unsafe driving. Second, they lack multi-objective optimization capabilities, making it difficult to simultaneously consider multiple mutually constraining objectives such as path smoothness, driving efficiency, and energy consumption optimization. The planned paths often have problems such as poor driving comfort and high energy consumption. Third, they have weak generalization ability. Under different road conditions (such as highways, urban roads, and rural roads) and weather conditions (such as rain, snow, and fog), parameters need to be manually adjusted, making it difficult to achieve adaptive planning.

[0004] With the development of machine learning technology, path planning methods based on reinforcement learning have gradually become a research hotspot. Reinforcement learning learns optimal decision-making strategies through continuous interaction between the agent and the environment, without the need to pre-build accurate environmental models, and has strong adaptability and generalization ability in dynamic and uncertain environments. However, existing reinforcement learning-based autonomous driving path planning methods still have many shortcomings: First, the training efficiency is low. Reinforcement learning agents require a large number of interaction samples to converge in complex traffic environments, resulting in long training cycles and high computational costs. Second, path safety is difficult to guarantee. Some methods do not fully consider vehicle dynamic constraints (such as maximum steering angle and maximum speed) and traffic rule constraints (such as speed limits, lane driving, and avoidance rules), and the planned paths may exceed the vehicle's driving capabilities or violate traffic rules. Third, the path smoothness is insufficient. The planned paths often have problems such as frequent changes of direction and excessively large steering angles, affecting driving comfort. Fourth, the reward function design is unreasonable. Many methods use single or simple composite reward functions, which are difficult to accurately quantify the multiple objectives of path safety, efficiency, and smoothness, and are prone to causing the agent to learn local optimal strategies rather than globally optimal paths.

[0005] Therefore, existing reinforcement learning-based autonomous driving path planning methods suffer from drawbacks such as low training efficiency, poor path safety, insufficient smoothness, and weak multi-objective optimization capabilities, resulting in poor driving path planning performance in dynamic and uncertain environments. Summary of the Invention

[0006] To address the aforementioned problems, this invention proposes an autonomous driving path planning method and system based on reinforcement learning. By optimizing the reinforcement learning framework, designing reasonable state space, action space, and reward function, and combining vehicle dynamics constraints and traffic rule constraints, it achieves safe, efficient, and smooth path planning for autonomous vehicles in dynamic and complex environments, while improving the training efficiency of the agent and reducing computational costs.

[0007] According to some embodiments, the present invention adopts the following technical solution: An autonomous driving path planning method based on reinforcement learning includes: Construct an autonomous driving path planning environment model to obtain the autonomous vehicle's own state, environmental state, and path prediction state, and provide an interaction interface between the agent and the environment, while loading vehicle dynamics constraints and traffic rule constraints. A path decision-making agent is constructed, comprising a state encoding module, a policy network module, a value network module, and an experience playback module. The state encoding module is used to encode high-dimensional data in the state space, the policy network module is used to output path decision actions, the value network module is used to evaluate the value of the actions, and the experience playback module is used to store experience samples generated by the interaction between the agent and the environment model. Define the state space, action space, and reward function for agent reinforcement learning. The state space integrates the vehicle's own state, the environment's state, and the path prediction state. The action space is designed based on vehicle dynamics constraints. The reward function adopts a multi-objective weighted composite reward function designed based on traffic rule constraints. The path decision-making agent is trained by continuously interacting with the environment model and the agent to generate experience samples and store them in the experience playback module. The policy network module and value network module are trained by an alternating training strategy. An exploration rate decay mechanism and a gradient pruning mechanism are introduced until the agent converges. The trained and converged path decision agent is deployed to autonomous vehicles to obtain the vehicle's own state and the environment state in real time. The agent outputs the optimal path decision action to generate a driving path and adjusts the path in real time according to the dynamic changes in the environment to achieve adaptive path planning.

[0008] According to some embodiments, the present invention adopts the following technical solution: An autonomous driving path planning system based on reinforcement learning includes: The environment model building module is configured to: build an autonomous driving path planning environment model to obtain the autonomous vehicle's own state, environmental state and path prediction state, and provide an interaction interface between the agent and the environment, while loading vehicle dynamics constraints and traffic rule constraints. The agent construction module is configured to: construct a path decision agent, which includes a state encoding module, a policy network module, a value network module and an experience playback module. The state encoding module is used to encode high-dimensional data in the state space, the policy network module is used to output path decision actions, the value network module is used to evaluate the value of the actions, and the experience playback module is used to store experience samples generated by the interaction between the agent and the environment model. The reinforcement learning definition module is configured to define the state space, action space, and reward function of the agent's reinforcement learning. The state space integrates the vehicle's own state, the environment's state, and the path prediction state. The action space is designed based on vehicle dynamics constraints. The reward function adopts a multi-objective weighted composite reward function designed based on traffic rule constraints. The agent training module is configured to: train the path decision agent, generate experience samples and store them in the experience playback module through continuous interaction between the environment model and the agent, train the policy network module and the value network module using an alternating training strategy, and introduce an exploration rate decay mechanism and a gradient pruning mechanism until the agent converges. The deployment planning module is configured to: deploy the trained and converged path decision agent to the autonomous vehicle, acquire the vehicle's own state and the environment state in real time, output the optimal path decision action through the agent, generate the driving path, and adjust the path in real time according to the dynamic changes of the environment to achieve adaptive path planning.

[0009] According to some embodiments, the present invention adopts the following technical solution: A computer program product includes a computer program that, when executed by a processor, implements the aforementioned reinforcement learning-based autonomous driving path planning method.

[0010] According to some embodiments, the present invention adopts the following technical solution: A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the aforementioned reinforcement learning-based autonomous driving path planning method.

[0011] According to some embodiments, the present invention adopts the following technical solution: An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the reinforcement learning-based autonomous driving path planning method.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a high-dimensional state space that integrates the vehicle's own state, the environment's state, and the path prediction state. By combining multi-source perception information fusion technology, it improves the accuracy and comprehensiveness of state perception, enabling the intelligent agent to accurately grasp the changes in complex dynamic environments and provide a reliable decision basis for path planning. At the same time, based on vehicle dynamics constraints, a continuous action space is designed to ensure that the planned path conforms to the vehicle's driving capabilities and avoids path decisions that exceed the vehicle's performance.

[0013] 2. This invention designs a multi-objective weighted composite reward function that comprehensively considers the safety, efficiency, smoothness, rule compliance, and goal orientation of the path. By reasonably setting the weight coefficients, it achieves multi-objective collaborative optimization, solves the local optimum problem caused by unreasonable reward function design in existing methods, and the planned path is not only safe and compliant, but also has good driving comfort and efficiency, thus improving the driving experience of autonomous vehicles.

[0014] 3. This invention employs the Deep Deterministic Policy Gradient (DDPG) algorithm combined with Priority Experience Replay (PER) mechanism to construct a reinforcement learning agent. It introduces an exploration rate decay mechanism and a gradient pruning mechanism, which not only improves the training efficiency of the agent, shortens the training cycle, and reduces the computational cost, but also improves the stability of training, avoids gradient explosion and training oscillation problems, and enables the agent to quickly converge to the global optimal policy.

[0015] 4. The autonomous driving path planning environment model constructed in this invention integrates vehicle dynamics constraints and traffic rule constraints, simulating real complex dynamic traffic scenarios. The intelligent agent can learn path planning strategies to adapt to different road conditions and environmental changes through continuous interaction with the environment. It has strong generalization ability and can achieve adaptive path planning in dynamic environments without manual parameter adjustment. It is applicable to various scenarios such as highways, urban areas, and rural roads, and has broad application prospects.

[0016] 5. In the path generation process, the present invention uses B-spline interpolation algorithm to smooth the path point sequence, which further improves the smoothness of the path, avoids the problems of frequent changes of direction and excessive steering angle, and improves driving comfort. At the same time, it can adjust the path in real time according to the dynamic changes of the environment, and has a rapid response capability to sudden obstacles, traffic signal changes, etc., which significantly improves the safety and reliability of autonomous driving path planning. Attached Figure Description

[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0018] Figure 1 This is the overall flowchart of Example 1; Figure 2 Flowchart for constructing the environment model of Example 1; Figure 3 Flowchart for constructing the agent model in Example 1; Figure 4 This is a flowchart of the agent training process in Example 1. Detailed Implementation

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0020] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0021] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0022] Terminology Explanation Deep Deterministic Policy Gradient (DDPG): A reinforcement learning algorithm for continuous action spaces, employing a deterministic policy gradient framework of "Actor + Critic". The Actor outputs continuous actions based on the state, and the Critic evaluates the value of the state-action pair. During training, it is usually combined with experience replay and a target network to reduce sample correlation and stabilize learning. The parameters of the target network can be slowly updated to follow the changes of the main network.

[0023] Prioritized Experience Replay (PER): When sampling in the replay pool, instead of uniformly and randomly sampling all experiences, higher sampling probabilities are assigned based on the "importance" of the samples (commonly based on the TD error magnitude corresponding to the sample; the larger the error, the less accurate the current estimate is and the more worthwhile it is to learn repeatedly), so that the learning is more focused on the "more informative" transfers; at the same time, importance sampling weights are usually used to correct the bias caused by this non-uniform sampling in order to maintain the rationality and stability of the updates.

[0024] Temporal Difference Error (TD error): In reinforcement learning, the next step value estimate measures the difference between the current value estimate and the one-step bootstrapping objective. "Current Value Estimate"; the larger the TD error, the less accurate the current network's estimate of the transition, so it is often used as the priority basis for PER and as the core signal of Critic loss.

[0025] Target network soft update: To improve training stability, a target network with the same structure as the main network but with slower parameter updates is introduced to construct the training target. Soft update refers to the proportional mixing and updating of the target network parameters. For example, the main network parameters are copied to the target network in a "small amount" with a coefficient τ, thereby avoiding drastic fluctuations in the target value.

[0026] ε-greedy strategy and ε decay: A strategy for balancing exploration and exploitation, where a random action is chosen with probability ε for exploration, and a random action is chosen with probability 1 for exploitation. ε selects the action given by the current policy to utilize; exploration rate decay refers to gradually reducing ε during training, so that the early stage is more fully explored and the learned policy is utilized more in the later stage, thereby improving convergence and stability.

[0027] Kalman Filter: A recursive state estimation algorithm that uses a system dynamic model and noisy sequential observations to fuse multi-source measurement information in a "prediction-update" loop, outputting a better estimate of the system state (accompanied by uncertainty / covariance metrics). In multi-sensor fusion, it is often used to fuse information from radar, cameras, positioning, and vehicle bus into a consistent state estimate.

[0028] Gradient clipping: After obtaining the parameter gradients through backpropagation and before executing the optimizer update, the overall norm of all parameter gradients is calculated. If this norm exceeds the threshold max_norm, the gradients of each parameter are scaled proportionally to limit the overall gradient norm to within max_norm (the gradients are modified in place). This is used to suppress training instability caused by gradient explosion (such as severe loss oscillations and numerical overflow), thereby improving training robustness.

[0029] Example 1 One embodiment of the present invention provides an autonomous driving path planning method based on reinforcement learning, which improves the safety, efficiency, and smoothness of path planning. It features high training efficiency, strong generalization ability, and is applicable to various autonomous driving vehicles and diverse traffic scenarios, such as... Figure 1 As shown, it includes: S1. Construct an autonomous driving path planning environment model to obtain the autonomous vehicle's own state, environmental state, and path prediction state, and provide an interaction interface between the agent and the environment, while loading vehicle dynamics constraints and traffic rule constraints.

[0030] In this embodiment, the environment model includes a state perception module, an environment interaction module, and a constraint condition module, such as... Figure 2 As shown, the specific construction includes: S11. Construction of the state-aware module: By integrating data from lidar, cameras, millimeter-wave radar, GPS / BeiDou navigation systems, and the vehicle's CAN bus, multi-source perception information is obtained, including the following three types of information: (1) The vehicle's own state, including vehicle speed v, heading angle θ, position coordinates (x,y), steering angle δ, and acceleration a.

[0031] (2) Environmental conditions, including the position (x, y), velocity v, heading angle θ, and size information of obstacles, lane line position, traffic signal status, and road speed limit. .

[0032] (3) Path prediction status, including navigation destination coordinates ( , ), the straight-line distance from the vehicle's current location to its destination .

[0033] S12. Construction of the environment interaction module: A mapping relationship is established between the agent's actions and the environment's state. After the agent outputs a path decision action, the environment interaction module updates the vehicle's own state and the environment's state according to the vehicle dynamics model and the rules of environmental change, simulating a real driving scenario. The vehicle dynamics model adopts a simplified two-degree-of-freedom model, and its expression is as follows:

[0034]

[0035]

[0036]

[0037] Where f is the rolling resistance coefficient, ρ is the air density, C is the air resistance coefficient, A is the vehicle's frontal area, m is the vehicle's mass, L is the vehicle's wheelbase, δ is the steering angle, and a is the acceleration.

[0038] S13. Construction of the constraint module: Load two types of constraints, namely vehicle dynamics constraints and traffic rule constraints; (1) Vehicle dynamics constraints include: maximum steering angle constraint Maximum steering angular velocity constraint Maximum speed constraint Maximum acceleration constraint ; (2) Traffic rules constraints include: lane driving constraints (vehicles must travel within the current lane and meet safe distance requirements when changing lanes) and speed limit constraints (vehicle speed must not exceed the road speed limit). ), avoidance constraints (distance from obstacles not less than a safe distance) Traffic signal constraints (driving according to traffic signal status: stop at red lights, drive at green lights, and slow down and stop at yellow lights).

[0039] S2. Construct a path decision-making agent, which includes a state encoding module, a policy network module, a value network module, and an experience replay module. The state encoding module encodes high-dimensional data in the state space, the policy network module outputs path decision actions, the value network module evaluates the value of the actions, and the experience replay module stores experience samples generated by the agent's interaction with the environment model, improving training stability and efficiency. Figure 3 As shown, the specific construction steps are as follows: S21. Construction of the status coding module: A structure combining convolutional neural networks (CNN) and multilayer perceptrons (MLP) is used to encode high-dimensional data in the state space. CNN is used to extract spatial features in the environmental state (such as obstacle distribution and lane line shape), while MLP is used to process one-dimensional features in the vehicle's own state and path prediction state. The encoded low-dimensional feature vector is output as the input to the policy network module and the value network module.

[0040] S22. Construction of the policy network module: The policy network structure employing the Deep Deterministic Policy Gradient (DDPG) algorithm is the Actor network in the Actor-Critic architecture. The input is a low-dimensional feature vector after state encoding, and the output is a continuous path decision action, including the steering angle increment Δδ and the acceleration increment Δa. The policy network uses the ReLU activation function, and the output layer uses the tanh activation function to limit the action output within a reasonable range and satisfy the vehicle dynamics constraints.

[0041] S23. Construction of the value network module: As the Critic network in the Actor-Critic architecture, the input is the low-dimensional feature vector after state encoding and the action vector output by the policy network. The output is the value evaluation value of the state-action pair. The value evaluation value is a prediction of the long-term future returns and is used to guide the update of the policy network. The value network uses the ReLU activation function, and the output layer is a linear activation function. S24. Construction of the experience replay module: The Priority Experience Replay (PER) mechanism is adopted to store experience samples (s, u, r, s', done) generated by the interaction between the agent and the environment, where s is the current state, u is the current action, and r is the reward value obtained by the current action, which is the immediate feedback.

[0042] s' represents the next state, and done is the task completion flag (done=1 indicates the task has ended, done=0 indicates the task continues).

[0043] The priority experience replay mechanism allocates sampling weights based on the TD error (temporal difference error) of the samples. Samples with larger TD errors have higher sampling probabilities, thus improving training efficiency. The formula for calculating TD error is as follows:

[0044] Where γ is the discount factor (0 < γ < 1). The value assessment value of the current state-action pair. The next state-optimal action pair is the value assessment value output by the target value network. The optimal action for the next state output by the target policy network.

[0045] S3. Define the state space, action space, and reward function for the agent's reinforcement learning. The state space integrates the vehicle's own state, the environment's state, and the predicted path state. The action space is designed based on vehicle dynamics constraints. The reward function adopts a multi-objective weighted composite reward function designed based on traffic rule constraints, quantifying the path's safety, efficiency, smoothness, and rule compliance. Specifically: S31. Definition of state space s: The state space is a high-dimensional continuous space that integrates the vehicle's own state, the environment's state, and the path prediction state. The specific expression is as follows:

[0046] in, Let θ be the vehicle's current speed, θ be the heading angle, (x,y) be the vehicle's position coordinates, δ be the steering angle, and a be the acceleration. , () represents the coordinates of the obstacle's position. Let θ be the speed of the obstacle and θ be the heading angle of the obstacle. , This refers to the width and height of the obstacle, i.e., the size information of the obstacle; , ) represents the center coordinates of the lane line, i.e., the position of the lane line, and sig represents the traffic signal status (sig=0 indicates a red light, sig=1 indicates a green light, and sig=2 indicates a yellow light). Speed ​​limits for roads; , () represents the coordinates of the navigation destination. This represents the straight-line distance from the vehicle's current location to its destination.

[0047] S32. Definition of action space u: The action space is a continuous space, designed based on vehicle dynamics constraints. The outputs are the steering angle increment Δδ and the acceleration increment Δa, with the specific expressions as follows:

[0048] The range of values ​​for the steering angle increment Δδ is as follows: The range of values ​​for the acceleration increment Δa is: Ensure that the action vector u output satisfies the vehicle dynamics constraints; the update formulas for the vehicle's current steering angle δ and acceleration a are:

[0049]

[0050] S33. Definition of reward function r: A multi-objective weighted composite reward function is adopted, comprehensively considering the path's safety, efficiency, smoothness, and rule compliance, to calculate the reward value obtained for the current action. The specific expression is as follows:

[0051] in, , , , , Let be the weight coefficients of each sub-reward function, and satisfy . .

[0052] Safety rewards are awarded to assess the safety of a path, based on the distance between the vehicle and obstacles. The calculation is based on the vehicle's own condition, and the specific formula is as follows:

[0053] in, , The security reward coefficient is (k2>k1>0). The distance between the vehicle and the nearest obstacle. To preset a safe distance; when At that time, the security reward decreases as the distance decreases. At that time, a fixed positive reward is given to encourage the intelligent agent to maintain a safe distance.

[0054] As a performance-based reward, it is used to evaluate the driving efficiency of a route, based on the vehicle's current speed v and the road speed limit. The formula for calculating the difference is as follows:

[0055] in, For high efficiency reward coefficient ( >0); when the vehicle speed is close to the road speed limit, the efficiency reward value is the maximum, encouraging the agent to maintain high driving efficiency under the premise of safety.

[0056] The smoothness reward, used to evaluate the driving comfort of the path, is calculated based on the steering angle increment Δδ and acceleration increment Δa, using the following formula:

[0057] in, , For smoothness reward coefficient ( >0, >0); the smaller the increment of steering angle and acceleration, the greater the smoothness reward value, which encourages the agent to output smooth movements and improves path smoothness; This is a rule compliance reward, used to assess whether a route complies with traffic rule constraints. The specific formula is:

[0058] in, , The reward coefficient for rule compliance ( >0, >0); When a vehicle violates traffic rules (such as speeding, crossing the line, running a red light, or failing to yield to an obstacle), a negative reward is given, and the violation is punished; The reward is a goal-oriented reward used to guide the agent to travel towards the destination, based on the straight-line distance from the vehicle's current location to the destination. The formula for calculating the change is as follows:

[0059] in, Goal-oriented reward coefficient ( >0), Let t be the straight-line distance from the vehicle to its destination. The straight-line distance from the vehicle to the destination is t+1. When the vehicle approaches the destination, a positive reward is given to encourage the agent to reach the destination quickly.

[0060] S4. Train the path decision-making agent. Through continuous interaction between the environment model and the agent, generate experience samples and store them in the experience replay module. Use an alternating training strategy to train the policy network module and the value network module, introducing an exploration rate decay mechanism and a gradient pruning mechanism until the agent converges. Figure 4 As shown, specifically: S41. Initialize parameters: Initialize the parameters of the policy network, value network, target policy network, and target value network, where the initial parameters of the target policy network and target value network are consistent with the initial parameters of the policy network and value network, respectively; initialize the capacity and exploration rate of the experience replay module. (Initial exploration rate) (∈[0.9,1.0]), exploration rate decay coefficient λ (λ∈[0.995,0.999]), minimum exploration rate (ε∈[0.01,0.1]), discount factor γ, learning rate η, batch size and number of training iterations T; S42. Generate experience samples: The agent interacts within the environment model. At time t, the state perception module acquires the current state s, encodes it through the state encoding module, and inputs it into the policy network to generate action u. An ε-greedy policy is used to select actions: when the random number rand() < ε, action u is randomly selected (exploration behavior); when rand() ≥ ε, the action u output by the policy network is selected (exploitation behavior). Action u is input into the environment interaction module to update the environment state, obtain the next state s and the corresponding reward value r, determine whether the task is complete, and obtain "done". Experience samples ( , , , Stored in the experience playback module; S43. Experience Sampling and Network Updates: When the number of samples in the experience replay module reaches a preset threshold, batch_size experience samples are sampled from the experience replay module according to the priority experience replay mechanism; based on the sampled samples, the loss function L of the value network and the loss function L of the policy network are calculated. The loss function for the value network uses the mean squared error loss, and the calculation formula is as follows:

[0061] in, For the first The value assessment of state-action pairs in a sample of experience. The output of the target policy network is the first The next state of an empirical sample The optimal action, The first output of the target value network The value assessment of the next state-optimal action pair for each empirical sample.

[0062] The loss function of the policy network is the negative expectation of the output value of the value network, and the calculation formula is as follows:

[0063] in, In the i-th sample output by the current value network, the current state Actions output by the policy network The appraised value.

[0064] The stochastic gradient descent (SGD) algorithm is used to update the parameters of the value network and the policy network respectively. At the same time, a gradient pruning mechanism is introduced to limit the gradient norm within a preset range to avoid gradient explosion. S44, Target network soft update: A soft update strategy is used to update the parameters of the target value network and the target policy network. The soft update formula is as follows:

[0065]

[0066] in, , These are the parameters for the value network and the policy network, respectively. , τ represents the parameters of the target value network and the target policy network, respectively, and τ is the soft update coefficient (τ∈[0.001,0.01]).

[0067] S45, Exploration Rate Decrease: After each iteration, the exploration rate ε is updated to... ,when ≤ At that time, keep ε= Reduce exploration behavior and increase utilization behavior.

[0068] S46. Determining convergence: Repeat steps S42-S45 until the number of training iterations reaches the preset number T, or the average reward value of the agent remains stable for a preset number of rounds (e.g., 50 rounds) with a fluctuation range of ≤5%. If the training of the agent is considered to have converged, training is stopped, and the trained policy network and value network parameters are saved. Otherwise, iterative training continues.

[0069] S5. Deploy the converged path decision agent to the autonomous vehicle, acquire the vehicle's own state and the environment state in real time, output the optimal path decision action through the agent to generate the driving path, and adjust the path in real time according to the dynamic changes in the environment to achieve adaptive path planning, specifically: S51, Real-time Perception: During the operation of an autonomous vehicle, the state perception module acquires the vehicle's own state, environmental state, and navigation destination information in real time. It then fuses the multi-source perception data using a Kalman filter algorithm to remove noise and obtain accurate state information s. S52, Path Decision: The state information s is input into the path decision agent after training and convergence. After being encoded by the state encoding module, the policy network outputs the optimal path decision action u=[Δδ, Δa], ensuring that the action satisfies vehicle dynamics constraints and traffic rule constraints. S53, Path Generation: Based on the path decision action u, combined with the vehicle dynamics model, the vehicle's steering angle δ and acceleration a are updated to generate a sequence of driving path points. The B-spline interpolation algorithm is used to smooth the path point sequence to obtain a continuous and smooth driving path. S54, Dynamic Adjustment: The environmental interaction module monitors changes in the environmental state in real time (such as sudden obstacles, changes in traffic signals, and lane changes). When the environmental state changes abruptly, it reacquires the current state s, and the agent re-outputs the optimal path decision action to adjust the driving path and ensure the safety and adaptability of the path. S55. Destination Arrival Determination: Real-time calculation of the straight-line distance from the vehicle's current location to its destination ,when If the distance is less than or equal to a preset threshold (e.g., 1m), the vehicle is considered to have reached its destination, and path planning is stopped; otherwise, steps S51-S54 are repeated to continue adaptive path planning.

[0070] In summary, the method in this embodiment improves the smoothness of the path, avoids the problems of frequent changes in direction and excessive steering angles, and enhances driving comfort. At the same time, it can adjust the path in real time according to dynamic changes in the environment, and has a rapid response capability to sudden obstacles, traffic signal changes, etc., which significantly improves the safety and reliability of autonomous driving path planning.

[0071] Example 2 One embodiment of the present invention provides an autonomous driving path planning system based on reinforcement learning, comprising: The environment model building module is configured to: build an autonomous driving path planning environment model to obtain the autonomous vehicle's own state, environmental state and path prediction state, and provide an interaction interface between the agent and the environment, while loading vehicle dynamics constraints and traffic rule constraints. The agent construction module is configured to: construct a path decision agent, which includes a state encoding module, a policy network module, a value network module and an experience playback module. The state encoding module is used to encode high-dimensional data in the state space, the policy network module is used to output path decision actions, the value network module is used to evaluate the value of the actions, and the experience playback module is used to store experience samples generated by the interaction between the agent and the environment model. The reinforcement learning definition module is configured to define the state space, action space, and reward function of the agent's reinforcement learning. The state space integrates the vehicle's own state, the environment's state, and the path prediction state. The action space is designed based on vehicle dynamics constraints. The reward function adopts a multi-objective weighted composite reward function designed based on traffic rule constraints. The agent training module is configured to: train the path decision agent, generate experience samples and store them in the experience playback module through continuous interaction between the environment model and the agent, train the policy network module and the value network module using an alternating training strategy, and introduce an exploration rate decay mechanism and a gradient pruning mechanism until the agent converges. The deployment planning module is configured to: deploy the trained and converged path decision agent to the autonomous vehicle, acquire the vehicle's own state and the environment state in real time, output the optimal path decision action through the agent, generate the driving path, and adjust the path in real time according to the dynamic changes of the environment to achieve adaptive path planning.

[0072] Example 3 One embodiment of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned reinforcement learning-based autonomous driving path planning method.

[0073] Example 4 In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the aforementioned reinforcement learning-based autonomous driving path planning method.

[0074] Example 5 One embodiment of the present invention provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the reinforcement learning-based autonomous driving path planning method.

[0075] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0077] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for autonomous driving path planning based on reinforcement learning, characterized in that, include: Construct an autonomous driving path planning environment model to obtain the autonomous vehicle's own state, environmental state, and path prediction state, and provide an interaction interface between the agent and the environment, while loading vehicle dynamics constraints and traffic rule constraints. A path decision-making agent is constructed, comprising a state encoding module, a policy network module, a value network module, and an experience playback module. The state encoding module is used to encode high-dimensional data in the state space, the policy network module is used to output path decision actions, the value network module is used to evaluate the value of the actions, and the experience playback module is used to store experience samples generated by the interaction between the agent and the environment model. Define the state space, action space, and reward function for agent reinforcement learning. The state space integrates the vehicle's own state, the environment's state, and the path prediction state. The action space is designed based on vehicle dynamics constraints. The reward function adopts a multi-objective weighted composite reward function designed based on traffic rule constraints. The path decision-making agent is trained by continuously interacting with the environment model and the agent to generate experience samples and store them in the experience playback module. The policy network module and value network module are trained by an alternating training strategy. An exploration rate decay mechanism and a gradient pruning mechanism are introduced until the agent converges. The trained and converged path decision agent is deployed to autonomous vehicles to obtain the vehicle's own state and the environment state in real time. The agent outputs the optimal path decision action to generate a driving path and adjusts the path in real time according to the dynamic changes in the environment to achieve adaptive path planning.

2. The autonomous driving path planning method based on reinforcement learning as described in claim 1, characterized in that, The environment model includes a state perception module, an environment interaction module, and a constraint condition module; The state perception module integrates data from lidar, camera, millimeter-wave radar, GPS / BeiDou navigation system and vehicle CAN bus to acquire multi-source perception information, and fuses the multi-source perception information through Kalman filtering algorithm; The environmental interaction module establishes a mapping relationship between the agent's actions and the environmental state, and uses a simplified two-degree-of-freedom vehicle dynamics model to update the vehicle state and the environmental state. The vehicle dynamics constraints loaded by the constraint module include maximum steering angle, maximum steering angular velocity, maximum vehicle speed, and maximum acceleration constraints, while the traffic rule constraints include lane driving, speed limits, yielding, and traffic signal constraints.

3. The autonomous driving path planning method based on reinforcement learning as described in claim 2, characterized in that, The expression for the two-degree-of-freedom vehicle dynamics model is: Where v is the vehicle speed, a is the acceleration, f is the rolling resistance coefficient, ρ is the air density, C is the air resistance coefficient, A is the vehicle's frontal area, m is the vehicle's mass, L is the vehicle's wheelbase, δ is the steering angle, θ is the heading angle, and (x,y) are the vehicle's position coordinates.

4. The autonomous driving path planning method based on reinforcement learning as described in claim 1, characterized in that, The state encoding module uses a structure combining a convolutional neural network (CNN) and a multilayer perceptron (MLP) to encode high-dimensional data in the state space. The policy network module uses an Actor network based on the Deep Deterministic Policy Gradient (DDPG) algorithm to output continuous path decision actions. The value network module uses the DDPG algorithm's Critic network to output the value assessment value of state-action pairs; The experience replay module adopts the priority experience replay (PER) mechanism, which allocates sampling weights based on the TD error of the samples.

5. The autonomous driving path planning method based on reinforcement learning as described in claim 1, characterized in that, The expression for the state space s is: in, Let θ be the vehicle's current speed, θ be the heading angle, (x,y) be the vehicle's position coordinates, δ be the steering angle, and a be the acceleration. , () represents the coordinates of the obstacle's position. Let θ be the speed of the obstacle and θ be the heading angle of the obstacle. , The width and height of the obstacle; , () represents the coordinates of the lane center, and sig represents the traffic signal status. Speed ​​limits for roads; , () represents the coordinates of the navigation destination. This represents the straight-line distance from the vehicle's current location to its destination.

6. The autonomous driving path planning method based on reinforcement learning as described in claim 1, characterized in that, The expression for the action space u is: u=[Δδ, Δa] Where Δδ is the steering angle increment and Δa is the acceleration increment, both of which satisfy vehicle dynamics constraints.

7. The autonomous driving path planning method based on reinforcement learning as described in claim 1, characterized in that, The expression for the multi-objective weighted composite reward function is: Where w1, w2, w3, w4, and w5 are the weight coefficients of each sub-reward function, and satisfy w1 + w2 + w3 + w4 + w5 = 1; As a security reward, Rewards for high efficiency For smooth reward, The reward for compliance with traffic rules is calculated based on traffic rule constraints. Rewards are based on goals.

8. An autonomous driving path planning system based on reinforcement learning, characterized in that, include: The environment model building module is configured to: build an autonomous driving path planning environment model to obtain the autonomous vehicle's own state, environmental state and path prediction state, and provide an interaction interface between the agent and the environment, while loading vehicle dynamics constraints and traffic rule constraints. The agent construction module is configured to: construct a path decision agent, which includes a state encoding module, a policy network module, a value network module and an experience playback module. The state encoding module is used to encode high-dimensional data in the state space, the policy network module is used to output path decision actions, the value network module is used to evaluate the value of the actions, and the experience playback module is used to store experience samples generated by the interaction between the agent and the environment model. The reinforcement learning definition module is configured to define the state space, action space, and reward function of the agent's reinforcement learning. The state space integrates the vehicle's own state, the environment's state, and the path prediction state. The action space is designed based on vehicle dynamics constraints. The reward function adopts a multi-objective weighted composite reward function designed based on traffic rule constraints. The agent training module is configured to: train the path decision agent, generate experience samples and store them in the experience playback module through continuous interaction between the environment model and the agent, train the policy network module and the value network module using an alternating training strategy, and introduce an exploration rate decay mechanism and a gradient pruning mechanism until the agent converges. The deployment planning module is configured to: deploy the trained and converged path decision agent to the autonomous vehicle, acquire the vehicle's own state and the environment state in real time, output the optimal path decision action through the agent, generate the driving path, and adjust the path in real time according to the dynamic changes in the environment to achieve adaptive path planning.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement an autonomous driving path planning method based on reinforcement learning as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform an autonomous driving path planning method based on reinforcement learning as described in any one of claims 1-7.

Images