Track guiding reinforcement learning method for automatic driving racing car

Abstract

The invention relates to a track guide reinforcement learning method for an automatic driving racing car, and the method comprises the steps: simulating a racing track environment, and determining a motion model for describing the dynamic behaviors of an intelligent agent; setting an action space and a state space of the intelligent agent; establishing a trajectory-guided reinforcement learning architecture by using a TD3 algorithm, wherein the trajectory-guided reinforcement learning architecture has an evaluation network and an execution network; the uncertainty of the friction coefficient is introduced into the trajectory-guided reinforcement learning architecture to perform cyclic training on the trajectory-guided reinforcement learning architecture, so that the intelligent agent gradually learns how to track the trajectory through interaction with the environment and adapts to different friction coefficient conditions; and optimizing parameters of the evaluation network and the execution network of the trajectory-guided reinforcement learning architecture.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle autonomous driving, and in particular to a trajectory guidance reinforcement learning method for autonomous driving racing cars. Background Art

[0002] In recent years, with the continuous advancement of artificial intelligence and sensor technology, autonomous driving technology has been significantly developed and widely used. Autonomous driving racing is an attempt to apply it to motorsports. Autonomous driving racing refers to racing cars controlled by an autonomous driving system without human control. The field of autonomous driving racing is developing rapidly, bringing new experiences and challenges to motorsports. It is of great significance in promoting the development of autonomous driving technology, improving driving safety and reliability, and promoting the optimization of transportation systems.

[0003] The existing high-speed autonomous driving control method for racing cars has the following defects:

[0004] Strong dependence on model parameters: Although the widely used model predictive control method can theoretically provide the optimal solution for trajectory planning and tracking, its performance is highly dependent on the accuracy of the vehicle dynamics model. Especially in the scenario of unmanned driving racing, the nonlinear characteristics of lateral tire force are difficult to accurately capture through traditional modeling methods. The resulting model error will have a significant impact on the control effect, and then affect the stability and reliability of the system.

[0005] Limitations of coping with model mismatch: In actual racing environments, the change in vehicle tire friction coefficient due to tire wear often leads to a large mismatch between the system model and the actual situation. Since traditional control methods such as model predictive control methods rely on model accuracy, this mismatch can significantly degrade controller performance, especially when approaching the vehicle stability limit.

[0006] Limitation of generalization ability: Existing control architectures based on end-to-end reinforcement learning can achieve good performance in specific environments, but their generalization ability is limited when faced with new tracks or unseen scenes. This lack of cross-scenario migration capability seriously restricts the practical application potential of this type of method. Summary of the invention

[0007] In order to solve at least some of the above problems in the prior art, the present invention provides a trajectory guided reinforcement learning method for an autonomous driving racing car, comprising:

[0008] Simulate the track environment and determine the motion model that describes the dynamic behavior of the agent;

[0009] Set up the agent’s action space and state space;

[0010] Using the TD3 algorithm to establish a trajectory-guided reinforcement learning architecture, where the trajectory-guided reinforcement learning architecture has an evaluation network and an execution network;

[0011] Introducing uncertainty in the friction coefficient into the trajectory-guided reinforcement learning framework;

[0012] Recurrent training of the trajectory-guided reinforcement learning architecture; and

[0013] Optimize the parameters of the evaluation network and the execution network of the trajectory-guided reinforcement learning architecture.

[0014] Furthermore, the F1TENTH track environment is used, and a single-track model is adopted to describe the dynamic behavior of the agent. The vehicle kinematic model in the single-track model is used to describe the displacement and posture changes of the agent, including the calculation of the lateral and longitudinal position change rates:

[0015]

[0016] Among them, v is the longitudinal velocity of the vehicle, ψ is the heading angle, and β is the sideslip angle.

[0017] The vehicle dynamics model expressions in the single track model include:

[0018] Longitudinal acceleration:

[0019]

[0020] Among them, x4 represents the current vehicle speed, and u2 is the longitudinal thrust control input;

[0021] Heading angular acceleration:

[0022]

[0023] Where μ represents the road friction coefficient, m is the vehicle mass, and I z is the moment of inertia about the vehicle's center of mass, l f and l r are the distances from the front axle and rear axle to the center of mass of the vehicle, C S,f and C S,r are the cornering stiffness of the front and rear wheels, h c,g represents the height of the center of mass, δ is the steering angle; and

[0024] Slip angular velocity:

[0025]

[0026] Furthermore, setting the action space and state space of the agent includes:

[0027] State space s t :

[0028] s t ={LiDAR scans,x,y,ψ,v,ω}, where LiDAR scans is the LiDAR scan data, x,y represents the position information of the intelligent body, and v represents the longitudinal velocity;

[0029] Action Space a t :a t ={δ,v}, where the range of the steering angle δ is [-1,1], the maximum steering angle = 0.4189 radians, and the range of the speed v is [-v max ,v max ], where v max =8m / s.

[0030] Further, the TD3 algorithm is used to establish a trajectory-guided reinforcement learning architecture, wherein the trajectory-guided reinforcement learning architecture has an evaluation network and an execution network including:

[0031] Initialize the evaluation network and the execution network. The TD3 algorithm includes two evaluation networks Q1 and Q2 and an execution network π. The initialization parameters are as follows:

[0032] set up To evaluate the parameters of networks Q1 and Q2 and the execution network π, the parameters are randomly initialized;

[0033] Set a target network for each of the two evaluation networks and one execution network, and initialize the parameters of the target network:

[0034] Evaluate the network parameters for two objectives and enforce the network parameters for one objective;

[0035] Initialize the experience replay buffer The experience replay buffer is used to store the four-tuple of the interaction between the agent and the environment, and the four-tuple of the environment interaction includes state, action, reward and next state;

[0036] Initialize a planner based on a pure pursuit algorithm, which is used to calculate the ideal trajectory action for the current state at each time step, including the reference speed v ref and steering angle δ ref ;

[0037] According to the current position of the agent, select the target point along the preview distance in front of the trajectory;

[0038] After determining the target point, the planner calculates the curvature of an arc passing through the agent's current position and the target point, denoted as κ;

[0039] Calculate the steering angle δ ref :

[0040]

[0041] Among them, L represents the wheelbase of the vehicle, that is, the distance between the front and rear wheels. f represents the lateral distance from the target point to the center of mass of the vehicle, and α is the yaw angle of the vehicle;

[0042] Adjust the reference speed v according to the curvature of the predetermined trajectory ref .

[0043] Furthermore, the uncertainty of the friction coefficient introduced into the trajectory-guided reinforcement learning framework includes:

[0044] Initialize the friction coefficient with Gaussian noise: Let the friction coefficient μ be the friction between the vehicle and the road, the initial setting value is μ0, and randomize the friction coefficient at the beginning of each training round:

[0045]

[0046] Where ∈ is Gaussian noise with mean 0 and variance σ 2 Normal distribution of

[0047] At each time step t, the friction coefficient is dynamically and randomly updated to simulate different road friction conditions.

[0048] Furthermore, the loop training of the trajectory guided reinforcement learning architecture includes:

[0049] The following steps are performed in each training epoch:

[0050] At each time step t, the reference velocity v of the reference action is generated using the planner ref and steering angle δ ref ;

[0051] Use the current state s to generate the optimal action a through the execution network π(s), and superimpose Gaussian noise ∈ on the output of the execution network:

[0052]

[0053] The standard deviation σ of the noise term ∈ controls the randomness of the action;

[0054] After the agent performs action a, the environment returns a new state s′ and an immediate reward r:

[0055] Based on the agent's actual speed v agent and steering angle δ agent With reference speed v ref and steering angle δ ref The deviation between them, calculate the reward:

[0056]

[0057] Store the four-tuple (s, a, r, s′) consisting of state s, action a, reward r and next state s′ into the replay buffer middle.

[0058] Furthermore, if the agent collides with any obstacle or goes off track, it will receive a reward of -1;

[0059] When no collision occurs, the reward is determined by the consistency between the agent's action and the planner's reference action: the actual velocity v agent and steering angle δ agent The reward is calculated based on the difference between the value and the reference value provided by the planner;

[0060] In the absence of collisions, the reward is scaled by 0.2 and capped above 0 to ensure a balanced reward structure.

[0061] Furthermore, the optimization of the parameters of the evaluation network and the execution network of the trajectory guided reinforcement learning architecture includes:

[0062] Calculate the target Q value through mini-batch training; and

[0063] Update the parameters of the evaluation network and the execution network based on the target Q value.

[0064] Furthermore, calculating the target Q value through mini-batch training includes:

[0065] When the experience replay buffer When the number of samples in reaches the mini-batch size N, N samples are randomly sampled from the buffer, each of which consists of state, action, reward and next state, i.e. (s, a, r, s′);

[0066] In TD3, the goal execution network π′ is used to generate the target action a′:

[0067] a′=π′(s′)+∈,∈~clip(N(0,σ),-c,c)

[0068] Among them, π′(s′) is the predicted action of the target execution network for the next state s′, and ∈ is a normal distribution. The noise term is clipped to the range [-c,c];

[0069] After obtaining the target action a′, the target evaluation network Q′1 and Q′2 are used to calculate the target Q value y, and the calculation formula is:

[0070] y=r+γ·min(Q′1(s′,a′),Q′2(s′,a′))

[0071] Where r is the immediate reward, γ is the discount factor, and Q′1(s′,a′) and Q′2(s′,a′) are the value estimates of the target evaluation network for the next state s′ and target action a′.

[0072] Further, updating the parameters of the evaluation network and the execution network based on the target Q value includes:

[0073] The targets of the evaluation networks Q1 and Q2 are updated by minimizing the following loss function to make their estimated values ​​close to the target Q value y:

[0074]

[0075] Where i = 1, is the parameter of the evaluation network Q1, the loss function (yQ i (s,a)) 2 represents the square error between the target Q value y and the estimated value of the evaluation network Q1 for the current state s and action a; when i = 2, is the parameter of the evaluation network Q2; the loss function (yQ i (s,a)) 2 represents the squared error between the target Q value y and the estimate of the current state s and action a by the critic network Q2; and

[0076] The execution network is updated once every d time steps. The update formula is as follows:

[0077]

[0078] Among them, θ π are the parameters of the execution network, α is the learning rate, Q1(s,π(s)) is the value estimate of the evaluation network Q1 for the execution network output action π(s) in state s, represents the gradient of the evaluation network with respect to the output of the execution network;

[0079] At each time step, the target evaluation networks Q′1 and Q′2 and the target execution network π′ are soft updated so that the parameters of the target network gradually follow the parameter changes of the current network:

[0080]

[0081] θ′ π ←τθ π +(1-τ)θ′ π

[0082] in, and θ′ π are the parameters of the target evaluation network and the target execution network respectively, and τ is the soft update coefficient, which controls the rate at which the target network parameters follow the current network.

[0083] The present invention has at least the following beneficial effects: the trajectory-guided reinforcement learning architecture of the present invention combines the high efficiency of the classical planning method with the adaptability of reinforcement learning, which not only improves the performance of the racing control system under extreme conditions, but also ensures stability and safety under variable friction and road conditions.

[0084] The present invention uses a domain randomization strategy to enable the reinforcement learning agent to cope with diverse environmental variables, thereby enhancing its adaptability and performance in new tracks and different environments.

[0085] The present invention utilizes the accuracy of classical path planning and the adaptability of reinforcement learning to optimize the real-time performance on resource-constrained hardware platforms, and can simultaneously improve the accuracy and efficiency of trajectory tracking and speed control. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] In order to further clarify the above and other advantages and features of various embodiments of the present invention, a more specific description of various embodiments of the present invention will be presented with reference to the accompanying drawings. It will be understood that these drawings only depict typical embodiments of the present invention and are therefore not to be considered as limiting the scope thereof.

[0087] Figure 1 A schematic diagram showing the logical relationship and working principle between various parts of the trajectory-guided reinforcement learning method for an autonomous driving racing car of the present invention.

[0088] Figure 2 A schematic diagram of single-track model building according to the present invention is shown.

[0089] Figure 3 A schematic diagram of optimizing the actions of an intelligent agent using a reward function according to the present invention is shown.

[0090] Figure 4 A computer system is shown that implements the system and / or method according to the present invention. DETAILED DESCRIPTION

[0091] It should be noted that various components in the various drawings may be shown exaggeratedly for illustrative purposes and are not necessarily correct to scale.

[0092] In the present invention, each embodiment is only intended to illustrate the aspects of the present invention and should not be construed as limiting.

[0093] In the present invention, unless otherwise specified, the quantifiers "a", "an" and "an" do not exclude the presence of a plurality of elements.

[0094] It should also be pointed out that in the embodiments of the present invention, for the sake of clarity and simplicity, only a portion of the parts or components may be shown, but a person of ordinary skill in the art will understand that under the teachings of the present invention, required parts or components may be added as needed in specific scenarios.

[0095] It should also be pointed out that within the scope of the present invention, the terms "same", "equal", "equal to" and the like do not mean that the two values ​​are absolutely equal, but allow a certain reasonable error, that is, the terms also cover "substantially the same", "substantially equal", and "substantially equal to".

[0096] It should also be pointed out that in the description of the present invention, the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than explicitly or implicitly indicating that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as explicitly or implicitly indicating relative importance.

[0097] In addition, the numbering of the steps of the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps can be executed in different orders.

[0098] The trajectory guidance reinforcement learning method for autonomous driving racing provided by the present invention can solve the following problems:

[0099] 1. Alleviate the model mismatch problem caused by the uncertainty of the dynamic model: In the racing control of autonomous driving, model predictive control (MPC) is highly dependent on the accuracy of the vehicle dynamics model. However, due to changes in tire wear and road conditions, errors may occur in the parameters of the dynamic model, affecting the control effect. The present invention uses adaptive reinforcement learning to optimize the accuracy of the dynamic model through the adaptive adjustment ability of the reinforcement learning agent, thereby effectively alleviating the model mismatch problem and significantly improving the stability and reliability of the system in a changing environment.

[0100] 2. Enhance the track adaptability of existing end-to-end reinforcement learning methods: Traditional end-to-end reinforcement learning performs poorly when faced with new untrained tracks. This invention significantly improves the generalization ability of the agent under different tracks and environmental conditions by combining domain randomization and trajectory-based planning guidance technology, enabling it to maintain efficient racing performance in unknown scenarios.

[0101] Figure 1A schematic diagram showing the logical relationship and working principle between various parts of the trajectory-guided reinforcement learning method for an autonomous driving racing car of the present invention. Figure 2 A schematic diagram of single-track model building according to the present invention is shown. Figure 3 A schematic diagram of optimizing the actions of an intelligent agent using a reward function according to the present invention is shown.

[0102] A trajectory guided reinforcement learning method for autonomous driving racing includes the following steps:

[0103] Step 1: Build a training environment.

[0104] F1TENTH is used to simulate the track environment. The simulation model of the car (agent) adopts a single track model to describe the dynamic behavior of the car (agent) on the track. The single track model is modeled as follows: Figure 2 shown.

[0105] The vehicle kinematic model expression in the single track model is:

[0106] The vehicle kinematics model is used to describe the displacement and posture changes of the racing car, mainly including the calculation of the lateral and longitudinal position change rates:

[0107]

[0108] Among them, v is the longitudinal velocity of the vehicle, ψ is the heading angle, and β is the sideslip angle.

[0109] The vehicle dynamics model expression in the single track model is:

[0110] The dynamics model describes how the car responds to forces, mainly including changes in longitudinal acceleration and heading angle.

[0111] Longitudinal acceleration:

[0112]

[0113] Among them, x4 represents the current speed of the vehicle, and u2 is the longitudinal thrust control input. This formula enables the car to flexibly accelerate or decelerate on different sections of the race (such as straights and curves) by controlling the longitudinal thrust.

[0114] Heading angular acceleration:

[0115]

[0116] Where μ represents the road friction coefficient; m is the vehicle mass; I z is the moment of inertia about the vehicle's center of mass; l f and l r are the distances from the front axle and rear axle to the center of mass of the vehicle; C S,f and C S,rare the cornering stiffness of the front and rear wheels respectively; h c,g represents the height of the center of mass. δ is the steering angle.

[0117] Side slip angular velocity:

[0118]

[0119] Step 2: Set up the action space and state space of the car (agent).

[0120] According to the environmental model, the state space of the car at the current time t is designed as:

[0121] State space s t : Defined as the observation information of the car, including LiDAR scan data, position information, heading angle, longitudinal speed and yaw rate:

[0122] s t ={LiDAR scans,x,y,ψ,v,ω}

[0123] Action Space a t : Contains two control variables: Steering angle δ: in the range [-1,1], where the maximum steering angle = 0.4189 radians.

[0124] a t ={δ,v}

[0125] Speed ​​v: Range [-v max ,v max ], where v max =8m / s.

[0126] Step 3: If Figure 1 As shown, a trajectory-guided reinforcement learning architecture is established using the TD3 algorithm, wherein the trajectory-guided reinforcement learning architecture has an evaluation network and an execution network, and the network is initialized.

[0127] 3.1 Initialize the evaluation network and execution network:

[0128] The TD3 algorithm consists of two evaluation networks Q1 and Q2 and an execution network π, which are used to estimate the value of state-action pairs and generate optimal strategies. The initialization parameters are as follows:

[0129] set up Parameters for evaluation networks Q1 and Q2 and an execution network π.

[0130] These parameters are randomly initialized to ensure that the network does not have any prior preferences and facilitates the exploration of multiple state-action pairs.

[0131] 3.2 Initialization of the target network: Set a target network for each network to provide a stable update benchmark. The initialization is as follows:

[0132]

[0133] The target network will follow the current network using a soft update method to avoid drastic fluctuations in parameters. The target network includes the target execution network π′, the target evaluation network Q′1 and Q′2.

[0134] The Actor Network is used to generate the actions of the agent in a given state, which is directly used for policy updates and outputs the best actions that the agent should perform in the current state to maximize the expected cumulative reward. The Actor Network is updated through the policy gradient method to optimize its parameters so that the actions it outputs get the highest Q value in the Critic Network. The Target Actor Network is used to calculate a more stable target action to assist in the calculation of the target Q value, providing a stable benchmark for generating target actions, which are used to calculate the target Q value, ensuring a more stable reference when updating the Critic Network.

[0135] 3.3 Initialize the experience replay buffer (Experience Replay Pool): The experience replay buffer is used to store the four-tuple (state, action, reward, next state) of the interaction between the agent and the environment, so as to facilitate the subsequent sampling of training data from historical experience and increase sample diversity.

[0136] 3.4 Set up the planner: Initialize the classic planner based on the pure pursuit algorithm. This planner is used to calculate the ideal trajectory action for the current state at each time step, including the reference speed v ref and steering angle δ ref , guiding the agent to better approach the optimal trajectory.

[0137] 3.4.1 Target point selection: Based on the current position of the vehicle (agent), a target point is selected at a certain distance ahead along the trajectory (called the "preview distance"). The preview distance can be dynamically adjusted according to the speed to ensure fast and smooth tracking.

[0138] 3.4.2 Curvature Calculation: After determining the target point, the planner calculates the curvature of an arc passing through the vehicle's current position and the target point, denoted as κ. The larger the curvature κ, the smaller the turning radius, reflecting the need for a sharper turn.

[0139] 3.4.3 Steering angle calculation: Steering angle δ ref Obtained by the following formula:

[0140]

[0141] Where: L represents the wheelbase of the vehicle, that is, the distance between the front and rear wheels; L f represents the lateral distance from the target point to the center of mass of the vehicle; α is the yaw angle of the vehicle (the angular difference between the vehicle's driving direction and the target point).

[0142] 3.4.4 Speed ​​adjustment: reference speed v ref Adapt the current path to the curvature of the planned trajectory. For example, on a curve with a large curvature, the planner will reduce v ref This prevents the vehicle from deviating from the track due to sharp turns. On straight sections, the planner will increase v ref To improve driving efficiency.

[0143] Step 4: Introducing uncertainty in the coefficient of friction

[0144] 4.1 Gaussian noise initialization of friction coefficient: Let the friction coefficient μ be the friction between the vehicle and the road, and the initial setting value is μ0 (for example, the friction coefficient of a standard dry road). In order to introduce uncertainty, the friction coefficient is randomized at the beginning of each training episode:

[0145]

[0146] Where ∈ is Gaussian noise with mean 0 and variance σ 2 This setting makes the environment of each training have different friction conditions, which increases the robustness of the agent.

[0147] 4.2 Dynamic update of friction coefficient: At each time step t, the friction coefficient is dynamically and randomly updated to simulate different road friction conditions. For example, the updated friction coefficient is:

[0148]

[0149] This design increases the uncertainty of friction, enabling the agent to cope with sudden changes in road conditions.

[0150] Step 5: Run a loop to train the trajectory-guided reinforcement learning architecture, performing the following steps in each training epoch:

[0151] In each training cycle T, the agent gradually learns how to follow the trajectory and adapt to different friction coefficient conditions by interacting with the environment. T represents the duration of a training round in time steps. A time step is the smallest unit of interaction between the agent and the environment. For example, at each time step t, the agent observes the current state s t , select action a t , perform actions and receive immediate rewards r t, and then enter the next state s t+1 A training cycle T contains multiple time steps t. If a complete training cycle lasts 100 steps, then T = 100, and each individual interaction is a time step t.

[0152] like Figure 1 and 3 As shown, the training steps are as follows:

[0153] 5.1 Generating reference actions

[0154] At each time step t, the Pure Pursuit planner is used to generate the reference velocity v of the reference action ref and steering angle δ ref : Reference speed v {ref} : The ideal speed calculated based on the relative position of the vehicle to the track, which can ensure that the vehicle remains stable and effectively approaches the track. Reference steering angle δ {ref} : The planner determines the ideal steering angle of the vehicle through curvature calculation to ensure that the vehicle's driving direction gradually approaches the trajectory.

[0155] 5.2 Agent Action Selection

[0156] The agent uses the current state s to generate the optimal action through the execution network π(s). In order to increase the exploration, Gaussian noise ∈ is superimposed on the output of the execution network:

[0157]

[0158] The standard deviation σ of the noise term ∈ controls the randomness of the actions so that the agent can explore more state-action pairs.

[0159] 5.3 Environment Interaction and Reward Calculation

[0160] After the agent performs action a, the environment returns a new state s′ and an immediate reward r:

[0161] Based on the agent's actual speed v agent and steering angle δ agent With reference speed v ref and steering angle δ ref The deviation between them is used to calculate the reward:

[0162]

[0163] The purpose of the reward is to encourage the agent to behave closer to the reference trajectory. If the agent collides with any obstacle or goes off track, it will receive a reward of -1, which is a significant penalty to strongly discourage such serious failures in autonomous driving tasks. When no collision occurs, the reward is determined by the consistency between the agent's action and the planner's reference action: the velocity v agent and steering angle δ agent The reward is calculated as the difference between its behavior and the reference value provided by the planner. This design incentivizes the agent to minimize the difference between its behavior and the planner's reference action. The closer the agent's speed and steering angle are to the planner's suggestions, the higher the reward. Conversely, significant deviations result in lower rewards, encouraging compliance with the planner's guidance. In addition, in the absence of collisions, the reward is scaled by 0.2 and capped above 0 to ensure a balanced reward structure.

[0164] 5.4 Storage Experience Samples

[0165] Store the four-tuple (s, a, r, s′) consisting of state s, action a, reward r and next state s′ into the replay buffer The historical data in the buffer can be used for training batch sampling to increase the independence and diversity of samples.

[0166] like Figure 1 As shown, the following are the steps to optimize the trajectory guided reinforcement learning architecture:

[0167] Step 6: Mini-batch training.

[0168] In the TD3 algorithm, the goal of mini-batch training is to gradually optimize the parameters of the evaluation network and the execution network through sampling and updating, so that the agent can generate high-value strategies while reducing the overestimation bias in value estimation. The following are the steps in mini-batch training:

[0169] 6.1 Sampling Data

[0170] When the experience replay buffer When the number of samples in reaches the mini-batch size N, N samples are randomly sampled from the buffer. Each sample consists of a state, an action, a reward, and a next state, i.e. (s, a, r, s′). The advantage of random sampling is that it reduces the temporal correlation between samples and ensures the diversity of sample data distribution. This is crucial for the stable training of the evaluation network and the execution network in TD3. The sampled mini-batch data is used for subsequent network updates to improve the adaptability of network parameters.

[0171] 6.2 Target Action Calculation

[0172] In TD3, the target execution network π′ is used to generate the target action a′ and reduce the sharp fluctuation of the target action by reducing noise. The purpose of this is to maintain the stability of the target Q value during optimization and avoid overestimation bias during the update process. The specific formula is as follows:

[0173] a′=π′(s′)+∈,∈~clip(N(0,σ),-c,c)

[0174] Where: π′(s′) is the predicted action of the target execution network for the next state s′; ∈ is a normal distribution The noise term is clipped to the range [-c, c] to prevent excessive noise from interfering with the smoothness of the action. Reducing the noise ensures that the target action a′ does not change drastically, thereby improving the stability of the agent's strategy. This method is particularly suitable for trajectory tracking in environments with high uncertainty.

[0175] 6.3 Target Q value calculation

[0176] After obtaining the target action a′, the next step is to use the target evaluation network Q′1 and Q′2 to calculate the target Q value y, which is calculated as follows:

[0177] y=r+γ·min(Q′1(s′,a′),Q′2(s′,a′))

[0178] Where: r is the immediate reward; γ is the discount factor used to balance the immediate reward with the future reward; Q′1(s′,a′) and Q′2(s′,a′) are the value estimates of the target evaluation network for the next state s′ and the target action a′. The use of a dual evaluation network (i.e. taking the minimum value of Q′1(s′,a′) and Q′2(s′,a′)) can effectively reduce the overestimation bias and make the Q value estimate more accurate. The discount factor γ is used to balance the long-term and short-term benefits of the agent to ensure that the agent pays attention to the overall tracking effect of the trajectory.

[0179] Step 7: Network parameter update

[0180] After calculating the target Q value y in the mini-batch data, the TD3 algorithm updates the parameters of the evaluation network and the execution network in turn. The following are the detailed update steps for each network parameter:

[0181] 7.1 Parameter Update of Evaluation Network

[0182] The goal of updating the evaluation network Q1 and Q2 is to make its estimated value close to the target Q value y, so as to accurately reflect the actual value of the state-action pair. The specific update method is to minimize the following loss function:

[0183]

[0184] Where: when i = 1, is the parameter of the evaluation network Q1; the loss function (y-

[0185] Q i (s,a)) 2 represents the square error between the target Q value y and the estimated value of the evaluation network Q1 for the current state s and action a. Similarly, when i = 2, is the parameter of the evaluation network Q2; the loss function (yQ i (s,a)) 2 Represents the square of the error between the target Q value y and the estimated value of the evaluation network Q2 for the current state s and action a.

[0186] By minimizing this loss function, the parameters of the evaluation network can be adjusted so that its estimated value gradually approaches the true Q value, thereby ensuring that the evaluation network can provide accurate value reference when the network is subsequently updated.

[0187] 7.2 Executing Network Policy Updates

[0188] The execution network is updated once every d time steps. The goal of this update is to optimize the execution network parameters through the policy gradient method so that the action output by the execution network in a given state can maximize the value of the evaluation network. The update formula is as follows:

[0189]

[0190] Here: θ π are the parameters of the execution network; α is the learning rate, which controls the amplitude of parameter update; Q1(s,π(s)) is the value estimate of the evaluation network Q1 for executing the network output action π(s) in state s; Represents the gradient of the evaluation network with respect to the output of the execution network. Through this step, the output of the execution network can gradually approach the optimal action, thereby improving the ability of the intelligent agent to select high-value actions in different states.

[0191] 7.3 Soft Update of Target Network

[0192] In order to ensure the stability of training, the target network Q′1, Q′2, π′ is soft-updated at each time step so that the parameters of the target network gradually follow the parameters of the current network. The specific update method is as follows:

[0193]

[0194] θ′ π ←τθ π +(1-τ)θ′ π

[0195] in: and θ′ π are the parameters of the target evaluation network and the target execution network respectively; τ is the soft update coefficient, which controls the rate at which the target network parameters follow the current network. Soft update ensures that the target network will not suddenly change significantly, thereby improving the stability of the target Q value calculation.

[0196] 1. Compared with the traditional reinforcement learning training racing scenario, the method of the present invention takes into account the external disturbance of the tire friction coefficient and uses the domain randomization method to simulate the change of the tire friction coefficient. As described in step 4, the Gaussian noise of the friction coefficient is initialized: let the friction coefficient μ be the friction between the vehicle and the road surface, and the initial setting value is μ0 (for example, the friction coefficient of a standard dry road surface). In order to introduce uncertainty, the friction coefficient is randomized at the beginning of each episode:

[0197]

[0198] Where ∈ is Gaussian noise with mean 0 and variance σ 2 This setting makes the environment of each training have different friction conditions, which increases the robustness of the agent. Dynamic update of friction coefficient: At each time step t, the friction coefficient is dynamically and randomly updated to simulate different road friction conditions. For example, the updated friction coefficient is:

[0199]

[0200] This design increases the uncertainty of friction, enabling the agent to cope with sudden changes in road conditions.

[0201] 2. Compared with traditional optimization algorithms such as model predictive control that rely on modeling or traditional reinforcement learning training racing scenarios, in our framework, the planner is responsible for generating a set of guidance actions based on the vehicle posture information and trajectory, which serve as the reference actions of the reward function to calculate the final reward, and the controller specifically generates control inputs so that the system can obtain the optimal strategy through training. As described in step 3.4. The learning ability of reinforcement learning is used to heuristically deal with model mismatch and trajectory generalization, while the safety and reliability of traditional planning methods are used in a hybrid way.

[0202] 3. Compared with the traditional reinforcement learning training racing scenario, the setting of the reward function is only related to the lap time. The present invention improves the setting of the reward function, aiming to allow the agent to implicitly infer and train the optimal strategy. As described in step 5.3. After the agent performs action a, the environment returns a new state s′ and an immediate reward r: based on the actual speed v of the agent agent and steering angle δ agent With reference speed v ref and steering angle δ ref The deviation between them, calculate the reward:

[0203]

[0204] The purpose of the reward is to encourage the agent to behave closer to the reference trajectory. If the agent collides with any obstacle or goes off track, it will receive a reward of -1, which is a significant penalty to strongly discourage such serious failures in autonomous driving tasks. When no collision occurs, the reward is determined by the consistency between the agent's behavior and the planner's behavior: the speed v agent and steering angle δ agent The reward is calculated as the difference between the speed and the reference value provided by the planner. This design incentivizes the agent to minimize the difference between its behavior and the planner's suggestions. The closer the agent's speed and steering angle are to the planner's suggestions, the higher the reward. Conversely, significant deviations result in lower rewards, encouraging compliance with the planner's guidance. In addition, in the absence of collisions, the reward is scaled by 0.2 and capped above 0 to ensure a balanced reward structure.

[0205] The hybrid control architecture of the present invention combines the efficiency of classical planning methods with the adaptability of reinforcement learning, which not only improves the performance of the racing control system under extreme conditions, but also ensures stability and safety under variable friction and road conditions.

[0206] The present invention uses a domain randomization strategy to enable the reinforcement learning agent to cope with diverse environmental variables, thereby enhancing its adaptability and performance in new tracks and different environments.

[0207] The method of the present invention combines the accuracy of classical path planning with the adaptability of reinforcement learning: the present invention utilizes the accuracy of classical path planning and the adaptability of reinforcement learning to optimize the real-time performance on resource-constrained hardware platforms, while improving the accuracy and efficiency of trajectory tracking and speed control.

[0208] Autonomous driving racing in changing environmental conditions: One of the best use cases for this solution is in complex environments with large changes in friction coefficients (such as slippery or different material tracks). In this case, domain randomization and adaptive reinforcement learning allow the agent to adapt to the changing friction coefficient and maintain the stability and safety of the race.

[0209] Adaptability when faced with new tracks or unknown environments: This solution performs well on new tracks that are unknown or untrained. By combining domain randomization with the guidance of the planner, the agent can quickly adapt and maintain efficient racing performance on new tracks. This makes this solution particularly suitable for autonomous driving racing tasks that require flexibility in responding to new environments.

[0210] Figure 4FIG. 1 shows a computer system 100 for implementing the system and / or method according to the present invention. Unless otherwise specified, the method and / or system according to the present invention may be implemented in Figure 4 The present invention can be implemented in the computer system 100 shown in the figure to achieve the purpose of the present invention, or the present invention can be implemented in a distributed manner in multiple computer systems 100 according to the present invention through a network, such as a local area network or the Internet. The computer system 100 of the present invention can include various types of computer systems, such as handheld devices, laptop computers, personal digital assistants (PDAs), multi-processor systems, microprocessor-based or programmable consumer electronic devices, network PCs, minicomputers, mainframes, network servers, tablet computers, etc.

[0211] like Figure 4 As shown, the computer system 100 includes a processor 111, a system bus 101, a system memory 102, a video adapter 105, an audio adapter 107, a hard disk drive interface 109, an optical drive interface 113, a network interface 114, and a universal serial bus (USB) interface 112. The system bus 101 can be any of several types of bus structures, such as a memory bus or a memory controller, a peripheral bus, and a local bus using various types of bus architectures. The system bus 101 is used for communication between various bus devices. In addition to Figure 4 In addition to the bus devices or interfaces shown in , other bus devices or interfaces are also conceivable. System memory 102 includes read-only memory (ROM) 103 and random access memory (RAM) 104, wherein ROM 103 can, for example, store basic input / output system (BIOS) data for basic routines for implementing information transmission at startup, and RAM 104 is used to provide the system with a running memory with a faster access speed. Computer system 100 also includes a hard disk drive 109 for reading and writing hard disk 110, an optical drive interface 113 for reading and writing optical media such as CD-ROM, etc. Hard disk 110 can, for example, store an operating system and application programs. The drive and its associated computer-readable medium provide non-volatile storage of computer-readable instructions, data structures, program modules and other data for computer system 100. Computer system 100 can also include a video adapter 105 for image processing and / or image output, which is used to connect output devices such as display 106. The computer system 100 may further include an audio adapter 107 for audio processing and / or audio output, which is used to connect to output devices such as speakers 108. In addition, the computer system 100 may further include a network interface 114 for network connection, wherein the network interface 114 may be connected to the Internet 116 through a network device such as a router 115, wherein the connection may be wired or wireless. In addition, the computer system 100 may further include a universal serial bus interface (USB) for connecting peripheral devices.

[0212] 112, wherein the peripheral devices include, for example, a keyboard 117, a mouse 118 and other peripheral devices, such as a microphone, a camera, etc.

[0213] Embodiments may be provided as computer program products that may include one or more machine-readable media having machine-executable instructions stored thereon, which, when executed by one or more machines such as a computer, a computer network, or other electronic device, may cause the one or more machines to perform operations according to embodiments of the present invention. Machine-readable media may include, but are not limited to, floppy disks, optical disks, CD-ROMs (Compact Disc Read Only Memory) and magneto-optical disks, ROMs (Read Only Memory), RAMs (Random Access Memory), EPROMs (Erasable Programmable Read Only Memory), EEPROMs (Electrically Erasable Programmable Read Only Memory), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0214] In addition, various embodiments may be downloaded as a computer program product, where the program may be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) using one or more data signals implemented and / or modulated by a carrier wave or other propagation medium. Thus, the machine-readable medium used herein may include such a carrier wave, but this is not required.

[0215] Although some embodiments of the present invention have been described in this application document, it will be appreciated by those skilled in the art that these embodiments are merely shown as examples. Those skilled in the art may conceive of numerous variations, alternatives, and improvements under the teachings of the present invention without departing from the scope of the present invention. The appended claims are intended to define the scope of the present invention and to cover methods and structures within the scope of these claims themselves and their equivalents.

Claims

1. A trajectory-guided reinforcement learning method for autonomous driving racing, characterized in that: include: Simulate the track environment and determine the motion model that describes the dynamic behavior of the agent; Set up the agent’s action space and state space; Using the TD3 algorithm to establish a trajectory-guided reinforcement learning architecture, where the trajectory-guided reinforcement learning architecture has an evaluation network and an execution network; Introducing uncertainty in the friction coefficient into the trajectory-guided reinforcement learning framework; Recurrent training of trajectory-guided reinforcement learning architectures; as well as Optimize the parameters of the evaluation network and the execution network of the trajectory-guided reinforcement learning architecture.

2. The method according to claim 1, characterized in that Using the F1TENTH track environment, a single-track model is used to describe the dynamic behavior of the agent. The vehicle kinematic model in the single-track model is used to describe the displacement and posture changes of the agent, including the calculation of the lateral and longitudinal position change rates: Among them, v is the longitudinal velocity of the vehicle, ψ is the heading angle, and β is the sideslip angle. The vehicle dynamics model expressions in the single track model include: Longitudinal acceleration: Among them, x4 represents the current vehicle speed, and u2 is the longitudinal thrust control input; Heading angular acceleration: Where μ represents the road friction coefficient, m is the vehicle mass, and I z is the moment of inertia about the vehicle's center of mass, l f and l r are the distances from the front axle and rear axle to the center of mass of the vehicle, C S,f and C S,r are the cornering stiffness of the front and rear wheels, h c,g represents the height of the center of mass, δ is the steering angle; and Slip angular velocity:

3. The method according to claim 2, characterized in that The setting of the action space and state space of the intelligent agent includes: State space s t : s t = {LiDAR scans, x, y, ψ, v, ω}, where LiDAR scans is the LiDAR scan data, x, y represent the position information of the intelligent body, and v represents the longitudinal velocity; Action Space a t :a t ={δ,v}, where the range of the steering angle δ is [-1,1], the maximum steering angle = 0.4189 radians, and the range of the speed v is [-v max ,v max ], where v max =8m / s.

4. The method according to claim 3, characterized in that: The TD3 algorithm is used to establish a trajectory-guided reinforcement learning architecture, where the trajectory-guided reinforcement learning architecture has an evaluation network and an execution network including: Initialize the evaluation network and the execution network. The TD3 algorithm includes two evaluation networks Q1 and Q2 and an execution network π. The initialization parameters are as follows: set up θ π To evaluate the parameters of networks Q1 and Q2 and the execution network π, the parameters are randomly initialized; Set a target network for each of the two evaluation networks and one execution network, and initialize the parameters of the target network: θ′ π Evaluate the network parameters for two objectives and enforce the network parameters for one objective; Initialize the experience replay buffer The experience replay buffer is used to store the four-tuple of the interaction between the agent and the environment, and the four-tuple of the environment interaction includes state, action, reward and next state; Initialize a planner based on a pure pursuit algorithm, which is used to calculate the ideal trajectory action for the current state at each time step, including the reference speed v ref and steering angle δ ref ; According to the current position of the agent, select the target point along the preview distance in front of the trajectory; After determining the target point, the planner calculates the curvature of an arc passing through the agent's current position and the target point, denoted as k; Calculate the steering angle δ ref : Among them, L represents the wheelbase of the vehicle, that is, the distance between the front and rear wheels. f represents the lateral distance from the target point to the center of mass of the vehicle, and α is the yaw angle of the vehicle; Adjust the reference speed v according to the curvature of the predetermined trajectory ref .

5. The method according to claim 4, characterized in that The uncertainty introduced into the friction coefficient in the trajectory-guided reinforcement learning framework includes: Initialize the friction coefficient with Gaussian noise: Let the friction coefficient μ be the friction between the vehicle and the road, the initial setting value is μ0, and randomize the friction coefficient at the beginning of each training round: Where ∈ is Gaussian noise with mean 0 and variance σ 2 Normal distribution of At each time step t, the friction coefficient is dynamically and randomly updated to simulate different road friction conditions.

6. The method according to claim 5, characterized in that The loop training of the trajectory-guided reinforcement learning architecture includes: The following steps are performed in each training epoch: At each time step t, the reference velocity v of the reference action is generated using the planner ref and steering angle δ ref ; Use the current state s to generate the optimal action a through the execution network π(s), and superimpose Gaussian noise ∈ on the output of the execution network: The standard deviation σ of the noise term ∈ controls the randomness of the action; After the agent performs action a, the environment returns a new state s′ and an immediate reward r: Based on the agent's actual speed v agent and steering angle δ agent With reference speed v ref and steering angle δ ref The deviation between them is used to calculate the reward: Store the four-tuple (s, a, r, s′) consisting of state s, action a, reward r and next state s′ into the replay buffer middle.

7. The method according to claim 6, characterized in that If the agent collides with any obstacle or goes off track, it will receive a reward of -1; When no collision occurs, the reward is determined by the consistency between the agent's action and the planner's reference action: the actual velocity v agent and steering angle δ agent The reward is calculated based on the difference between the value and the reference value provided by the planner; In the absence of collisions, the reward is scaled by 0.2 and capped above 0 to ensure a balanced reward structure.

8. The method according to claim 6, characterized in that The optimization of the parameters of the evaluation network and execution network of the trajectory-guided reinforcement learning architecture includes: Calculate the target Q value through mini-batch training; and Update the parameters of the evaluation network and the execution network based on the target Q value.

9. The method according to claim 8, characterized in that The target Q value calculated through small batch training includes: When the experience replay buffer When the number of samples in reaches the mini-batch size N, N samples are randomly sampled from the buffer, each of which consists of state, action, reward and next state, i.e. (s, a, r, s′); In TD3, the goal execution network π′ is used to generate the target action a′: a′=π′(s′)+∈,∈~clip(N(0,σ),-c,c) Among them, π′(s′) is the predicted action of the target execution network for the next state s′, and ∈ is a normal distribution. The noise term is clipped to the range [-c,c]; After obtaining the target action a′, the target evaluation network Q′1 and Q′2 are used to calculate the target Q value y, and the calculation formula is: y=r+γ·min(Q′1(s′,a′),Q′2(s′,a′)) Where r is the immediate reward, γ is the discount factor, and Q′1(s′,a′) and Q′2(s′,a′) are the value estimates of the target evaluation network for the next state s′ and target action a′.

10. The method according to claim 9, characterized in that The parameters of the evaluation network and the execution network are updated based on the target Q value, including: The targets of the evaluation networks Q1 and Q2 are updated by minimizing the following loss function to make their estimated values ​​close to the target Q value y: Where i = 1, is the parameter of the evaluation network Q1, the loss function (yQ i (s,a)) 2 represents the square error between the target Q value y and the estimated value of the evaluation network Q1 for the current state s and action a; when i = 2, is the parameter of the evaluation network Q2; the loss function (yQ i (s,a)) 2 represents the square of the error between the target Q value y and the estimated value of the evaluation network Q2 for the current state s and action a; and the execution network is updated once every d time steps. The update formula is as follows: Among them, θ π are the parameters of the execution network, α is the learning rate, Q1(s,π(s)) is the value estimate of the evaluation network Q1 for the execution network output action π(s) in state s, represents the gradient of the evaluation network with respect to the output of the execution network; At each time step, the target evaluation networks Q′1 and Q′2 and the target execution network π′ are soft updated so that the parameters of the target network gradually follow the parameter changes of the current network: θ′ π ←tth π +(1-τ)θ′ π in, and θ′ π are the parameters of the target evaluation network and the target execution network respectively, and τ is the soft update coefficient, which controls the rate at which the target network parameters follow the current network.

Images