Autonomous race car control method based on constrained residual reinforcement learning

Abstract

The application provides an autonomous racing car control method based on constraint residual reinforcement learning, comprising: using a prior control strategy and a reinforcement learning control strategy, so that the prior control strategy provides a guide action to reduce invalid exploration of the reinforcement learning control strategy and accelerate convergence of the reinforcement learning control strategy; improving performance of a final control strategy by training the reinforcement learning control strategy; and limiting action output of the reinforcement learning control strategy network to ensure that the racing car preferentially explores a region more relevant to the prior control strategy in a training process to find an optimal driving strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a self-driving racing car control method based on constraint residual reinforcement learning. BACKGROUND

[0002] The automobile industry is rapidly developing towards sharing, electrification, networking and intelligence, and artificial intelligence technology has shown great value and potential in the direction of intelligent driving. Deep reinforcement learning (DRL) algorithm can enable the agent to continuously learn and represent the state of the environment and give the current best action plan at each given time, which also promotes the development of intelligent and automatic driving cars. The automatic driving car, also known as the unmanned driving car, is an intelligent car that realizes unmanned driving through an automatic driving system and has shown a trend of approaching practicality in the 21st century. The automatic driving car relies on artificial intelligence, visual computing, radar, monitoring devices and global positioning system to cooperate, so that the car can automatically and safely operate the motor vehicle without any human initiative.

[0003] However, the existing automatic driving control method has the problems of complex strategy making, tedious control parameter debugging and poor environmental adaptability. Therefore, the present application provides a self-driving racing car control method based on constraint residual reinforcement learning. SUMMARY

[0004] The present application aims to provide a self-driving racing car control method based on constraint residual reinforcement learning to wholly or partially solve the problem of the existing automatic driving control method having complex strategy making, tedious control parameter debugging and poor environmental adaptability.

[0005] To solve the above technical problems, the present application provides a self-driving racing car control method based on constraint residual reinforcement learning, comprising:

[0006] The prior control strategy and the reinforcement learning control strategy are used, so that the prior control strategy provides a guided action to reduce the invalid exploration of the reinforcement learning control strategy and accelerate the convergence of the reinforcement learning control strategy;

[0007] The performance of the final control strategy is improved by training the reinforcement learning control strategy;

[0008] The action output of the reinforcement learning control strategy network is limited to ensure that the racing car preferentially explores the area more relevant to the prior control strategy during the training process to find the optimal driving strategy;

[0009] The prior control strategy comprises: according to the motion speed of the racing car, using the center of the maximum gap detected by the laser radar and the farthest point in the maximum gap range to control the safe steering of the racing car, wherein the input power of the racing car is proportional to the farthest distance detected by the laser radar, and the safe driving speed is ensured.

[0010] The reinforcement learning strategy comprises: using a proximal policy optimization algorithm, and simultaneously using a discrete reward signal and a continuous reward signal to improve the prior control strategy, so as to improve the performance of the final control strategy of the racing car.

[0011] The maximum gap is the maximum value in the distance between each adjacent two observation points in the point cloud array detected by the laser radar.

[0012] Optionally, in the autonomous racing car control method based on the constraint residual reinforcement learning, the method further comprises:

[0013] Step one: establishing a partially observable Markov process corresponding to the autonomous racing car problem;

[0014] Step two: generating a safe guidance action according to the observation data of the laser radar in accordance with the gap method: by selecting the maximum gap in the laser radar measurement data, guiding the racing car to enter the center of the maximum gap as much as possible, and providing a target point to ensure safety, a more safe trajectory is generated;

[0015] Step three: generating a supplementary action using a proximal policy optimization algorithm, the proximal policy optimization algorithm being a policy gradient algorithm based on an Actor-Critic architecture, the training being stabilized by limiting the policy network update to the relevant area of the driving trajectory generated by the prior control strategy, and a truncated target function being used to prevent rapid changes in the policy;

[0016] Step four: to reduce the frequency of dangerous actions in the racing car training process, the action output of the policy network of the proximal policy optimization algorithm is constrained, so as to ensure that the racing car only explores the area most related to the output trajectory of the prior policy in the training process, to reduce invalid exploration, and the proximal policy optimization algorithm compensates for the deficiencies of the prior controller through learning;

[0017] Step five: according to the control action obtained in step four, the control action is input into the controlled racing car to interact with the environment, and the observation information of the next time after the interaction and the reward value after the execution of the last step are obtained; the observation information of the next state is fed back to the gap method algorithm module, and the observation information of the next state and the reward value after the execution of the last step are fed back to the proximal policy optimization algorithm module.

[0018] Optionally, in the autonomous racing car control method based on the constraint residual reinforcement learning, the method further comprises:

[0019] Step 1: Establish a partially observable Markov process corresponding to the autonomous racing problem, represented by a tuple (S, A, Ω, O, T, R), where S is the state space, A is the action space, Ω is the observation set, and R is the reward function;

[0020] The state transition function T:S×A×S→[0,1] represents the state transition probability under a specific state and action;

[0021] The observation function O:S×Ω→[0,1] represents the probability of observation in a specific state;

[0022] A finite trajectory is defined as τ=(s0,a0,r0,…,s T ,a T ,r T ), where s t ∈S,a t ∈A,(0≤t≤T), T represents the maximum time step;

[0023] The reward signal is r t =R(s t ,a t )(0≤t≤T) represents; given a discount factor γ∈[0,1), the reinforcement learning control strategy obtains the optimal strategy π by optimizing the strategy parameter θ θ , to maximize the expectation of the cumulative reward on the trajectory τ:

[0024]

[0025] The continuous action of the car is set as a=(δ,F), where δ∈[-45 ° ,+45 ° ] is the steering angle input to the car, F∈[-1,+1] is the power input to the car;

[0026] The car is equipped with a 2D lidar with a maximum detection range of D = 10m and N = 675 distance measurements, evenly distributed at 270 ° within the field of vision;

[0027] The racing car is equipped with a 60HZ inertial measurement unit to obtain vehicle motion information, including lateral acceleration, longitudinal acceleration, lateral speed and longitudinal speed.

[0028] Optionally, in the autonomous racing car control method based on constrained residual reinforcement learning,

[0029] The gap method algorithm in step 2 includes: calculating the gap array from the lidar data, calculating the center angle of the maximum gap, and calculating the steering angle and power;

[0030] Calculating gap array from lidar data includes:

[0031] The point cloud observed by the lidar is regarded as a group of small obstacles, and the point cloud sequence obtained is Obs=[o1,o2,…,o N N, and the gap array corresponding to the observation data is obtained by calculating the distance between each adjacent two points Gap=[g1,g2,…,g N+1 N, where N=675;

[0032] Calculating the center angle of the maximum gap includes:

[0033] Suppose the maximum gap is composed of observation points o i ,o j d i ,d j represent the distance between the racing car and the observation points o i ,o j , and φ i ,φ j is the angle;

[0034] Point c is the center point of the maximum gap o i o j , and point goal is the target point, which is the farthest point detected by the lidar within the maximum gap, and φ goal is the angle of point goal;

[0035] The calculation formula of the gap center angle is:

[0036]

[0037] Calculating the steering angle and power includes:

[0038] The steering angle is calculated according to the gap center angle and the target angle φ goal :

[0039]

[0040] Wherein, α=0.65 is a weighting factor, and d min is the nearest distance observed by the lidar; the combination structure depends on the size of the nearest distance to the surrounding obstacles and the weighting factor; if the obstacles are near the vehicle, safety should be considered first, and the steering angle of the vehicle should be more biased towards the gap center angle On the contrary, the steering angle of the vehicle should be more biased towards the target angle φ goal ; the power calculation formula input to the vehicle is:

[0041]

[0042] Wherein, d maxFor the farthest distance detected by the laser radar during the driving of the vehicle, when the detected farthest distance is less than 3m, the vehicle keeps a low speed driving; otherwise, the power input to the vehicle is proportional to the detected farthest distance; the output action of the gap method algorithm is represented as a prior =(δ prior ,F prior )。

[0043] Optionally, in the autonomous racing car control method based on the constraint residual reinforcement learning, further comprising:

[0044] Step three: according to the problem definition in step one, the policy network objective function of the proximal policy optimization algorithm is as follows:

[0045]

[0046] Wherein, is the probability ratio describing the similarity between the new policy and the old policy; clip(·) is a clipping function, ∈=0.2 is a hyperparameter, and the advantage function is defined as:

[0047]

[0048] Wherein, is the value function of state s t , and the advantage function represents the advantage of a certain action a relative to the average under state s. When , it indicates that the current action is worth encouraging, and the probability value of π θ (a t |s t ) is increased, thereby causing r t (θ) to increase; however, too high r t (θ) will lead to unstable policy learning, so the objective function L CLIP (θ) limits r t (θ) to not exceed 1+∈. Similarly, when , r t (θ) is limited to not less than 1-∈;

[0049] The ultimate goal of the proximal policy optimization algorithm is to maximize the following objective function:

[0050]

[0051] Wherein, is the mean square error loss of the value network, is the TD target of time step t; maximizing The error between the predicted value of the value network and the true value is represented, so that the prediction of the value network becomes more and more accurate; H represents the entropy value of the strategy, and maximizing H ensures sufficient exploration of the strategy;

[0052] According to the goal of the autonomous racing car is to minimize the single lap time of the racing car, a reward function combining discrete reward signals and continuous reward signals is used:

[0053] R = R lap + R v -P a -P crash

[0054] Where R lap = 100 is the reward when the racing car completes a lap, R v is linearly related to the speed of the racing car, encouraging the racing car to produce faster driving speed; the penalty term P a penalizes the racing car for excessive action changes in adjacent time steps to ensure smooth driving, and the penalty term P crash = 1 when the racing car collides with the track; this reward function combines a discrete reward signal and a continuous reward signal, while penalizing the racing car for collisions with the track and excessive action changes.

[0055] Optionally, in the autonomous racing car control method based on constraint residual error reinforcement learning, further comprising:

[0056] Step four: constrain the output of the policy network through the constraint proximal policy optimization algorithm, and obtain the action fusion formula:

[0057] a = clip (a prior (1 + beta a rl ), -1, +1)

[0058] Where a prior is the output action of the prior controller, a rl is the output action of the PPO policy network, and beta = 0.6 is the constraint factor for constraining the output of the PPO policy network.

[0059] Optionally, in the autonomous racing car control method based on constraint residual error reinforcement learning, further comprising:

[0060] Step five: input the control action obtained in step four into the controlled racing car to interact with the environment, and obtain the observation information o t+1 and the reward value r t after the last step is executed. The observation information o t+1 of the next state is fed back to the FTG algorithm module, and the observation information o t+1 of the next state and the reward value r tFeedback to the PPO algorithm module.

[0061] Optionally, in the autonomous racing car control method based on constraint residual reinforcement learning, further comprising:

[0062] In the prior control strategy, the safe driving of the racing car is considered, the vehicle is guided to the center of the track as much as possible, and the calculation method of the calculation guiding action is ensured to ensure the safe driving of the racing car.

[0063]

[0064]

[0065] By combining the maximum gap center angle and the target angle, the safety of the steering in the guiding action is ensured; when the maximum distance detected by the laser radar does not exceed 3m, the minimum input power is maintained to avoid collision.

[0066] Optionally, in the autonomous racing car control method based on constraint residual reinforcement learning, further comprising:

[0067] The autonomous racing car control method is designed by combining double control strategies, the near-optimal policy optimization algorithm with stable performance is selected in the reinforcement learning control strategy, and the reward function combining the discrete reward signal and the continuous reward signal is designed, so that the optimal strategy is more easily learned.

[0068] In order to minimize the single lap time of the racing car, the discrete reward value when the racing car completes a lap and the speed continuous reward value of the racing car are combined, and the collision and the change of the control action are punished, so that the racing car driving track is more smooth, and the reward function is obtained:

[0069] R=R lap +R v -P a -P crash

[0070] Optionally, in the autonomous racing car control method based on constraint residual reinforcement learning, further comprising:

[0071] The fusion mode of the prior control strategy and the reinforcement learning control strategy is designed, the action output of the reinforcement learning strategy network relative to the prior strategy is limited, so that the racing car only explores in the most relevant area of the prior strategy, the invalid exploration of the racing car is reduced, the reinforcement learning control strategy can improve the performance of the prior control strategy after training, the single lap time of the racing car is reduced, and the fusion mode of the output actions of the two strategies is designed in step four:

[0072] a=clip(a prior (1+βa rl ),-1,+1)

[0073] In the formula, the output of the reinforcement learning control policy network is affected by the constraint factor β, and the proportion of the final control action relative to the prior control policy output is low, which ensures that the prior policy will not be covered by the reinforcement learning policy in the early exploration stage, and reduces the generation of invalid exploration and dangerous actions.

[0074] The inventors of the present application have found through research that:

[0075] 1) The traditional control method decouples the autonomous racing problem into path planning and path tracking sub-modules. The optimal racing line is calculated by the known vehicle dynamics model, and then the path tracking module ensures that the vehicle travels on the known optimal path. However, this method requires accurate dynamic model and detailed map information and a large amount of hardware support, and processing nonlinear optimization requires a large amount of computing resources, which has certain limitations.

[0076] 2) The reinforcement learning method can directly map the perceived high-dimensional environment information to low-level control commands, thereby realizing an end-to-end driving. This method does not require an accurate vehicle dynamics model and a large amount of hardware resources. However, both model-free reinforcement learning algorithms and model-based reinforcement learning algorithms start from scratch, have low exploration efficiency and sample efficiency, and are difficult to learn the optimal driving strategy.

[0077] 3) The reinforcement learning method from behavior prior is mainly divided into demonstration reinforcement learning and residual reinforcement learning in the application of autonomous racing. For demonstration reinforcement learning, this method is very dependent on the accuracy of expert data and is easily disturbed by noise of expert data. For residual reinforcement learning, the reinforcement learning strategy will cover the prior strategy in the early training, and there is still part of the invalid exploration. Therefore, it is necessary to better handle the fusion between the prior strategy and the reinforcement learning strategy.

[0078] Based on the above insights, the present application provides an autonomous racing control method based on constraint residual reinforcement learning. For the autonomous racing problem, a control algorithm is proposed that combines the traditional reactive navigation algorithm Follow-The-Gap (FTG) and the Proximal Policy Optimization (PPO) algorithm in reinforcement learning. The sample inefficiency problem of single reinforcement learning method is solved, the policy network output relative to the prior policy is limited by using constraints, and the problem of prior policy being covered in residual reinforcement learning is solved. And by learning and training the reinforcement learning strategy, the performance of the final fusion strategy is improved, realizing an efficient and high-performance autonomous racing control method.

[0079] The application designs a racing car control method combining a traditional reactive navigation algorithm with a reinforcement learning algorithm in an environment without detailed track information, and only uses real-time observation data of a 2D laser radar and a 60HZ inertial measurement unit (IMU) to realize high-performance autonomous racing car control, which plays an important role in the fields of unmanned driving and robot control. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 is an algorithm function framework schematic diagram of an autonomous racing car control method based on constraint residual reinforcement learning according to an embodiment of the application;

[0081] Figure 2 is a step schematic diagram of an autonomous racing car control method based on constraint residual reinforcement learning according to an embodiment of the application;

[0082] Figure 3 is a geometric principle schematic diagram of an FTG algorithm of an autonomous racing car control method based on constraint residual reinforcement learning according to an embodiment of the application;

[0083] Figure 4 is a schematic diagram of four tracks for simulation experiments of an autonomous racing car control method based on constraint residual reinforcement learning according to an embodiment of the application;

[0084] Figure 5 is a simulation result schematic diagram of an autonomous racing car control method based on constraint residual reinforcement learning according to an embodiment of the application. DETAILED DESCRIPTION

[0085] The application will be further described below in conjunction with the specific embodiments and the accompanying drawings.

[0086] It should be noted that the components in the drawings can be exaggerated for illustration, and are not necessarily to scale. In the drawings, the same or similar components are provided with the same reference numerals.

[0087] In the application, unless specifically indicated, “arranged on”, “arranged above” and “arranged over” do not exclude the presence of an intermediate object between them. In addition, “arranged on or above” only indicates the relative position relationship between the two components, and in some cases, such as after reversing the product direction, it can also be converted to “arranged below or below”, and vice versa.

[0088] In the application, each embodiment is only intended to illustrate the scheme of the application, and should not be understood as limiting.

[0089] In the application, unless specifically indicated, the quantifier “one” does not exclude the scenario of multiple elements.

[0090] It should also be noted that, in the embodiments of the present application, only a part of components or assemblies can be shown for the purpose of clarity and simplicity, but those skilled in the art can understand that, under the teaching of the present application, the required components or assemblies can be added according to the specific scene. In addition, the features in different embodiments of the present application can be combined with each other unless otherwise stated. For example, a feature in the second embodiment can replace a corresponding or functionally similar feature in the first embodiment, and the resulting embodiment also falls within the scope or range of the present application.

[0091] It should also be noted that, in the scope of the present application, the expressions "same", "equal", "equal to" and the like do not mean that the numerical values of the two are absolutely equal, but allow a certain reasonable error, that is, the expressions also cover "substantially same", "substantially equal", "substantially equal to". By analogy, in the present application, the terms "perpendicular to", "parallel to" and the like in the direction of the table also cover the meanings of "substantially perpendicular to", "substantially parallel to".

[0092] In addition, the numbering of the steps of the methods of the present application does not limit the execution order of the method steps. Unless otherwise specified, the method steps can be executed in different orders.

[0093] The autonomous racing car control method based on constraint residual reinforcement learning proposed by the present application will be further described in detail below in combination with the accompanying drawings and specific embodiments. The advantages and features of the present application will be clearer according to the following description. It should be noted that the drawings are very simplified and use non-precise proportions, only for the purpose of facilitating and clearly assisting the description of the embodiments of the present application.

[0094] The purpose of the present application is to provide an autonomous racing car control method based on constraint residual reinforcement learning to solve the problems of existing automatic driving control methods, such as complex strategy making, tedious control parameter debugging and poor environmental adaptability, in whole or in part.

[0095] To achieve the above object, the application provides an autonomous racing car control method based on constraint residual reinforcement learning, comprising: using a prior control strategy and a reinforcement learning control strategy, so that the prior control strategy provides a guiding action to reduce the invalid exploration of the reinforcement learning control strategy and accelerate the convergence of the reinforcement learning control strategy; improving the performance of the final control strategy by training the reinforcement learning control strategy; limiting the action output of the reinforcement learning control strategy network to ensure that the racing car preferentially explores the area more relevant to the prior control strategy during the training process to find the optimal driving strategy; the prior control strategy comprises: controlling the safe steering of the racing car according to the motion speed of the racing car, the center of the maximum gap detected by the laser radar and the farthest point in the maximum gap range, wherein the input power of the racing car is proportional to the farthest distance detected by the laser radar to ensure the safe driving speed; and the reinforcement learning strategy comprises: using the proximal policy optimization algorithm, and simultaneously using the discrete reward signal and the continuous reward signal to improve the prior control strategy to improve the performance of the final control strategy of the racing car.

[0096] The application proposes a control algorithm that fuses the traditional reactive navigation algorithm Follow-The-Gap (FTG) and the proximal policy optimization algorithm Proximal Policy Optimization (PPO) in reinforcement learning for autonomous racing car problems; solves the sample inefficiency problem of a single reinforcement learning method, limits the output of the policy network relative to the prior strategy by using constraints, solves the problem that the prior strategy is covered in residual reinforcement learning; and improves the performance of the final fusion strategy by training the reinforcement learning strategy, realizing an efficient and high-performance autonomous racing car control method.

[0097] Firstly, the two control strategies of the prior strategy and the reinforcement learning strategy are fused, the prior strategy provides a guiding action, greatly reduces the invalid exploration of the reinforcement learning strategy, and accelerates the convergence of the reinforcement learning strategy. At the same time, the reinforcement learning strategy can further improve the performance of the final control strategy through training.

[0098] Secondly, for the prior control strategy, considering the high-speed motion of the racing car, the center of the maximum gap detected by the laser radar and the farthest point in the maximum gap range are used to ensure the safe steering of the racing car, the input power of the racing car is proportional to the farthest distance detected by the laser radar to ensure the safe driving speed. For the reinforcement learning strategy, the excellent proximal policy optimization algorithm is used, and the discrete reward signal and the continuous reward signal are combined to improve the prior strategy and improve the final performance of the racing car.

[0099] In addition, for the fusion of prior strategies and reinforcement learning strategies, a constraint formula was designed to limit the action output of the reinforcement learning strategy network, ensuring that the car only explores the areas most relevant to the prior strategy during training, making it easier to find the optimal driving strategy.

[0100] The algorithm function framework of the autonomous racing car control method based on constrained residual reinforcement learning (CRR) of the present invention is as follows: Figure 1 As shown in the figure, the system consists of two action generators: a priori control (FTG) algorithm module and a reinforcement learning (PPO) algorithm module. The FTG algorithm module generates safe guidance actions based on observed information. Simultaneously, the PPO algorithm module, under the influence of a constraint formula, only explores the most relevant areas for the guidance action. Through training and learning, it improves the final control action to reduce the car's lap time.

[0101] Figure 2 The figure shows a schematic diagram of the steps of the autonomous racing car control method based on constrained residual reinforcement learning of the present invention, including:

[0102] Step 1: Establish a partially observable Markov process (POMDP) ​​corresponding to the autonomous racing problem, represented by a tuple (S, A, Ω, O, T, R), where S is the state space, A is the action space, Ω is the observation set, and R is the reward function. The state transition function T: S × A × S → [0, 1] represents the state transition probability under a specific state and action, and the observation function O: S × Ω → [0, 1] represents the observation probability under a specific state. A finite trajectory is defined as τ = (s0, a0, r0, …, s T ,a T ,r T ), where (s t ∈S,a t ∈A,(0≤t≤T)), T represents the maximum time step. The reward signal is represented by r t =R(s t ,a t )(0≤t≤T). Given a discount factor γ∈[0,1), the goal of the reinforcement learning algorithm is to obtain the optimal policy π by optimizing the policy parameter θ θ , to maximize the expectation of the cumulative reward on the trajectory τ:

[0103]

[0104] The continuous action of the car is set as a=(δ,F), where δ∈[-45 ° ,+45 °is the steering angle input to the car, F e [-1, +1] is the force input to the car. The car is equipped with a 2D lidar with a maximum detection range of D = 10m, with N = 675 range measurements uniformly distributed over the 270 ° field of view. The car is also equipped with a 60HZ inertial measurement unit (IMU) to obtain the motion information of the vehicle, including lateral and longitudinal acceleration, speed.

[0105] Step two: According to the observation data of the lidar, the safe guidance action can be generated by following the gap method FTG. FTG is a geometric-based reactive obstacle avoidance algorithm that guides the car to enter the center of the largest gap as much as possible by selecting the largest gap in the lidar measurement data, while providing a target point to ensure safety, which can generate a relatively safe trajectory. The implementation of this algorithm includes the following three steps:

[0106] First: Calculate the gap array from the lidar data

[0107] The point cloud observed by the lidar is regarded as a group of small obstacles, and the obtained point cloud sequence is Obs = [o1, o2, …, o N N]. By calculating the distance between each adjacent two points, the gap array corresponding to the observation data can be obtained as Gap = [g1, g2, …, g N+1 N], where N = 675.

[0108] Second, calculate the center angle of the largest gap:

[0109] As shown in Figure 3 , assume that the largest gap is composed of observation points o i , o j , d i , d j represent the distance between the car and the observation points o i , o j , and φ i , φ j is the angle. Point c is the center point of the largest gap o i o j , and point goal is the target point, which is the farthest distance point detected by the lidar within the largest gap, and φ goal is the angle of point goal. The calculation formula of the gap center angle is:

[0110]

[0111] Finally, calculate the steering angle and power:

[0112] According to the gap center angle and the target angle φ goal , the steering angle can be calculated:

[0113]

[0114] where α = 0.65 is a weighting factor, d min is the closest distance observed by the lidar. This combination structure depends on the closest distance to the surrounding obstacles and the size of the weighting factor. If the obstacle is near the vehicle, safety should be considered first, and the steering angle of the vehicle should be more biased towards the gap center angle On the contrary, the steering angle of the vehicle should be more biased towards the target angle φ goal . The input to the vehicle's power calculation formula is:

[0115]

[0116] where d max is the farthest distance detected by the lidar during the vehicle's driving process. When the detected farthest distance is less than 3m, the vehicle keeps a low speed. Otherwise, the input to the vehicle's power is proportional to the detected farthest distance. The output action of the priori strategy FTG algorithm is represented as a prior = (δ prior , F prior ).

[0117] Step three: adopt the proximal policy optimization algorithm (PPO) to generate supplementary actions. PPO is a powerful policy gradient algorithm based on the Actor-Critic architecture, which has been widely used in solving discrete or continuous control problems. Unlike traditional policy gradient methods, PPO stabilizes training by limiting policy network updates to a specific range and uses a truncated objective function to prevent rapid changes in policy. According to the problem definition in step one, the policy network objective function of the PPO algorithm is as follows:

[0118]

[0119] where, is the probability ratio that describes the similarity between the new policy and the old policy. clip(·) is a clipping function, ∈ = 0.2 is a hyperparameter, and the advantage function is defined as:

[0120]

[0121] where, is the value function of state s t , and the advantage function represents the advantage of a certain action a relative to the average under state s. When , it indicates that the current action is worth encouraging, and π θ (a t |st ) increases, resulting in an increase in r t (θ). t However, too high a r CLIP (θ) can lead to unstable policy learning, so the objective function L t (θ) limits r t (θ) to no more than 1 + ε.

[0122] Similarly, when r lap (θ) is limited to no less than 1 - ε.

[0123] The ultimate goal of the PPO algorithm is to maximize the following objective function:

[0124]

[0125] where is the mean squared error loss of the value network, is the TD target at time step t. Maximizing indicates reducing the error between the predicted value of the value network and the true value, so that the prediction of the value network becomes more and more accurate. H represents the entropy value of the policy, and maximizing H ensures sufficient exploration of the policy, which is to improve the robustness of the PPO algorithm.

[0126] The goal of the autonomous racing car is to minimize the lap time of the racing car. Considering that the lap time is a very sparse reward signal that cannot be attributed to a specific action, a reward function design is adopted that combines discrete and continuous reward signals. The specific reward function is:

[0127] R = R v - R a - P crash

[0128] where R lap = 100 is the reward when the racing car completes a lap, R v is linearly related to the speed of the racing car, encouraging the racing car to produce faster driving speeds. The penalty term P a penalizes the racing car for excessive action changes in adjacent time steps to ensure smooth driving, and the penalty term P crash = 1 is produced when the racing car collides with the track. This reward function combines a discrete reward signal and a continuous reward signal, while penalizing the racing car for collisions with the track and excessive action changes.

[0129] Step 4: To reduce the frequency of dangerous maneuvers during racing training, constrain the action outputs of the PPO policy network. By constraining the outputs of the PPO policy network, a safer fusion strategy can be obtained. This fusion method ensures that the car only explores the areas most relevant to the output trajectory of the prior strategy during training, thereby reducing ineffective exploration and improving sampling efficiency. The PPO algorithm can also compensate for the shortcomings of the prior controller through learning. The action fusion formula is:

[0130] a=clip(a prior (1+βa rl ),-1,+1)

[0131] Among them, a prior is the output action of the prior controller, a rl is the output action of the PPO strategy network, and β = 0.6 is the constraint factor that constrains the output of the PPO strategy network.

[0132] Step 5: Based on the control action obtained in step 4, input it into the controlled car to interact with the environment and obtain the observation information o at the next moment after the interaction. t+1 And the reward value r after the previous step t . The observation information o in the next state t+1 Feedback to the FTG algorithm module, the observation information o in the next state t+1 And the reward value r after the previous step t Feedback to the PPO algorithm module.

[0133] Compared with the prior art, the present invention has three important improvements, including:

[0134] Improvement 1: Design an autonomous racing car control method that integrates dual control strategies, considers the safe driving of the car in the prior strategy, guides the vehicle to the center of the track as much as possible, and ensures the safe driving of the car.

[0135] Since racing is a high-speed sport, safe guidance actions are very necessary. The prior control strategy FTG algorithm ensures the safety of the output action. In step 2, a calculation method for calculating the guidance action is designed:

[0136]

[0137]

[0138] By combining the maximum gap center angle and the target angle, the steering safety during the guidance action is ensured; at the same time, considering the high speed of the car, when the maximum distance detected by the lidar does not exceed 3m, the input power is kept to the minimum to avoid collision.

[0139] Improvement point 2: design a double control strategy fusion autonomous racing car control method, select a stable performance proximal policy optimization algorithm in the reinforcement learning strategy, and design a reward function combining discrete reward signals and continuous reward signals, which is easier to learn the optimal strategy.

[0140] In order to realize the minimization of the single lap time of the racing car, the discrete reward value when the racing car completes a lap and the continuous reward value of the speed of the racing car are combined, and the collision and the large change of the control action are punished, so as to ensure that the driving track of the racing car is smoother. In step three, the reward function is designed:

[0141] R=R lap +R v -R a -P crash

[0142] Improvement point 3: the fusion mode of the priori strategy and the reinforcement learning strategy is designed, the action output of the reinforcement learning strategy network relative to the priori strategy is limited, so as to ensure that the racing car only explores in the most relevant area of the priori strategy, on the one hand, the invalid exploration of the racing car is reduced, and the sample efficiency is improved; on the other hand, the reinforcement learning algorithm can improve the performance of the priori strategy, and the single lap time of the racing car is reduced.

[0143] In step four, the fusion mode of the two strategy output actions is designed:

[0144] a=clip(a prior (1+βa rl ),-1,+1)

[0145] In the formula, the output of the reinforcement learning strategy network is constrained by the constraint factor β, and the proportion of the final control action relative to the priori strategy output is low, so that in the early exploration stage, the priori strategy will not be covered by the reinforcement learning strategy, and the generation of invalid exploration and dangerous actions is reduced.

[0146] As Figure 4 shown, the scheme (CRR) of the present application is compared with part of the prior art in the simulation experiment on four tracks:

[0147] Brief description of prior art for experimental comparison:

[0148] 1) PPO: only the reinforcement learning algorithm is used to control the racing car.

[0149] 2) FTG: only the priori control strategy is used to control the racing car.

[0150] 3) CORE-RL: this method balances the output proportion between the priori strategy and the reinforcement learning strategy based on the value of Td error.

[0151] 4) ResRace(APF): This method is based on residual policy learning, and uses the artificial potential field method (APF) as the prior control strategy. The final input of the racing action is the sum of the outputs of the prior strategy and the reinforcement learning strategy.

[0152] 5) ResRace(FTG): This method is based on residual policy learning, and uses the follow-the-gap method (FTG) as the prior control strategy. The final input of the racing action is the sum of the outputs of the prior strategy and the reinforcement learning strategy.

[0153] As shown in Figure 5 the training curves, the designed method CRR shows excellent performance on the four tracks. Under the action of the constraint formula designed in step four, the racing car reduces invalid exploration in the training process, and the learning curves on the Columbia and Austria two-day tracks fully prove this point. And it is easier to learn the optimal driving strategy on the complex Berlin and Treitlstrasse tracks.

[0154] Table 1. Comparison of single lap time of the invention (CRR) and prior art. "-" indicates that it is not completed.

[0155]

[0156]

[0157] As shown in Table 1, in the comparison of single lap time of various methods, it can be concluded that the invented method (CRR) achieves the optimal single lap time in most cases, and the final performance is higher than that of the FTG method, which proves the complementary advantages between the prior control strategy and the reinforcement learning strategy in the invented method. That is, the prior control strategy can guide and reduce the invalid exploration of the reinforcement learning strategy, and the reinforcement learning strategy can improve the performance of the prior control strategy after learning and training.

[0158] The application is suitable for autonomous racing car control in the case of unknown track details. The application only uses the observation information of 2D laser radar and IMU to achieve high-performance racing car control. By using a traditional obstacle avoidance control algorithm as a prior guide, the learning efficiency of the reinforcement learning algorithm is effectively improved; and by training the reinforcement learning algorithm, the performance of the prior control algorithm is further improved, achieving the complementary advantages of the two modules, thereby improving the final performance of the racing car.

[0159] To sum up, the above embodiments detail different configurations of the autonomous racing car control method based on constraint residual reinforcement learning. Of course, the present application includes but is not limited to the configurations listed in the above embodiments. Any transformation based on the configurations provided in the above embodiments belongs to the scope of protection of the present application. Those skilled in the art can easily deduce other embodiments from the above embodiments.

[0160] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiments, the description is relatively simple because it corresponds to the method disclosed in the embodiments. The relevant parts can be referred to the method part.

[0161] The above description is only a description of the preferred embodiments of the present application, and does not limit the scope of the present application. Any modification or change made by those skilled in the art based on the above disclosure belongs to the protection scope of the claims.

Claims

1. An autonomous racing car control method based on constrained residual reinforcement learning, characterized in that: include: Using a priori control strategy and reinforcement learning control strategy, the priori control strategy provides guidance actions to reduce the ineffective exploration of the reinforcement learning control strategy and accelerate the convergence of the reinforcement learning control strategy; Improve the performance of the final control strategy by training the reinforcement learning control strategy; Limiting the action outputs of the reinforcement learning control policy network ensures that the car prioritizes exploring areas more relevant to the prior control policy during training to find the optimal driving strategy; The a priori control strategy includes: controlling the safe steering of the car based on the speed of the car using the center of the maximum gap detected by the laser radar and the farthest point within the maximum gap, wherein the input power of the car is proportional to the farthest distance detected by the laser radar to ensure a safe driving speed; and The reinforcement learning strategy includes: using a proximal policy optimization algorithm to simultaneously improve the prior control strategy using discrete reward signals and continuous reward signals to improve the performance of the final control strategy of the racing car; The maximum gap is the maximum value of the distance between each two adjacent observation points in the point cloud array detected by the lidar.

2. The autonomous racing car control method based on constrained residual reinforcement learning according to claim 1, characterized in that: Also includes: Step 1: Establish a partially observable Markov process corresponding to the autonomous racing problem; Step 2: Based on the LiDAR observation data, a safety guidance action is generated using the gap method. This method selects the largest gap in the LiDAR measurement data and guides the car into the center of the largest gap as much as possible. A target point is provided to ensure safety, resulting in a safer trajectory. Step 3: Use the proximal policy optimization algorithm to generate supplementary actions. The proximal policy optimization algorithm is a policy gradient algorithm based on the actor-critic architecture. It stabilizes training by restricting policy network updates to the relevant area of ​​the driving trajectory generated by the prior control policy and uses a truncated objective function to prevent rapid changes in the policy. Step 4: To reduce the frequency of dangerous maneuvers during training, the action output of the policy network of the proximal policy optimization algorithm is constrained. This ensures that the car only explores the areas most relevant to the output trajectory of the prior policy during training, thereby reducing ineffective exploration. The proximal policy optimization algorithm also compensates for the shortcomings of the prior controller through learning. as well as Step 5: Based on the control action obtained in step 4, it is input into the controlled car to interact with the environment, and the observation information at the next moment after the interaction and the reward value after the previous step are obtained; The observation information in the next state is fed back to the gap algorithm module, and the observation information in the next state and the reward value after the previous step are fed back to the proximal strategy optimization algorithm module.

3. The autonomous racing car control method based on constrained residual reinforcement learning according to claim 2, characterized in that: Also includes: Step 1: Establish a partially observable Markov process corresponding to the autonomous racing problem, using tuples Indicates that is the state space, is the action space, is the observation set, is the reward function; State transition function Represents the state transition probability under a specific state and action; Observation function represents the probability of observation in a specific state; A finite trajectory is defined as ,in , Indicates the maximum time step; Reward signal Indicates; given a discount factor , the reinforcement learning control strategy optimizes the policy parameters To obtain the optimal strategy , to maximize the trajectory The expected cumulative reward on: The continuous action of the car is set to ,in is the steering angle input to the car, The power input to the car; Equipped with 2D laser radar for racing cars, the maximum detection range is ,have distance measurements, evenly distributed over within the field of vision; The racing car is equipped with a 60HZ inertial measurement unit to obtain vehicle motion information, including lateral acceleration, longitudinal acceleration, lateral speed and longitudinal speed.

4. The autonomous racing car control method based on constrained residual reinforcement learning according to claim 3, characterized in that: The gap method algorithm in step 2 includes: calculating the gap array from the lidar data, calculating the center angle of the maximum gap, and calculating the steering angle and power; Computing the gap array from lidar data involves: The point cloud observed by the lidar is regarded as a group of tiny obstacles, and the resulting point cloud sequence is By calculating the distance between each two adjacent points, the gap array corresponding to the observation data is obtained as , where N = 675; Calculating the center angle of maximum gap includes: Assume that the maximum gap is determined by the observation point composition, Represents the racing distance observation point distance, It is the angle; point is the maximum gap The center point is the target point, which is the farthest point detected by the lidar within the maximum gap. Yes Angle; The calculation formula for the gap center angle is: Calculating steering angle and power involves: According to the gap center angle and target angle Calculate the steering angle: in, is a weighting factor, It is the closest distance observed by the lidar; the combined structure depends on the closest distance to the surrounding obstacles and the size of the weighting factor; if the obstacle is near the vehicle, safety should be considered first, and the steering angle of the vehicle should be more biased towards the gap center angle , otherwise, the vehicle's steering angle should be more towards the target angle ; The formula for calculating the power input to the vehicle is: in, The farthest distance detected by the laser radar during vehicle driving. When , the vehicle maintains a low speed; in other cases, the power input to the vehicle is proportional to the farthest distance detected; the output action of the gap method algorithm is expressed as .

5. The autonomous racing car control method based on constrained residual reinforcement learning according to claim 4, characterized in that: Also includes: Step 3: Based on the problem definition in step 1, the policy network objective function of the proximal policy optimization algorithm is as follows: in, is the probability ratio describing the similarity between the new strategy and the old strategy; is a clipping function, is a hyperparameter, advantage function Defined as: in, Status The value function, advantage function is expressed in the state Next, an action Advantage relative to the average; when When The probability value of increases, resulting in Increase; however, too high This will lead to unstable policy learning, so the objective function Will Limited to no more than , similarly, when hour, Limit to no less than ; The ultimate goal of the proximal policy optimization algorithm is to maximize the following objective function: in, is the mean squared error loss of the value network, is the time step TD target; maximize It means reducing the error between the predicted value and the true value of the value network, making the prediction of the value network more and more accurate; Represents the entropy value of the strategy, maximizing Ensure adequate exploration of strategies; The goal of the autonomous car is to minimize the lap time of the car, using a reward function that combines discrete and continuous reward signals: in, It is the reward when the car completes a lap. It is linearly related to the speed of the car, encouraging the car to drive faster; the penalty term Penalize the car's excessive action changes in adjacent time steps to ensure smooth driving. The penalty term This reward function combines a discrete reward signal with a continuous reward signal, penalizing both collisions with the track and excessive action changes.

6. The autonomous racing car control method based on constrained residual reinforcement learning according to claim 5, characterized in that: Also includes: Step 4: By constraining the output of the proximal policy optimization algorithm policy network, the action fusion formula is obtained: in, is the output action of the prior controller, is the output action of the PPO policy network, is the constraint factor that constrains the output of the PPO policy network.

7. The autonomous racing car control method based on constrained residual reinforcement learning according to claim 6, characterized in that: Also includes: Step 5: Based on the control action obtained in step 4, input it into the controlled car to interact with the environment and obtain the observation information at the next moment after the interaction And the reward value after the previous step , the observation information in the next state Feedback to the FTG algorithm module, the observation information in the next state And the reward value after the previous step Feedback to the PPO algorithm module.

8. The autonomous racing car control method based on constrained residual reinforcement learning according to claim 7, characterized in that: Also includes: Considering the safe driving of the car in the prior control strategy, the car is guided to the center of the track as much as possible to ensure the safe driving of the car. The calculation method of the guidance action is as follows: By combining the maximum gap center angle and the target angle, the steering safety during the guidance action is ensured; when the maximum distance detected by the lidar does not exceed 3m, the input power is kept to the minimum to avoid collision.

9. The autonomous racing car control method based on constrained residual reinforcement learning according to claim 8, characterized in that: Also includes: Design an autonomous racing car control method that integrates dual control strategies. Select a proximal policy optimization algorithm with stable performance within the reinforcement learning control strategy. Also, design a reward function that combines discrete and continuous reward signals to make it easier to learn the optimal strategy. To minimize the car's lap time, the discrete reward value for completing a lap is combined with the continuous reward value for the car's speed. At the same time, collisions and changes in control actions are penalized to ensure a smoother trajectory. The reward function is obtained: 。 10. The autonomous racing car control method based on constrained residual reinforcement learning according to claim 9, characterized in that: Also includes: Design a fusion method for the prior control strategy and the reinforcement learning control strategy. By limiting the action output of the reinforcement learning strategy network relative to the prior strategy, ensure that the car only explores the area most relevant to the prior strategy, reducing ineffective exploration of the car. After training, the reinforcement learning control strategy can improve the performance of the prior control strategy and reduce the car's lap time. In step 4, design a fusion method for the output actions of the two strategies: In this formula, the output of the reinforcement learning control strategy network is subject to the constraint factor The constraint influence of the prior control strategy makes the proportion of the final control action lower than that of the prior control strategy output, ensuring that the prior strategy will not be covered by the reinforcement learning strategy in the early stage of exploration, reducing the occurrence of invalid exploration and dangerous actions.

Images