A Reinforcement Learning-Based Autonomous Driving Method for Racing Cars Based on Action Space Safety Mapping

Abstract

This invention discloses a reinforcement learning-based autonomous driving method for racing cars based on action space safety mapping. The technical solution employs a neural network to establish an autonomous driving controller for the racing car. The controller takes the car's own motion state data and the data from the track ahead as input, and outputs corresponding throttle / brake control and steering wheel control quantities. Based on the car's dynamic model, an action space safety mapping function is established, transforming the neural network output quantities that could lead the vehicle into safe control quantities before inputting them into the car's control system. Using a reinforcement learning algorithm, the neural network is trained based on reward feedback signals, continuously updating its parameters to ultimately obtain an optimized autonomous driving strategy for the racing car.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving, specifically a reinforcement learning-based autonomous driving method for racing cars based on action space safety mapping. Background Technology

[0002] With advancements in in-vehicle sensors, computing platforms, and planning and control algorithms, autonomous driving technology in automobiles has made significant progress. A large number of vehicles on the market now achieve Level 2 autonomous driving, automatically maintaining lane position and adjusting distance from the vehicle in front based on speed. Unlike general autonomous driving on ordinary roads, the goal of autonomous driving in motorsports is to complete the race in the shortest possible time, requiring the use of specialized steering and acceleration / deceleration control techniques to maximize the car's maneuverability. In recent years, with the development of reinforcement learning technology, artificial intelligence has surpassed human levels in video games and board games, but its application in autonomous driving for motorsports is relatively limited. This is mainly because reinforcement learning training involves a random exploration process, and random control inputs can lead the car into dangerous situations, posing safety risks to the strategy learning process. Therefore, to apply reinforcement learning to autonomous driving in motorsports to obtain excellent competitive driving strategies, it is essential to address the safety issues during training.

[0003] The technical differences compared to existing technologies are as follows:

[0004] With application number CN202211222692.5, application date: October 8, 2022. Patent title: A reinforcement learning method for autonomous driving based on approximate safe actions.

[0005] Both this application and the prior art employ reinforcement learning-based methods to learn autonomous driving strategies. However, this patent uses a different approach when dealing with safety issues arising from strategy exploration during the training process of autonomous driving strategies.

[0006] The comparison document uses a safety indicator method based on Lagrange optimal multipliers to modify the current strategy to obtain an approximately safe action. Specifically, in each control iteration, for the control action output by the current strategy, a correction value is calculated using the safety indicator and Lagrange optimal multipliers. This correction value is then used to modify the current action to obtain an approximately safe vehicle control quantity.

[0007] This application employs a method based on action space safety mapping to map the unrestricted control action space to a control quantity space with safety constraints, thereby ensuring the safety of the control quantities input to the racing car. Specifically, firstly, based on the racing car's dynamic model and safety constraints, a safety boundary function for the action space is constructed using numerical approximation methods. Then, the safety boundary function for the action space is transformed into an action space safety mapping function. In each control iteration, for the control action output by the actuator network, the action space safety mapping function is used to map the control action into a safe control quantity.

[0008] The difference between this application and the prior art lies in the following: the safety indicator used in the prior art requires parameter updates using experience samples from an experience pool during training, thus its safety indicator function can only be gradually improved during training. In contrast, the motion space safety mapping function used in this patent is directly established based on the dynamics model of the racing car, and does not require gradual improvement during training, thus better ensuring safety during training. Summary of the Invention

[0009] This invention addresses the application of reinforcement learning algorithms to the problem of autonomous driving in racing cars by proposing a reinforcement learning-based autonomous driving method for racing cars based on action space safety mapping. This method solves the safety problem caused by random exploration during the training of reinforcement learning-based racing car control algorithms.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] A reinforcement learning-based autonomous driving method for racing cars based on action space safety mapping includes the following steps:

[0012] Step S1: Establish the dynamics model and safety limits of the race car;

[0013] Step S2: Define the state, actions, and neural network of the racing car reinforcement learning control algorithm;

[0014] Step S3: Construct the motion space safety boundary function based on the racing car dynamics model;

[0015] Step S4: Construct the action space safety mapping function based on the action space safety boundary function;

[0016] Step S5: Use a reinforcement learning algorithm, combining a reward function and an action space mapping function, to train the driving strategy until a racing car autopilot controller that meets performance requirements is obtained.

[0017] As a further improvement of the present invention, the dynamic model of the race car in step S1 consists of a longitudinal dynamic model, a lateral dynamic model, a kinematic model, and a tire grip limiting model, wherein the longitudinal dynamic model of the race car is expressed as:

[0018]

[0019] Where m is the total mass of the race car; For the longitudinal acceleration of the race car; F tx The longitudinal traction or braking force of the racing car is controlled by the longitudinal control quantity u. x The decision, i.e., the decision based on the accelerator and brake signals; F aero The air resistance experienced by a racing car is determined by its aerodynamic parameters and speed; F roll The rolling friction force experienced by the racing car tires is determined by the mass of the racing car and the rolling resistance parameters of the tires. The lateral dynamics model of the racing car is expressed as follows:

[0020]

[0021] in For the lateral acceleration of the race car; F yf F represents the lateral force exerted on the front wheels of the racing car. yr The kinematic model of the race car, representing the lateral force acting on its rear wheels, is as follows:

[0022]

[0023]

[0024] in Let be the velocity components of the race car along the X and Y axes in the Cartesian coordinate system on the track plane; v be the car's speed; ψ be the angle in which the car's nose points; β be the sideslip angle of the race car, i.e., the angle between the angle in which the car's nose points and the actual direction of the car's motion; sin and cos are the sine and cosine functions, respectively. The sideslip angle β of the race car is expressed as:

[0025]

[0026] Where tan is the tangent function; arctan is the arctangent function; l r It is the distance from the race car's center of gravity to the rear axle; f δ is the distance from the race car's center of gravity to the front axle; δ is the front wheel steering angle, controlled by the lateral control variable u. y The decision, i.e., the steering wheel rotation signal determines the rate of change of the car's pointing angle. Represented as:

[0027]

[0028] For the longitudinal control variable, u x =1 indicates maximum acceleration, u x =-1 indicates that maximum deceleration is performed. For lateral control, uy =1 indicates turning the steering wheel to the left at the fastest speed, u y =-1 indicates turning the steering wheel to the right at the fastest speed, u y =0 indicates that the current steering wheel angle remains unchanged. The control quantity of the race car at time t is defined as u. t =[u x u y The tire grip limitation model for racing cars is represented as follows:

[0029]

[0030] in and These are the resultant force vectors of all the tires of the race car in its longitudinal and lateral directions. It is the vector of friction force between the racing tires and the ground. Let μ be the magnitude of the frictional force vector. max It is the maximum coefficient of friction between the racing tires and the track, F z It is the pressure of the race car on the track. To ensure the safe and normal handling of the race car, the friction force between the race car tires and the ground is... It cannot exceed its maximum grip force μ max F z If the car inputs too much control, causing the required friction to exceed the maximum grip, it will cause the car to sideslip or fishtail, enter a dangerous state, and lose control.

[0031] As a further improvement of the present invention, the state s of the reinforcement learning algorithm at time t in step S2 t Defined as:

[0032]

[0033] Where v represents the speed of the race car; d represents the rate of change of the vehicle's pointing angle. c φ represents the distance of the race car from the center line of the track; φ represents the angle between the car's pointing angle and the direction of the track from the car's current position; V FO It represents a set of vectors pointing from the center of gravity of the race car to the center line of the track at a certain distance in front of it;

[0034] In step S2, the action a of the reinforcement learning algorithm at time t t Defined as:

[0035] a t =[a x ,ay]

[0036] Where a x Indicates longitudinal control action, a yIndicates lateral control actions;

[0037] In step S2, the neural network of the reinforcement learning algorithm includes an executor network and an evaluator network. The executor network includes an input layer, two hidden layers, and an output layer, and its input is the state s. t Its output is action a. t The evaluator network consists of an input layer, two hidden layers, and an output layer, with the input being the state s. t and action a t Its output is the evaluation value of the current executor strategy.

[0038] As a further improvement of the present invention, the action space safety boundary function in step S3 is constructed using a numerical approximation method based on the dynamic model of the racing car. Firstly, the racing car control quantity u, which conforms to the tire grip limit of the racing car, is... t The racing car control actions output by the reinforcement learning actuator network. t Equivalently rewritten in polar coordinate vector form, i.e., u t =[ρ u θ u ], a t =[ρ a θ a ], where ρ u and ρ a θ represents the length of its corresponding vector. u and θ a This represents the rotation angle of the corresponding vector, discretizing the race car speed into N parts, i.e., v. i ={v1, v2, ..., v N Discretize the front wheel steering angle into N parts, i.e., δ j ={δ1, δ2, ..., δ N The rotation angles of the racing car control vector and the racing car control motion vector are also discretized into N parts, θ. k ={θ1, θ2, ..., θ N Then, the discretized state variables and discretized control vectors are input into the racing car kinematics model to find the maximum vector length that satisfies the tire grip limit at the corresponding vehicle speed, front wheel steering angle, and control vector rotation angle. Using the obtained data, a discrete motion space safety boundary function can be obtained, and then a continuous motion space safety boundary function Ψ is obtained through an interpolation algorithm, expressed as:

[0039]

[0040] in The length of the maximum control vector that a race car can apply without exceeding the tire grip limit, given a speed v, front wheel steering angle δ, and control vector angle θ.

[0041] As a further improvement of the present invention, the action space safety mapping function in step S4 maps the racing car control action space output by the actuator network to a racing car control quantity space that conforms to the racing car tire grip limit. The action space safety mapping function is constructed based on the action space safety boundary function constructed in step S3, and is expressed as follows:

[0042]

[0043] Specifically, when the race car is at a specific speed and front wheel steering angle, if the length of the control action vector output by the actuator network does not exceed the action safety boundary, the action space safety mapping function outputs the original control action vector given by the actuator network; if the length of the control action vector output by the actuator network exceeds the action safety boundary, the action space safety mapping function shortens the length of the original control action vector in the same direction, so that it just enters the safety boundary range.

[0044] As a further improvement of the present invention, the reinforcement learning algorithm used in step S5 employs a dual-delay deep deterministic policy gradient algorithm, and the reward function of the reinforcement learning algorithm used at time t is defined as follows:

[0045] r t =r vel +r our +r ww

[0046] Where r v el Speed ​​bonus, which is the component of the car's speed in the direction of the track; r out Penalty for going off the track; ww As a penalty for driving in the opposite direction, during the training of the driving strategy using this reinforcement learning method, at each control time step t, the actuator network inputs the current state s. t Output racing car control action a t =[a x a y Then first convert it to polar coordinate form a. t =[ρ a θ a Then, the motion space safety mapping function established in step S4 is used to transform it into a control quantity u that conforms to the racing car tire grip limit. t =[ρ u θ u Then convert it to rectangular coordinate form u. t =[u x u yFinally, the control quantity is transmitted to the actuator of the race car, and the control and strategy training iterations are repeatedly performed until a race car autopilot controller that meets the performance requirements is obtained.

[0047] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects:

[0048] The automatic driving method for racing cars based on action space safety mapping described in this invention combines the dynamic model of the racing car to establish action space safety boundary functions and action space safety mapping functions. The racing car control action vectors output by the reinforcement learning execution network are reasonably adjusted before being transmitted to the racing car's actuators. This avoids the dangerous situation caused by exceeding the tire grip limit due to directly using the reinforcement learning execution network to control the racing car. At the same time, it can also improve the learning speed of training automatic driving strategies for racing cars using reinforcement learning algorithms and obtain higher performance automatic racing car driving strategies. Attached Figure Description

[0049] Figure 1 Schematic diagram of racing car dynamics model;

[0050] Figure 2 Schematic diagram of racing car state vectors;

[0051] Figure 3 Network structure of reinforcement learning executor and evaluator, and data processing flow of reinforcement learning;

[0052] Figure 4 Schematic diagram of action space safety mapping function;

[0053] Figure 5 The racing car trajectory map obtained using the invented racing car autonomous driving strategy on test track 1;

[0054] Figure 6 The racing car trajectory map obtained on test track 2 using the invented autonomous driving strategy for racing cars. Detailed Implementation

[0055] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0056] The technical solution adopted in this invention is to use a neural network to establish an automatic driving controller for a racing car. The controller takes into account the car's own motion state data and the data from the track ahead, and outputs corresponding throttle / brake control and steering wheel control quantities. Based on the car's dynamic model, a motion space mapping function is established, transforming the neural network output quantities that might lead the vehicle into safe control quantities before inputting them into the car's control system. Using reinforcement learning algorithms, the neural network is trained based on reward feedback signals, continuously updating its parameters to ultimately obtain an optimized driving strategy.

[0057] Step S1: Establish the dynamic model and safety limits of the race car. For example... Figure 1 As shown, the race car is positioned in a Cartesian coordinate system, where ψ represents the angle at which the car's front is pointing, and F... yf F is the lateral force acting on the front wheels of the race car. yr The lateral force acting on the rear wheels of the race car, l r It is the distance from the center of gravity of the race car to the rear axle. f It is the distance from the race car's center of gravity to the front axle, and δ is the steering angle of the race car's front wheels.

[0058] v x It is the longitudinal speed of the race car, v y β is the lateral velocity of the race car, v is the combined velocity of the race car, and β is the sideslip angle of the race car, which is the angle between the pointing angle of the car and the actual direction of the race car's motion. The dynamic model of the race car consists of a longitudinal dynamic model, a lateral dynamic model, a kinematic model, and tire grip safety limits. The longitudinal dynamic model of the race car is expressed as:

[0059]

[0060] Where m is the total mass of the race car; For the longitudinal acceleration of the race car; F tx The longitudinal traction or braking force of the racing car is controlled by the longitudinal control quantity u. x (i.e., accelerator and brake signals) determine; F aero The air resistance experienced by a racing car is determined by its aerodynamic parameters and speed; F roll The rolling friction force experienced by the racing car tires is determined by the mass of the racing car and the rolling resistance parameters of the tires. The lateral dynamics model of the racing car is expressed as:

[0061]

[0062] in For the lateral acceleration of the race car; F yf F represents the lateral force exerted on the front wheels of the racing car. yr This represents the lateral force acting on the rear wheels of the race car. The kinematic model of the race car is represented as:

[0063]

[0064]

[0065] in Let be the velocity components of the race car along the X and Y axes in a Cartesian coordinate system on the track plane; v be the car's speed; ψ be the angle the car is pointing; β be the sideslip angle, which is the angle between the car's pointing angle and its actual direction of motion; sin and cos are the sine and cosine functions, respectively. The sideslip angle β of the race car is expressed as:

[0066]

[0067] Where tan is the tangent function; arctan is the arctangent function; l r It is the distance from the race car's center of gravity to the rear axle; f δ is the distance from the race car's center of gravity to the front axle; δ is the front wheel steering angle, controlled by the lateral control variable u. y Determined by (i.e., the steering wheel rotation signal). The rate of change of the car's pointing angle. Represented as:

[0068]

[0069] For the longitudinal control variable, u x =1 indicates maximum acceleration, u x =-1 indicates maximum deceleration. For lateral control variables, u y =1 indicates turning the steering wheel to the left at the fastest speed, u y =-1 indicates turning the steering wheel to the right at the fastest speed, u y =0 indicates that the current steering wheel angle remains unchanged. The control quantity of the race car at time t is defined as u. t =[u x u y The safety limits for tire grip on racing cars are expressed as follows:

[0070]

[0071] in and These are the resultant force vectors of all the tires of the race car in its longitudinal and lateral directions. It is the vector of friction force between the racing tires and the ground. Let μ be the magnitude of the frictional force vector. max It is the maximum coefficient of friction between the racing tires and the track, F z It refers to the pressure exerted by the race car on the track. To ensure the safe and proper handling of the race car, the friction force between the race car tires and the track is controlled. It cannot exceed its maximum grip force μ max F z If the car is given too much control input, causing the required friction to exceed the maximum grip, it will cause the car to sideslip or fishtail, enter a dangerous state, and lose control.

[0072] Step S2, define the state, actions, and neural network of the racing car reinforcement learning control algorithm. For example... Figure 2 v represents the composite speed of the race car, d c This indicates the distance of the race car from the center line of the track, and φ represents the angle between the car's pointing angle and the direction of the track from the car's current position. Let represent the vector of the race car pointing from its center of gravity towards a point on the track's centerline a certain distance ahead. The state s of the reinforcement learning algorithm at time t. t Defined as:

[0073]

[0074] in This indicates the rate of change of the vehicle's pointing angle; This represents a set of vectors pointing from the center of gravity of the race car towards a point on the center line of the track a certain distance ahead. The action 'a' of the reinforcement learning algorithm at time t... t Defined as:

[0075] a t =[a x a y ]

[0076] Where a x Indicates longitudinal control action, a y This represents lateral control actions. Reinforcement learning neural networks consist of an actuator network and an evaluator network, and their structure is as follows: Figure 3 As shown, the input layer of the actuator network takes state s as its input. t The activation function is ReLU. The input layer is sequentially connected to two hidden layers, each with 256 nodes. The activation functions of the two hidden layers are ReLU and Tanh, respectively. The second hidden layer is connected to the output layer, and the output is action 'a'. t The input layer of the evaluator network takes state s as input. t and action a t The input layer is formed by concatenating two vectors and using ReLU as the activation function. The input layer is sequentially connected to two hidden layers. The first hidden layer has the ReLU activation function, while the second hidden layer has no activation function and is directly connected to the output layer. The output is the evaluation value of the current executor policy.

[0077] Step S3: Construct the motion space safety boundary function using numerical approximation based on the race car dynamics model. First, the race car control quantity u, which conforms to the race car tire grip limits, is... t The racing car control actions output by the reinforcement learning actuator network. t Equivalently rewritten in polar coordinate vector form, i.e., u t =[ρ u θ u], a t =[ρ a θ a ], where ρ u and ρ a θ represents the length of its corresponding vector. u and θ a This represents the rotation angle of its corresponding vector. The race car speed is discretized into N parts, i.e., v i ={v1, v2, ..., v N Discretize the front wheel steering angle into N parts, i.e., δ j ={δ1, δ2, ..., δ N The rotation angles of the racing car control vector and the racing car control motion vector are also discretized into N parts, θ. k ={θ1, θ2, ..., θ N Then, the discretized state variables and discretized control vectors are input into the racing car kinematics model to find the maximum vector length that satisfies the tire grip limit at the corresponding vehicle speed, front wheel steering angle, and control vector rotation angle. Using the obtained data, a discrete motion space safety boundary function can be obtained, and then a continuous motion space safety boundary function Ψ is obtained through an interpolation algorithm, expressed as:

[0078]

[0079] in The length of the maximum control vector that a race car can apply without exceeding the tire grip limit, given a speed v, front wheel steering angle δ, and control vector angle θ.

[0080] Step S4: Based on the action safety boundary function constructed in step S3, construct the action space safety mapping function, expressed as:

[0081]

[0082] Its function is to map the racing car control action space output by the actuator network to a racing car control quantity space that conforms to the grip limitations of the racing car's tires. Specifically, at a specific speed and front wheel steering angle, when the length of the control action vector output by the actuator network does not exceed the action safety boundary, the action space safety mapping function outputs the original control action vector given by the actuator network. When the length of the control action vector output by the actuator network exceeds the action safety boundary, the action space safety mapping function shortens the length of the original control action vector in the same direction, bringing it just within the safety boundary range. For example... Figure 4 In the given example, when the race car speed is v = 15.4 m / s and the front wheel steering angle is δ = 7.9°, the safety boundary for the action is as follows: Figure 4 As shown on the right. If the action vector output by the network is at1 If the action exceeds the safety boundary, it will be mapped to u through the action safety mapping function. t1 The vector direction remains unchanged, but its length is shortened to within the safe boundary. If the action vector output by the network is a... t2 If the action does not exceed the safety boundary and there is no safety risk, then no mapping processing is required, i.e., u t2 =a t2 .

[0083] Step S5: Using a reinforcement learning algorithm, combining the reward function and the action space mapping function, the driving strategy is trained until a racing car autonomous driving controller that meets the performance requirements is obtained. The reinforcement learning algorithm used in this invention is the Twin Delayed Deep Deterministic Policy Gradient algorithm. The reward function of the reinforcement learning algorithm used at time t is defined as:

[0084] r t =r vel +r out +r ww

[0085] Where r vel Speed ​​bonus, which is the component of the car's speed in the direction of the track; r out Penalty for going off the track; ww Penalty for driving in the opposite direction. For example... Figure 3 The algorithm data processing flow shown indicates that at each control time step t, the actuator network inputs the current state s. t Output racing car control action a t =[a x a y Then first convert it to polar coordinate form a. t =[ρ a θ a Then, the motion space safety mapping function established in step S4 is used to transform it into a control quantity u that conforms to the racing car tire grip limit. t =[ρ u θ u Then convert it to rectangular coordinate form u. t =[u x u y This control input is then transmitted to the race car's actuators. Control and strategy training iterations are repeated until a race car autopilot controller that meets performance requirements is obtained.

[0086] The autonomous driving strategy for racing cars trained using the method described in this invention was tested on test track 1 and test track 2, and the trajectories are as follows. Figure 5 , Figure 6 As shown, the speeds marked on the track and corresponding positions demonstrate that the autonomous driving strategy of the racing car can fully utilize the geometric features of the track to select a smoother driving route and achieve faster exit speeds from corners. The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any modifications or equivalent variations made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A racing car reinforcement learning autonomous driving method based on action space safety mapping, comprising the following steps, characterized in that: Step S1, establish the dynamic model and safety limits of the racing car; Step S2, define the state, action and neural network of the racing car reinforcement learning control algorithm; Step S3, construct an action space safety boundary function according to the racing car dynamic model; The action space safety boundary function in step S3 is constructed by means of numerical approximation based on the dynamic model of the racing car. First, the control quantity of the racing car that meets the grip limit of the racing car tires and the racing car control action output by the reinforcement learning actuator network are equivalently rewritten in the form of polar coordinate vectors, that is , , where and represent the lengths of their corresponding vectors, and represent the rotation angles of their corresponding vectors. The racing car speed is discretized into N parts, that is , the front wheel steering angle is discretized into N parts, that is , and the rotation angles of the racing car control quantity vector and the racing car control action vector are also discretized into N parts respectively, , and then the discretized state quantity and the discretized control quantity vector are brought into the kinematic model of the racing car to find the maximum vector length that meets the tire grip limit at the corresponding vehicle speed, front wheel steering angle, and control quantity vector rotation angle. The discrete action space safety boundary function can be obtained by using the data obtained, and then the continuous action space safety boundary function can be obtained through the interpolation algorithm , which is expressed as: ； where is the maximum control amount vector length that can be applied without exceeding the tire grip limit when the racing car is at speed , the front wheel steering angle , and the control vector angle . Step S4, construct an action space safety mapping function according to the action space safety boundary function; Step S5, use the reinforcement learning algorithm, combine the reward function and the action space mapping function to train the driving strategy until a racing car autonomous driving controller that meets the performance requirements is obtained.

2. The racing car reinforcement learning autonomous driving method based on action space safety mapping according to claim 1, characterized in that In step S1, the dynamic model of the racing car is composed of a longitudinal dynamic model, a lateral dynamic model, a kinematic model and a tire grip limit model, wherein the longitudinal dynamic model of the racing car is expressed as: ； where is the total mass of the racing car; is the longitudinal acceleration of the racing car; is the longitudinal traction force or braking force of the racing car, which is determined by the longitudinal control quantity , that is, determined by the throttle and brake signals; is the air resistance suffered by the racing car, which is determined by the aerodynamic parameters and vehicle speed of the racing car; is the rolling friction force suffered by the racing car tires, which is determined by the mass of the racing car and the rolling resistance parameters of the tires. The lateral dynamic model of the racing car is expressed as: ； where is the lateral acceleration of the racing car; is the lateral force on the front wheels of the racing car; is the lateral force on the rear wheels of the racing car, and the kinematic model of the racing car is expressed as: ； where are the velocity components of the racing car along the X-axis and Y-axis in the rectangular coordinate system of the track plane; is the vehicle speed of the racing car; is the angle of the vehicle's nose; is the sideslip angle of the racing car, that is, the angle between the angle of the vehicle's nose and the actual movement direction of the racing car; are the sine function and cosine function respectively. The sideslip angle of the racing car is expressed as: ； where tan is the tangent function; arctan is the arctangent function; is the distance from the center of gravity of the racing car to the rear axle; is the distance from the center of gravity of the racing car to the front axle; is the steering angle of the front wheels of the racing car, which is determined by the lateral control quantity That is, it is determined by the steering wheel rotation signal, and the change rate of the vehicle head pointing angle is expressed as: ； For the longitudinal control quantity, Indicates to perform maximum acceleration, Indicates to perform maximum deceleration. For the lateral control quantity, Indicates to turn the steering wheel to the left at the fastest speed, Indicates to turn the steering wheel to the right at the fastest speed, Indicates to keep the current steering wheel angle unchanged. The control quantity of the racing car at time t is defined as , and the tire grip limit model of the racing car is expressed as: ； where and are the resultant force vectors of all the tires of the racing car in its longitudinal and transverse directions respectively, is the frictional force vector between the racing car tires and the ground, is the magnitude of this frictional force vector, is the maximum friction coefficient between the racing car tires and the track, is the pressure of the racing car on the track. To ensure the normal and safe operation of the racing car, the magnitude of the frictional force between the racing car tires and the ground shall not exceed the maximum grip it can withstand . If the racing car inputs an excessive control amount, resulting in the required frictional force exceeding the maximum grip, the racing car will skid or spin out, entering a dangerous state and losing control.

3. The racing car reinforcement learning autonomous driving method based on action space safety mapping according to claim 1, characterized in that The state of the reinforcement learning algorithm at time t in step S2 is defined as: ； Among them represents the speed of the racing car; represents the rate of change of the angle of the vehicle's nose; represents the distance of the racing car from the center line of the track; represents the angle between the angle of the vehicle's nose and the direction of the track where the racing car is located; represents a set of a series of vectors from the center of gravity of the racing car to the points on the center line of the track at a certain distance ahead; The action of the reinforcement learning algorithm at time t in step S2 is defined as: ； Among them represents a longitudinal control action represents a lateral control action In step S2, the neural network of the reinforcement learning algorithm includes an actor network and a critic network. The actor network includes an input layer, two hidden layers, and an output layer. Its input is the state , and its output is the action . The critic network includes an input layer, two hidden layers, and an output layer. Its input is the state and the action , and its output is the evaluation value of the current actor policy.

4. The racing car reinforcement learning autonomous driving method based on action space safety mapping according to claim 1, characterized in that The action space safety mapping function in step S4 is used to map the racing car control action space output by the actuator network to the racing car control quantity space that meets the racing car tire grip limit. The action space safety mapping function is constructed based on the action space safety boundary function constructed in step S3 and is expressed as: ； Specifically, when the racing car is at a specific speed and front wheel steering angle, when the length of the control action vector output by the actuator network does not exceed the action safety boundary, the action space safety mapping function outputs the original control action vector given by the actuator network; when the length of the control action vector output by the actuator network exceeds the action safety boundary, the action space safety mapping function shortens the length of the original control action vector in the same direction so that it just enters the safety boundary range.

5. The racing car reinforcement learning autonomous driving method based on action space safety mapping according to claim 1, characterized in that The reinforcement learning algorithm used in step S5 uses the Twin Delayed Deep Deterministic Policy Gradient algorithm, and the reward function of the reinforcement learning algorithm used is defined at time t as: ； where is the speed reward, i.e., the component of the racing car's speed in the track direction; is the penalty for going off the track; is the penalty for driving in the opposite direction. During the process of using this reinforcement learning for driving strategy training, at each control time step t, the actuator network inputs the current state , outputs the racing car control action , then first converts it into polar coordinate form , then uses the action space safety mapping function established in step S4 to transform it into a control quantity that conforms to the grip limit of the racing car tires , and then converts it back to rectangular coordinate form . Finally, this control quantity is transmitted to the actuator of the racing car, and the control and policy training iterations are repeatedly executed until a racing car automatic driving controller that meets the performance requirements is obtained.

Images