Airport guiding vehicle automatic parking method based on reinforcement learning

Abstract

The application provides an airport guide vehicle automatic parking method based on reinforcement learning, which comprises the following steps: step 1, setting the vehicle contour coordinates and the parking coordinate system of the airport guide vehicle; step 2, establishing a vehicle kinematics model in combination with the parking coordinate system; step 3, establishing a vehicle collision determination mechanism; step 4, setting the state space and the action space of an agent; step 5, setting an instant reward return mechanism of the agent in the movement process; step 6, improving the experience playback mechanism and the agent exploration mechanism in the algorithm; and step 7, training the improved automatic parking algorithm framework in combination with the airport apron layout, inputting the obtained real-time acceleration and steering angle into the vehicle kinematics model, and realizing automatic parking control. The application has a short training cycle and a relatively optimal learning effect, so that the vehicle can autonomously park in the limited space under the complex environment of the airport apron, thereby reducing the burden of the ground crew and avoiding causing safety accidents.

Description

Technical Field

[0001] This invention relates to an automatic parking method for airport guidance vehicles based on reinforcement learning, belonging to the field of intelligent driving technology. Background Technology

[0002] With the rapid development of the global aviation industry, the complexity and frequency of apron traffic management have increased accordingly. Automated parking technology, as a crucial component of intelligent ground traffic management, is gradually becoming a key requirement for airport guidance vehicle operations. Based on real-time information from the vehicle's environment, it plans the driving path at each moment and precisely controls speed and steering by combining its own status information. This effectively avoids parking accidents caused by driver inexperience, thereby reducing waiting time for guidance vehicles and ensuring the normal operation of aircraft and related ground facilities. Automated parking technology originated in the early 1990s, employing an algorithm structure that combines path planning with trajectory tracking control. After identifying a parking space, a parking path is first planned using geometric splicing or numerical fitting, followed by traditional control methods to track and control the parking path. However, the parking process involves multiple control variables such as speed and acceleration. Early automated parking methods typically relied on a series of preset rule-based control algorithms. As the parking environment and demands change, parameters need to be redesigned or adjusted, lacking a certain degree of generalization ability.

[0003] In contrast, advancements in computer science and artificial intelligence have facilitated the application of deep reinforcement learning. Control algorithms based on deep reinforcement learning can learn optimal policies through interaction with the environment. Especially after training with a large amount of diverse data, the agent can gradually adapt to various unknown environments and output correct actions. Among these, the Deep Deterministic Policy Gradient (DDPG) algorithm is widely used in continuous and high-precision control tasks such as automated parking due to its excellent stability and outstanding ability to handle continuous states and action spaces. This algorithm, combined with an experience playback mechanism, employs an offline learning method, breaking the correlation of learning samples while ensuring the stability of the training process. It utilizes the learning capabilities of deep neural networks to handle the dynamic relationship between the high-dimensional real-time state and execution decisions of the vehicle in the parking environment. By continuously adjusting its internal parameters during network training, the output action minimizes prediction errors and maximizes task benefits, thereby achieving safe and stable parking operations.

[0004] The presence of various ground service facilities on the tarmac, such as jet bridges, power supply vehicles, and towing vehicles, results in a complex parking environment for airport guidance vehicles with limited available space. Therefore, using the DDPG algorithm to complete the parking task for guidance vehicles typically suffers from the following drawbacks:

[0005] (1) The feedback of the parking environment is usually highly random, while the capacity of the experience replay pool in the algorithm is limited. The agent may generate a large number of invalid experience samples during the training process, which will cause the successful samples to be overwhelmed by updates, resulting in an excessively long training cycle and poor parking quality.

[0006] (2) The algorithm adopts a deterministic strategy. The agent lacks sufficient exploration ability in the early stage of training. The added environmental exploration signals may interfere with the agent's later strategy learning, causing the agent's training process to oscillate and the learning efficiency to be low. Summary of the Invention

[0007] To address the technical problem of poor parking quality in existing technologies, this invention proposes an automatic parking method for airport guidance vehicles based on reinforcement learning.

[0008] The technical solution adopted by this invention to solve the above problems is as follows: This invention proposes an automatic parking method for airport guidance vehicles based on reinforcement learning, comprising:

[0009] Step 1: Set the vehicle outline coordinates and parking coordinate system for the airport guide vehicle;

[0010] Step 2: Establish a vehicle kinematic model using the parking coordinate system;

[0011] Step 3: Set up the vehicle collision detection mechanism;

[0012] Step 4: Set the vehicle's motion state space and action space;

[0013] Step 5: Set up an instant reward mechanism for the agent during its movement;

[0014] Step 6: Construct an automatic parking algorithm framework based on deep reinforcement learning, and improve the experience playback mechanism and agent exploration mechanism in the algorithm to obtain the improved automatic parking algorithm framework.

[0015] Step 7: Train the improved automatic parking algorithm framework based on the apron layout to obtain the real-time acceleration and steering angle of the airport guide vehicle. Input the real-time acceleration and steering angle into the vehicle kinematics model to realize automatic parking control.

[0016] Optionally, the vehicle outline coordinates of the airport guide vehicle in step 1 include: vehicle outline vertices A, B, C, and D; the rear axle center point o(x,y) as the motion reference point; F and R as the projections of the front and rear axles onto the ground; and L. f L r L is the front and rear overhang length, L is the wheelbase, δ is the front wheel steering angle, φ is the angle between the vehicle body and the horizontal direction, V is the vehicle speed, and the parking coordinate system is xoy.

[0017] Optionally, the expression for the vehicle kinematics model in step 2 is:

[0018]

[0019] Optionally, the vehicle collision determination mechanism in step 3 specifically includes:

[0020] Define the conditions that the coordinates of the vehicle's outline vertices must satisfy at the time of a collision, and the conditions that the area of ​​the geometric figure constructed around the collision point P must satisfy.

[0021] The conditions for satisfying the coordinates of the vehicle's contour vertices are:

[0022]

[0023] In formula (2), (x i ,y i (x) represents the coordinates of the vehicle's vertex. obstacle ,y obstacle () represents the coordinates of the obstacle;

[0024] The area of ​​the geometric figure constructed around the collision point P satisfies the following condition:

[0025]

[0026] Optionally, in step 4, the agent's state space is s = (x, y, V, φ), and its action space is... in, Let δ be the vehicle acceleration and δ be the front wheel steering angle.

[0027] Optionally, the instant reward mechanism for the agent during movement in step 5 includes:

[0028] Positive rewards for guiding a vehicle to a parking space, target rewards for completing a parking maneuver, and penalties for a vehicle collision.

[0029] The expression for the positive reward for guiding a vehicle to a parking space is:

[0030]

[0031] In formula (4), (x T ,y T () indicates the target parking location;

[0032] The expression for the target reward obtained when parking a vehicle is completed is:

[0033]

[0034] In formula (5), R base Let ξ be the positive basic reward constant, and ξ be the time penalty factor;

[0035] The expression for the penalty obtained when a vehicle collides is:

[0036] r collision =-R base (6).

[0037] Optionally, the improved automatic parking algorithm framework in step 6 includes:

[0038] Actor decision-making module, Critic evaluation module, deep reinforcement learning environment, experience replay pool;

[0039] The experience replay pool is used to provide experience samples, which include the current state, actions, and rewards of the airport guidance vehicle and the state of the airport guidance vehicle in the next moment. The state of the airport guidance vehicle includes the lateral and longitudinal movement distance, heading angle, and speed. The actions of the airport guidance vehicle include the wheel steering angle and acceleration.

[0040] Actor decision-making module, Critic evaluation module, deep reinforcement learning environment, experience replay pool;

[0041] The experience replay pool is used to provide the experience samples needed for the agent to learn. The experience samples include the current state, action information, reward of the airport guide vehicle, and the state of the airport guide vehicle at the next moment. The state of the airport guide vehicle includes the vehicle's lateral and longitudinal position information, heading angle, and speed. The action information of the airport guide vehicle includes the wheel steering angle and acceleration. The reward is the cumulative reward for the agent's actions, calculated by combining the immediate reward with the future reward adjusted by the discount factor γ, and is used to incentivize the agent to achieve long-term goals.

[0042] The Critic evaluation module is used to output the evaluation value of the actions performed by the agent;

[0043] The Actor decision module is used to output the actions performed by the intelligent agent, including the acceleration and steering angle of the airport guidance vehicle.

[0044] The Actor decision-making module and the Critic evaluation module are used for policy generation and value estimation, and their interaction improves the learning efficiency and policy quality of the agent.

[0045] Optionally, the steps in step 6 to improve the experience replay mechanism and the agent exploration mechanism include:

[0046] Step 6.1: Update the experience samples in the experience replay pool based on the Actor decision module and the Critic evaluation module, output the action information of the airport guidance vehicle at the current moment, input the action information of the airport guidance vehicle at the current moment into the kinematic model to obtain the state information of the airport guidance vehicle, combine the state information of the airport guidance vehicle at each moment with the real-time reward mechanism to calculate the cumulative reward of the agent, and finally store the updated information as experience samples in the experience replay pool.

[0047] Step 6.2: Guide the agent to park within the set motion state space based on the updated experience samples in the experience replay pool;

[0048] Step 6.3: Design a priority experience replay mechanism to filter the updated experience samples, obtain high-quality experience samples, and complete the improvement of the experience replay mechanism and the agent exploration mechanism.

[0049] Optionally, obtaining high-quality experience samples in step 6.3 specifically includes:

[0050] Calculate the mean and variance of the instant reward results for all updated experience samples, and design a priority P based on the variance and mean of the instant reward results. i High-quality experience samples were obtained through screening;

[0051] Priority P i The expression is:

[0052]

[0053] In formula (7), r i As an empirical sample, Let be the mean reward for each round, and α and β be the weights of the mean and variance of the rewards, respectively.

[0054] Optionally, step 7, which involves measuring the real-time acceleration and steering angle of the airport guidance vehicle, includes:

[0055] Step 7.1: Based on the selected high-quality experience, the gradient optimization method is used to iteratively calculate the parameters of the Actor and Critic evaluation modules, update the Actor and Critic evaluation modules, and design an environment exploration decay mechanism during the training process to complete the training of the improved automatic parking algorithm framework.

[0056] Step 7.2: Extract the action output results of the Actor decision module after training, and input the action output results into the vehicle kinematics model to realize automatic parking control. The action output results include acceleration. and steering angle δ;

[0057] The expressions for gradient optimization and parameter update rate in the Critic evaluation module are as follows:

[0058]

[0059] In formula (8), η is the network learning rate, and M is the number of experiences selected in the batch of experiences;

[0060] The expressions for gradient optimization and parameter update rate in the Actor decision module are:

[0061]

[0062] The expression for the environmental exploration decay mechanism is:

[0063]

[0064] In formula (10), σ0 is the initial standard deviation and ε is the decay rate.

[0065] The beneficial effects of this invention are:

[0066] (1) The automatic parking method provided by this invention has a complete structure, and with the priority experience replay mechanism, it can effectively screen high-quality experience samples for agent learning. It solves the problems of excessively long training cycle of parking algorithms and poor parking quality (pose accuracy, time consumption and motion state).

[0067] (2) The environmental exploration decay mechanism provided by this invention can balance the exploration of the parking environment and the utilization of effective strategies by the agent during the learning process, thus solving the problems of training fluctuation and low learning efficiency of parking algorithms.

[0068] (3) The present invention has a shorter training cycle and better learning effect, enabling vehicles to perform autonomous parking operations in a limited space in the complex environment of the tarmac, thereby reducing the burden on ground staff and avoiding safety accidents. Attached Figure Description

[0069] Figure 1 A flowchart of the airport guidance vehicle automatic parking method based on reinforcement learning provided by the present invention;

[0070] Figure 2 A simplified diagram of the vehicle model provided by this invention;

[0071] Figure 3 A schematic diagram of a vehicle collision provided for this invention;

[0072] Figure 4 The structural diagram of the parking frame based on the DDPG algorithm provided by this invention;

[0073] Figure 5A flowchart of the priority experience replay mechanism based on reward features provided by the present invention;

[0074] Figure 6 The experience screening flowchart provided for this invention;

[0075] Figure 7 This is a schematic diagram of an airport guidance vehicle parking scenario provided by the present invention;

[0076] Figure 8 The cumulative reward curve for the reinforcement learning-based automatic parking method for airport guidance vehicles provided by this invention;

[0077] Figure 9 Simulation results of the airport guidance vehicle automatic parking method based on reinforcement learning provided by this invention;

[0078] Figure 10 A graph showing the change in vehicle motion state during the automatic parking process provided by the present invention;

[0079] Figure 11 This invention provides a diagram showing the parking posture error distribution during automatic parking. Example

[0080] Combination Figure 1-11 This embodiment will be described as follows: Figure 1 As shown, the steps of the airport guidance vehicle automatic parking method based on reinforcement learning described in this embodiment include:

[0081] S1: Set the vehicle outline coordinates and parking coordinate system of the airport guide vehicle;

[0082] like Figure 2 As shown, this embodiment constructs a simplified vehicle model diagram based on the vehicle outline coordinates and the parking coordinate system.

[0083] S2: Establish a vehicle kinematic model by combining the parking coordinate system;

[0084] The expression for the vehicle's kinematic model is:

[0085]

[0086] S3: Set the vehicle collision determination mechanism;

[0087] like Figure 3 As shown, this embodiment sets the conditions for satisfying the coordinates of the vehicle outline vertices when a collision occurs and the conditions for satisfying the area of ​​the geometric shape constructed around the collision point P.

[0088] The conditions for satisfying the coordinates of the vehicle's contour vertices are:

[0089]

[0090] In formula (2), (x i ,y i (x) represents the coordinates of the vehicle's vertex. obstacle ,y obstacle () represents the coordinates of the obstacle;

[0091] The area of ​​the geometric figure constructed around the collision point P satisfies the following condition:

[0092]

[0093] S4: Set the vehicle's motion state space and action space;

[0094] The parking space parameters in this embodiment model are shown in Table 1, and the vehicle parameters are shown in Table 2.

[0095] Table 1

[0096]

[0097]

[0098] Table 2

[0099]

[0100] S5: Set up an instant reward mechanism for the agent during movement;

[0101] S501: Establish a parking coordinate system xOy with the midpoint of the lower boundary of the parking space as the origin. The parking scene layout is shown in Table 3. When the vehicle position enters the circular region: (x+0.425) 2 +(y-1.5) 2 ≤0.09, and vehicle orientation deviation Parking was deemed successful;

[0102] Table 3

[0103]

[0104] S502: Instant reward for computational agents R s and cumulative reward returns;

[0105] This embodiment calculates the agent's immediate reward R based on setting a positive reward for guiding the vehicle to the parking space, a target reward for the vehicle when parking is completed, and a penalty for the vehicle when a collision occurs. s and cumulative reward returns;

[0106] The expression for the positive reward for guiding a vehicle to a parking space is:

[0107]

[0108] In formula (4), (x T ,y T () indicates the target parking location;

[0109] The expression for the target reward obtained when parking a vehicle is completed is:

[0110]

[0111] In formula (5), R base Let ξ be the positive basic reward constant, and ξ be the time penalty factor;

[0112] The expression for the penalty obtained when a vehicle collides is:

[0113] r collision =-R base (6).

[0114] Instant reward R for the agent s The expression is:

[0115] R s =ω1·r guide +ω2·r end +ω3·r collision (7);

[0116] In formula (7), the basic reward R base =15, time penalty factor ξ=0.5; weights ω1=0.3, ω2=0.2, ω3=0.5. The expression for calculating the cumulative reward is:

[0117]

[0118] In formula (8), the discount factor γ = 0.95 and the time step per round t = 100000.

[0119] S6: Design an improved automatic parking algorithm framework to achieve automatic parking control;

[0120] S601: As Figure 4 As shown, the improved automatic parking algorithm framework in this embodiment is based on the DDPG algorithm-based automatic parking framework, with the addition of a Critic network. The experience samples in the experience replay pool are updated based on the Actor network and the Critic network.

[0121] The algorithm parameters in this embodiment are shown in Table 4;

[0122] Table 4

[0123]

[0124]

[0125] S602: Based on the Actor decision module and the Critic evaluation module, update the experience samples in the experience replay pool, output the action information of the airport guidance vehicle at the current moment, input the action information of the airport guidance vehicle at the current moment into the kinematic model to obtain the state information of the airport guidance vehicle, combine the state information of the airport guidance vehicle at each moment with the instant reward mechanism to calculate the cumulative reward of the agent, and finally store the updated information as experience samples in the experience replay pool.

[0126] S603: As Figure 5 As shown, regarding the priority experience replay mechanism in the parking algorithm of this invention, this invention designs the minimum number of experiences Y required to meet the extraction conditions. min =10000, the number of experience samples drawn each time is b=500, and the updated experience samples are filtered. The filtering process is as follows: Figure 6 As shown, high-quality experience samples are obtained. This invention, combined with a priority experience replay mechanism, can effectively filter high-quality experience samples for agent learning. It solves the problems of excessively long training cycles and poor parking quality (pose accuracy, time consumption, and motion state) in parking algorithms.

[0127] Priority P i The expression is:

[0128]

[0129] In formula (9), the weights corresponding to the mean and variance of the reward are α = 0.75 and β = 0.25.

[0130] S604: Based on the selected high-quality experience, gradient optimization is used to iteratively calculate the parameters of the Actor and Critic evaluation modules, completing the update of the Actor and Critic evaluation modules. An environment exploration decay mechanism is designed during training to complete the training of the improved automatic parking algorithm framework. The action output results of the Actor network after training are extracted, including acceleration. and steering angle δ;

[0131] The expressions for gradient optimization and parameter update rate of the Critic network are:

[0132]

[0133] In formula (10), η is the network learning rate, and M is the number of experiences selected in the batch of experiences;

[0134] The expressions for gradient optimization and parameter update rate of the Actor network are:

[0135]

[0136] The expression for the environmental exploration decay mechanism is:

[0137]

[0138] In formula (12), σ0 is the initial standard deviation and ε is the decay rate.

[0139] S605: Input the action output of the Actor network into the vehicle kinematics model to achieve automatic parking control.

[0140] In this embodiment, after completing the parameter settings for the automatic parking method, Figure 7 Parking training is conducted in the scenario shown, and the training effect is as follows: Figure 8 As shown in Table 5, after training, to verify the learning effect and generalization ability of the parking algorithm, this embodiment randomly selects three different vehicle starting poses within the vehicle's starting area for parking tests. The obtained vehicle starting poses are shown in Table 5, and the parking results are as follows: Figure 9-11 As shown, this invention has a smaller error than the traditional DDP G algorithm, and has a shorter training cycle and better learning effect, enabling vehicles to perform autonomous parking operations in a limited space in the complex environment of the tarmac, thereby reducing the burden on ground staff and avoiding safety accidents.

[0141] Table 5

[0142]

[0143] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the scope of the present invention, based on the technical essence of the present invention and within the spirit and principles of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. An automatic parking method for airport guidance vehicles based on reinforcement learning, characterized in that, The steps of the reinforcement learning-based airport guidance vehicle automatic parking method include: Step 1: Set the vehicle outline coordinates and parking coordinate system for the airport guide vehicle; Step 2: Establish a vehicle kinematic model using the parking coordinate system; Step 3: Set up the vehicle collision detection mechanism; The vehicle collision determination mechanism in step 3 specifically includes: Define the conditions that the coordinates of the vehicle's outline vertices must satisfy at the time of a collision, and the conditions that the area of ​​the geometric figure constructed around the collision point P must satisfy. The conditions for satisfying the coordinates of the vehicle's contour vertices are: （2）； In formula (2), The coordinates of the vehicle's vertices. The coordinates of the obstacle; The area of ​​the geometric figure constructed around the collision point P satisfies the following condition: （3）； Step 4: Define the state space and action space of the agent; Step 5: Set up an instant reward mechanism for the agent during its movement; The instant reward mechanism for the agent during the movement process in step 5 includes: Positive rewards for guiding a vehicle to a parking space, target rewards for completing a parking maneuver, and penalties for a vehicle collision. The expression for the positive reward for guiding a vehicle to a parking space is: （4）； In formula (4), The target parking location; The expression for the target reward obtained when parking a vehicle is completed is: （5）； In formula (5), As a positive basic reward constant, The time penalty factor; The expression for the penalty obtained when a vehicle collides is: （6）； Step 6: Construct an automatic parking algorithm framework based on deep reinforcement learning, and improve the experience playback mechanism and agent exploration mechanism in the algorithm to obtain the improved automatic parking algorithm framework. Step 7: Train the improved automatic parking algorithm framework based on the apron layout to obtain the real-time acceleration and steering angle of the airport guide vehicle. Input the real-time acceleration and steering angle into the vehicle kinematics model to realize automatic parking control.

2. The airport guidance vehicle automatic parking method based on reinforcement learning according to claim 1, characterized in that, The vehicle outline coordinates of the airport guide vehicle in step 1 include: the vertex of the vehicle body outline. , , , The center point of the rear axle of the vehicle As a reference point for motion, , This represents the projection of the front and rear axles onto the ground. , The front and rear overhang lengths are... Wheelbase For the front wheel steering angle, The angle between the vehicle body and the horizontal direction. The vehicle's speed is given by the parking coordinate system. .

3. The airport guidance vehicle automatic parking method based on reinforcement learning according to claim 1, characterized in that, The expression for the vehicle kinematics model in step 2 is: （1）。 4. The airport guidance vehicle automatic parking method based on reinforcement learning according to claim 1, characterized in that, In step 4, the state space of the agent is: The action space is ,in, To accelerate the vehicle, This refers to the steering angle of the front wheels.

5. The airport guidance vehicle automatic parking method based on reinforcement learning according to claim 1, characterized in that, The improved automatic parking algorithm framework in step 6 includes: Actor decision-making module, Critic evaluation module, deep reinforcement learning environment, experience replay pool; The experience replay pool is used to provide the experience samples needed for the agent to learn. The experience samples include the current state of the airport guidance vehicle, action information, immediate reward, and the state of the airport guidance vehicle at the next moment. The state of the airport guidance vehicle includes the vehicle's lateral and longitudinal position information, heading angle, and speed. The action information of the airport guidance vehicle includes the wheel steering angle and acceleration. The immediate reward is the cumulative reward for the agent's actions, calculated by combining the immediate reward with the future reward adjusted by the discount factor γ, and is used to incentivize the agent to achieve long-term goals. The Critic evaluation module is used to output the evaluation value of the actions performed by the agent; The Actor decision module is used to output the actions performed by the agent, including the acceleration and steering angle of the airport guidance vehicle.

6. The airport guidance vehicle automatic parking method based on reinforcement learning according to claim 1, characterized in that, Step 6, which involves improving the experience replay mechanism and the agent exploration mechanism, includes the following steps: Step 6.1: Update the experience samples in the experience replay pool based on the Actor decision module and the Critic evaluation module, output the action information of the airport guidance vehicle at the current moment, input the action information of the airport guidance vehicle at the current moment into the kinematic model to obtain the state information of the airport guidance vehicle, combine the state information of the airport guidance vehicle at each moment with the real-time reward mechanism to calculate the cumulative reward of the agent, and finally store the updated information as experience samples in the experience replay pool. Step 6.2: Guide the agent to park within the set motion state space based on the updated experience samples in the experience replay pool; Step 6.3: Design a priority experience replay mechanism to filter the updated experience samples, obtain high-quality experience samples, and complete the improvement of the experience replay mechanism and the agent exploration mechanism.

7. The airport guidance vehicle automatic parking method based on reinforcement learning according to claim 6, characterized in that, Step 6.3, which involves obtaining high-quality experience samples, specifically includes: Calculate the mean and variance of the instant reward returns for all updated experience samples, and design a priori based on the variance and mean of the instant reward returns. High-quality experience samples were obtained through screening; Priority The expression is: （7）； In formula (7), As an empirical sample, The average reward per round. , These are the weights of the mean and variance of the rewards, respectively.

8. The airport guidance vehicle automatic parking method based on reinforcement learning according to claim 1, characterized in that, Step 7, which involves measuring the real-time acceleration and steering angle of the airport guidance vehicle, includes the following steps: Step 7.1: Based on the selected high-quality experience, the gradient optimization method is used to iteratively calculate the parameters of the Actor and Critic evaluation modules, update the Actor and Critic evaluation modules, and design an environment exploration decay mechanism during the training process to complete the training of the improved automatic parking algorithm framework. Step 7.2: Extract the action output results of the Actor decision module after training, and input the action output results into the vehicle kinematics model to realize automatic parking control. The action output results include acceleration. and steering angle ; The expressions for gradient optimization and parameter update rate in the Critic evaluation module are as follows: （8）； In formula (8), For network learning rate, The number of experiences selected in the batch of experiences; The expressions for gradient optimization and parameter update rate in the Actor decision module are: （9）； The expression for the environmental exploration decay mechanism is: （10）； In formula (10), The initial standard deviation, This represents the attenuation rate.

Images