Airport guide vehicle automatic parking method based on reinforcement learning

Abstract

The invention provides an airport guide vehicle automatic parking method based on reinforcement learning. The airport guide vehicle automatic parking method comprises the steps that 1, vehicle contour coordinates and a parking coordinate system of an airport guide vehicle are set; 2, establishing a vehicle kinematics model in combination with the parking coordinate system; 3, establishing a vehicle collision judgment mechanism; 4, setting a state space and an action space of the intelligent agent; 5, setting an instant reward return mechanism of the intelligent agent in the movement process; 6, improving an experience playback mechanism and an intelligent agent exploration mechanism in the algorithm; and 7, training the improved automatic parking algorithm framework in combination with the airport apron layout, and inputting the obtained real-time acceleration and steering angle into a vehicle kinematic model to realize automatic parking control. The method has the advantages that the training period is short, the learning effect is excellent, the vehicle can be automatically parked in a limited space under the complex environment of the airport apron, the burden of ground crew is reduced, and safety accidents are avoided.

Description

Technical Field

[0001] The invention relates to an automatic parking method for an airport guide vehicle based on reinforcement learning, and belongs to the technical field of intelligent driving. Background Art

[0002] With the rapid development of the global aviation industry, the complexity and frequency of apron traffic scheduling have increased. As an important part of intelligent ground traffic management, automatic parking technology is gradually becoming a key requirement for airport guide vehicle operations. According to the real-time information feedback from the vehicle's environment, the driving path at each moment is planned, and the speed and steering are accurately controlled in combination with its own state information. It can effectively avoid parking accidents caused by insufficient driver experience, thereby reducing the waiting time of the guide vehicle and ensuring the normal operation of the aircraft and related ground facilities. Automatic parking technology originated in the early 1990s. It adopts an algorithm structure of path planning combined with trajectory tracking control. That is, after finding a parking space, the parking path is first planned by geometric splicing or numerical fitting, and then the parking path is tracked and controlled by traditional control methods. However, the vehicle involves multiple control variables such as speed and acceleration during the parking process. This early automatic parking method is usually a control algorithm that relies on a series of preset rules. As the parking environment and needs change, the parameters need to be redesigned or adjusted, and it lacks a certain generalization ability.

[0003] In contrast, the development of computer science and artificial intelligence today has facilitated the application of deep reinforcement learning. Control algorithms based on deep reinforcement learning can learn optimal strategies by interacting with the environment. Especially after training with a large amount of diverse data, the intelligent agent can gradually adapt to various unknown environments and output correct actions. Among them, the Deep Deterministic Policy Gradient (DDPG) algorithm is widely used in continuous and high-precision control tasks such as automatic parking due to its excellent stability and outstanding ability to handle continuous state and action space. The algorithm adopts an offline learning method in conjunction with the experience replay mechanism, which breaks the correlation of learning samples while ensuring the stability of the training process. The learning ability of deep neural networks is used to process the dynamic relationship between the high-dimensional real-time state and execution decision of the vehicle in the parking environment. By continuously adjusting its internal parameters during the network training process, the output action can minimize the prediction error and maximize the task benefit, so as to achieve safe and stable parking operations.

[0004] Due to the presence of various flight ground service facilities such as jet bridges, power trucks, and tractors on the apron, the parking environment of the airport guide vehicle is complex and the available space is limited. Therefore, the use of the DDPG algorithm to complete the parking task of the guide vehicle usually has the following defects:

[0005] (1) The feedback from the parking environment is usually highly random, and the experience replay pool in the algorithm has a limited capacity. The agent’s exploration of the environment during training may generate a large number of invalid experience samples, causing successful samples to be overwhelmed by updates, which in turn leads to a long algorithm training cycle and poor parking quality.

[0006] (2) The algorithm adopts a deterministic strategy, and the agent lacks sufficient exploration ability in the early stage of training. The added environmental exploration signal may interfere with the agent's later strategy learning, causing the agent's training process to oscillate and the learning efficiency to be low. Summary of the invention

[0007] The present invention aims to solve the technical problem of poor parking quality in the prior art, and further proposes an automatic parking method for an airport guide vehicle based on reinforcement learning.

[0008] The technical solution adopted by the present invention to solve the above problems is: the present invention proposes an automatic parking method for an airport guide vehicle based on reinforcement learning, comprising:

[0009] Step 1: Set the vehicle contour coordinates and parking coordinate system of the airport guidance vehicle;

[0010] Step 2: Establish the vehicle kinematic model based on the parking coordinate system;

[0011] Step 3: Set up vehicle collision determination mechanism;

[0012] Step 4: Set the vehicle's motion state space and action space;

[0013] Step 5: Set up the agent’s immediate reward mechanism during the movement;

[0014] Step 6: Build an automatic parking algorithm framework based on deep reinforcement learning, and improve the experience replay mechanism and agent exploration mechanism in the algorithm to obtain an improved automatic parking algorithm framework;

[0015] Step 7: Train the improved automatic parking algorithm framework in combination with the apron layout to obtain the real-time acceleration and steering angle of the airport guide vehicle, input the real-time acceleration and steering angle into the vehicle kinematic model to achieve automatic parking control.

[0016] Optionally, the vehicle contour coordinates of the airport guide vehicle in step 1 include: body contour vertices A, B, C, D, the rear axle center point o(x, y) of the vehicle is the motion reference point, F and R are the projections of the front and rear axles on the ground, L f , L r is the front and rear overhang length, L is the wheelbase, δ is the front wheel steering angle, φ is the angle between the vehicle body and the horizontal direction, V is the vehicle speed, and the parking coordinate system is xoy.

[0017] Optionally, the vehicle kinematic model in step 2 is expressed as:

[0018]

[0019] Optionally, the vehicle collision determination mechanism in step 3 specifically includes:

[0020] Setting the conditions for the coordinates of the vertices of the vehicle contour when the vehicle collides and the conditions for the area of ​​the geometric figure constructed around the collision point P to be satisfied;

[0021] The conditions for the coordinates of the vehicle contour vertices are as follows:

[0022]

[0023] In formula (2), (x i ,y i ) is the coordinate of the vehicle vertex, (x obstacle ,y obstacle ) are the obstacle coordinates;

[0024] The area of ​​the geometric figure constructed around the collision point P satisfies the following conditions:

[0025]

[0026] Optionally, in step 4, the state space of the agent is s = (x, y, V, φ), and the action space is in, is the vehicle acceleration, and δ is the front wheel steering angle.

[0027] Optionally, the immediate reward mechanism of the agent during the movement in step 5 includes:

[0028] Positive rewards for guiding the vehicle to the parking space, target rewards obtained when the vehicle completes parking, and penalties obtained when the vehicle collides;

[0029] The expression for the positive reward for guiding the vehicle closer to the berth is:

[0030]

[0031] In formula (4), (x T ,y T ) is the parking target position;

[0032] The expression of the target reward obtained when the vehicle completes parking is:

[0033]

[0034] In formula (5), R base is the positive basic reward constant, ξ is the time penalty factor;

[0035] The penalty that a vehicle receives when it collides is expressed as:

[0036] r collision =-R base (6).

[0037] Optionally, the improved automatic parking algorithm framework in step 6 includes:

[0038] Actor decision module, Critic evaluation module, deep reinforcement learning environment, experience replay pool;

[0039] The experience replay pool is used to provide experience samples, which include the current state, action, reward of the airport guide vehicle and the next state of the airport guide vehicle, wherein the state of the airport guide vehicle includes the lateral and longitudinal moving distance, heading angle, and speed of the vehicle, and the action of the airport guide vehicle includes the wheel steering angle and acceleration;

[0040] Actor decision module, Critic evaluation module, deep reinforcement learning environment, experience replay pool;

[0041] The experience replay pool is used to provide experience samples required for agent learning, and the experience samples include the current state, action information, reward and the state of the airport guide vehicle at the next moment, wherein the state of the airport guide vehicle includes the lateral and longitudinal position information, heading angle and speed of the vehicle, and the action information of the airport guide vehicle includes the wheel steering angle and acceleration. The reward is the cumulative reward return for the action taken by the agent, which is calculated by combining the immediate reward with the future reward adjusted by the discount factor γ, and is used to motivate the agent to achieve long-term goals;

[0042] The Critic evaluation module is used to output the evaluation value of the action performed by the agent;

[0043] The Actor decision module is used to output the agent execution action, wherein the agent execution action includes the acceleration and steering angle of the airport guide vehicle;

[0044] The Actor decision-making module and the Critic evaluation module are used for strategy generation and value estimation, and their interaction improves the learning efficiency and strategy quality of the intelligent agent.

[0045] Optionally, the steps of improving the experience replay mechanism and the agent exploration mechanism in step 6 include:

[0046] Step 6.1: Update the experience samples in the experience replay pool based on the Actor decision module and the Critic evaluation module, output the action information of the airport guide vehicle at the current moment, bring the action information of the airport guide vehicle at the current moment into the kinematic model to obtain the state information of the airport guide vehicle, combine the state information of the airport guide vehicle at each moment with the instant reward return mechanism to calculate the cumulative reward return of the agent, and finally store the above updated information as experience samples in the experience replay pool;

[0047] Step 6.2: Guide the agent to park in the set motion state space based on the updated experience samples in the experience replay pool;

[0048] Step 6.3: Design a priority experience replay mechanism to screen the updated experience samples to obtain high-quality experience samples, thereby improving the experience replay mechanism and the agent exploration mechanism.

[0049] Optionally, the high-quality experience samples obtained in step 6.3 specifically include:

[0050] Calculate the mean and variance of the instant reward return results of all updated experience samples, and design the priority P based on the variance and mean of the instant reward return results. i , screen and obtain high-quality experience samples;

[0051] Priority P i The expression is:

[0052]

[0053] In formula (7), r i For experience samples, is the mean reward of the round, α and β are the weights of the mean and variance of the reward respectively.

[0054] Optionally, the step of obtaining the real-time acceleration and steering angle of the airport guidance vehicle in step 7 includes:

[0055] Step 7.1: Based on the selected high-quality experience, the gradient optimization method is used to iteratively calculate the parameters of the Actor and Critic evaluation modules, complete the update of the Actor and Critic evaluation modules, and design the environment exploration attenuation mechanism during the training process to complete the training of the improved automatic parking algorithm framework;

[0056] Step 7.2: Extract the action output results of the Actor decision module after training, and input the action output results into the vehicle kinematic model to realize automatic parking control. The action output results include acceleration and steering angle δ;

[0057] The expressions of gradient optimization and parameter update rate of the Critic evaluation module are:

[0058]

[0059] In formula (8), η is the network learning rate, and M is the number of experiences selected from the batch experience;

[0060] The expressions for gradient optimization and parameter update rate of the Actor decision module are:

[0061]

[0062] The expression of the environmental exploration attenuation mechanism is:

[0063]

[0064] In formula (10), σ0 is the initial standard deviation and ε is the decay rate.

[0065] The beneficial effects of the present invention are:

[0066] (1) The automatic parking method provided by the present invention has a complete structure and can effectively screen high-quality experience samples for intelligent agent learning in conjunction with the priority experience playback mechanism. It solves the problems of long parking algorithm training cycle and poor parking quality (position accuracy, time consumption and motion state).

[0067] (2) The environment exploration attenuation mechanism provided by the present invention can balance the agent's exploration of the parking environment and the use of effective strategies during the learning process, solving the problems of parking algorithm training fluctuations and low learning efficiency.

[0068] (3) The present invention has a shorter training cycle and better learning effect, so that the vehicle can perform autonomous parking operations in a limited space under the complex environment of the apron, thereby reducing the burden on ground staff and avoiding safety accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flow chart of the automatic parking method for an airport guide vehicle based on reinforcement learning provided by the present invention;

[0070] Figure 2 A simplified diagram of a vehicle model provided by the present invention;

[0071] Figure 3 A schematic diagram of a vehicle body collision provided by the present invention;

[0072] Figure 4 A structural diagram of a parking framework based on the DDPG algorithm provided by the present invention;

[0073] Figure 5A flow chart of a priority experience replay mechanism based on reward features provided by the present invention;

[0074] Figure 6 The experience screening flow chart provided by the present invention;

[0075] Figure 7 A schematic diagram of the airport guide vehicle parking scenario provided by the present invention;

[0076] Figure 8 The cumulative reward return curve of the airport guide vehicle automatic parking method based on reinforcement learning provided by the present invention;

[0077] Fig. 9 A simulation effect diagram of the automatic parking method for an airport guide vehicle based on reinforcement learning provided by the present invention;

[0078] Fig.10 A curve diagram of vehicle motion state changes during the automatic parking process provided by the present invention;

[0079] Fig.11 A distribution diagram of parking posture errors during the automatic parking process provided by the present invention. Example

[0080] Combination Figure 1-11 This embodiment is described as follows. Figure 1 As shown, the steps of the automatic parking method for an airport guide vehicle based on reinforcement learning described in this embodiment include:

[0081] S1: Set the vehicle contour coordinates and parking coordinate system of the airport guide vehicle;

[0082] like Figure 2 As shown, this embodiment constructs a simplified diagram of the vehicle model based on the vehicle outline coordinates and the parking coordinate system.

[0083] S2: Establish the vehicle kinematic model in combination with the parking coordinate system;

[0084] The expression of the vehicle kinematic model is:

[0085]

[0086] S3: Setting the vehicle collision determination mechanism;

[0087] like Figure 3 As shown, this embodiment sets the conditions for satisfying the coordinates of the vertices of the vehicle contour when the vehicle collides and the conditions for satisfying the area of ​​the geometric figure constructed around the collision point P;

[0088] The conditions for the coordinates of the vehicle contour vertices are as follows:

[0089]

[0090] In formula (2), (x i ,y i ) is the coordinate of the vehicle vertex, (x obstacle ,y obstacle ) are the obstacle coordinates;

[0091] The area of ​​the geometric figure constructed around the collision point P satisfies the following conditions:

[0092]

[0093] S4: Set the vehicle's motion state space and action space;

[0094] The berth parameters in the model of this embodiment are shown in Table 1, and the vehicle parameters are shown in Table 2;

[0095] Table 1

[0096]

[0097]

[0098] Table 2

[0099]

[0100] S5: Set the instant reward mechanism for the agent during the movement;

[0101] S501: Establish a parking coordinate system xOy with the midpoint of the lower boundary of the parking space as the origin. The parking scene layout is shown in Table 3. When the vehicle position enters the circular area: (x+0.425) 2 +(y-1.5) 2 ≤0.09, and the body direction deviation Determine that parking is successful;

[0102] Table 3

[0103]

[0104] S502: Calculate the agent's immediate reward R s and cumulative reward returns;

[0105] In this embodiment, the agent's immediate reward R is calculated based on the above contents by setting the positive reward for guiding the vehicle to approach the parking space, the target reward obtained when the vehicle completes parking, and the penalty obtained when the vehicle collides. s and cumulative reward returns;

[0106] The expression for the positive reward for guiding the vehicle closer to the berth is:

[0107]

[0108] In formula (4), (x T ,y T ) is the parking target position;

[0109] The expression of the target reward obtained when the vehicle completes parking is:

[0110]

[0111] In formula (5), R base is the positive basic reward constant, ξ is the time penalty factor;

[0112] The penalty that a vehicle receives when it collides is expressed as:

[0113] r collision =-R base (6).

[0114] The agent’s immediate reward R s The expression is:

[0115] R s =ω1·r guide +ω2·r end +ω3·r collision (7);

[0116] In formula (7), the basic reward R base =15, time penalty factor ξ = 0.5; weights ω1 = 0.3, ω2 = 0.2, ω3 = 0.5. The expression for cumulative reward return is:

[0117]

[0118] In formula (8), the discount factor γ = 0.95, and the time step per round t = 100000.

[0119] S6: Design an improved automatic parking algorithm framework to achieve automatic parking control;

[0120] S601: Figure 4 As shown, the improved automatic parking algorithm framework in this embodiment is based on the automatic parking framework based on the DDPG algorithm, and a designed Critic network is added to update the experience samples in the experience replay pool based on the Actor network and the Critic network;

[0121] The algorithm parameters in this embodiment are shown in Table 4;

[0122] Table 4

[0123]

[0124]

[0125] S602: Based on the Actor decision module and the Critic evaluation module, the experience samples in the experience replay pool are updated, the action information of the airport guide vehicle at the current moment is output, the action information of the airport guide vehicle at the current moment is brought into the kinematic model to obtain the state information of the airport guide vehicle, and the state information of the airport guide vehicle at each moment is combined with the instant reward return mechanism to calculate the cumulative reward return of the agent, and finally the above updated information is stored as the experience sample in the experience replay pool;

[0126] S603: Figure 5 As shown, for the priority experience playback mechanism in the parking algorithm of the present invention, the present invention designs the minimum experience quantity Y to meet the extraction condition min = 10000, the number of experiences drawn each time is b = 500, and the updated experience samples are screened. The screening process is as follows Figure 6 As shown, high-quality experience samples are obtained. The present invention can effectively screen high-quality experience samples for intelligent agent learning in conjunction with the priority experience playback mechanism. The problem of long parking algorithm training cycle and poor parking quality (position accuracy, time consumption and motion state) is solved.

[0127] Priority P i The expression is:

[0128]

[0129] In formula (9), the weights corresponding to the reward mean and variance are α=0.75 and β=0.25.

[0130] S604: Based on the selected high-quality experience, the gradient optimization method is used to iteratively calculate the parameters of the Actor and Critic evaluation modules, complete the update of the Actor and Critic evaluation modules, and design the environment exploration attenuation mechanism during the training process to complete the training of the improved automatic parking algorithm framework, and extract the action output results of the Actor network after training. The action output results of the Actor network include acceleration and steering angle δ;

[0131] The expressions of gradient optimization and parameter update rate of the Critic network are:

[0132]

[0133] In formula (10), η is the network learning rate, and M is the number of experiences selected from the batch experience;

[0134] The expressions for gradient optimization and parameter update rate of the Actor network are:

[0135]

[0136] The expression of the environmental exploration attenuation mechanism is:

[0137]

[0138] In formula (12), σ0 is the initial standard deviation and ε is the decay rate.

[0139] S605: Input the action output results of the Actor network into the vehicle kinematic model to achieve automatic parking control.

[0140] After completing the automatic parking method parameter setting in this embodiment, Figure 7 Parking training is carried out in the scene shown in the figure. The training effect is as follows Figure 8 As shown in Table 5, after the training is completed, in order to verify the learning effect and generalization ability of the parking algorithm, this embodiment randomly selects three groups of different vehicle starting positions in the vehicle starting area for parking tests. The vehicle starting positions obtained by the test are shown in Table 5. The parking effect of the test is shown in Table 5. Figure 9-11 As shown, it can be seen that the error of the present invention is smaller than that of the traditional DDP G algorithm, and it has a shorter training cycle and better learning effect, so that the vehicle can perform autonomous parking operations in a limited space under the complex environment of the apron, thereby reducing the burden on ground staff and avoiding safety accidents.

[0141] Table 5

[0142]

[0143] The above is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement made to the above embodiments without departing from the content of the technical solution of the present invention, based on the technical essence of the present invention, within the spirit and principles of the present invention, still fall within the protection scope of the technical solution of the present invention.

Claims

1. An automatic parking method for an airport guide vehicle based on reinforcement learning, characterized in that: The steps of the automatic parking method for an airport guide vehicle based on reinforcement learning include: Step 1: Set the vehicle contour coordinates and parking coordinate system of the airport guidance vehicle; Step 2: Establish the vehicle kinematic model based on the parking coordinate system; Step 3: Set up vehicle collision determination mechanism; Step 4: Set the state space and action space of the agent; Step 5: Set up the agent’s immediate reward mechanism during the movement; Step 6: Build an automatic parking algorithm framework based on deep reinforcement learning, and improve the experience replay mechanism and agent exploration mechanism in the algorithm to obtain an improved automatic parking algorithm framework; Step 7: Train the improved automatic parking algorithm framework in combination with the apron layout to obtain the real-time acceleration and steering angle of the airport guide vehicle, input the real-time acceleration and steering angle into the vehicle kinematic model to achieve automatic parking control.

2. The method for automatic parking of an airport guide vehicle based on reinforcement learning according to claim 1, characterized in that: The vehicle contour coordinates of the airport guide vehicle in step 1 include: the body contour vertices A, B, C, D, the rear axle center point o(x, y) of the vehicle is the motion reference point, F and R are the projections of the front and rear axles on the ground, L f , L r is the front and rear overhang length, L is the wheelbase, δ is the front wheel steering angle, φ is the angle between the vehicle body and the horizontal direction, V is the vehicle speed, and the parking coordinate system is xoy.

3. The automatic parking method for an airport guide vehicle based on reinforcement learning according to claim 1, characterized in that: The expression of the vehicle kinematic model in step 2 is:

4. The method for automatic parking of an airport guide vehicle based on reinforcement learning according to claim 1, characterized in that: The vehicle collision determination mechanism in step 3 specifically includes: Setting the conditions for the coordinates of the vertices of the vehicle contour when the vehicle collides and the conditions for the area of ​​the geometric figure constructed around the collision point P to be satisfied; The conditions for the coordinates of the vehicle contour vertices are as follows: In formula (2), (x i ,y i ) is the coordinate of the vehicle vertex, (x obstacle ,y obstacle ) are the obstacle coordinates; The area of ​​the geometric figure constructed around the collision point P satisfies the following conditions:

5. The automatic parking method for an airport guide vehicle based on reinforcement learning according to claim 1, characterized in that: In step 4, the state space of the agent is s = (x, y, V, φ), and the action space is in, is the vehicle acceleration, and δ is the front wheel steering angle.

6. The method for automatic parking of an airport guide vehicle based on reinforcement learning according to claim 1, characterized in that: The immediate reward mechanism of the agent during the movement in step 5 includes: Positive rewards for guiding the vehicle to the parking space, target rewards obtained when the vehicle completes parking, and penalties obtained when the vehicle collides; The expression for the positive reward for guiding the vehicle closer to the berth is: In formula (4), (x T ,y T ) is the parking target position; The expression of the target reward obtained when the vehicle completes parking is: r end =R base -e ξt (5); In formula (5), R base is the positive basic reward constant, ξ is the time penalty factor; The penalty that a vehicle receives when it collides is expressed as: r collision ＝-R base (6)。 7. The method for automatic parking of an airport guide vehicle based on reinforcement learning according to claim 1, characterized in that: The improved automatic parking algorithm framework in step 6 includes: Actor decision module, Critic evaluation module, deep reinforcement learning environment, experience replay pool; The experience replay pool is used to provide experience samples required for agent learning, and the experience samples include the current state, action information, reward and the state of the airport guide vehicle at the next moment, wherein the state of the airport guide vehicle includes the lateral and longitudinal position information, heading angle and speed of the vehicle, and the action information of the airport guide vehicle includes the wheel steering angle and acceleration. The reward is the cumulative reward return for the action taken by the agent, which is calculated by combining the immediate reward with the future reward adjusted by the discount factor γ, and is used to motivate the agent to achieve long-term goals; The Critic evaluation module is used to output the evaluation value of the action performed by the agent; The Actor decision module is used to output the agent execution action, wherein the agent execution action includes the acceleration and steering angle of the airport guide vehicle.

8. The method for automatic parking of an airport guide vehicle based on reinforcement learning according to claim 1, characterized in that: The steps to improve the experience replay mechanism and agent exploration mechanism in step 6 include: Step 6.1: Update the experience samples in the experience replay pool based on the Actor decision module and the Critic evaluation module, output the action information of the airport guide vehicle at the current moment, bring the action information of the airport guide vehicle at the current moment into the kinematic model to obtain the state information of the airport guide vehicle, combine the state information of the airport guide vehicle at each moment with the instant reward return mechanism to calculate the cumulative reward return of the agent, and finally store the above updated information as experience samples in the experience replay pool; Step 6.2: Guide the agent to park in the set motion state space based on the updated experience samples in the experience replay pool; Step 6.: Design a priority experience replay mechanism to screen the updated experience samples to obtain high-quality experience samples, thereby improving the experience replay mechanism and the agent exploration mechanism.

9. The method for automatic parking of an airport guide vehicle based on reinforcement learning according to claim 8, characterized in that: The high-quality experience samples obtained in step 6.4 specifically include: Calculate the mean and variance of the instant reward return results of all updated experience samples, and design the priority P based on the variance and mean of the instant reward return results. i , screen and obtain high-quality experience samples; Priority P i The expression is: In formula (7), r i For experience samples, is the mean reward of the round, α and β are the weights of the mean and variance of the reward respectively.

10. The automatic parking method for an airport guide vehicle based on reinforcement learning according to claim 1, characterized in that: The steps of the real-time acceleration and steering angle of the airport guidance vehicle in step 7 include: Step 7.1: Based on the selected high-quality experience, the gradient optimization method is used to iteratively calculate the parameters of the Actor and Critic evaluation modules, complete the update of the Actor and Critic evaluation modules, and design the environment exploration attenuation mechanism during the training process to complete the training of the improved automatic parking algorithm framework; Step 7.2: Extract the action output results of the Actor decision module after training, and input the action output results into the vehicle kinematic model to realize automatic parking control. The action output results include acceleration and steering angle δ; The expressions of gradient optimization and parameter update rate of the Critic evaluation module are: In formula (8), η is the network learning rate, and M is the number of experiences selected from the batch experience; The expressions for gradient optimization and parameter update rate of the Actor decision module are: The expression of the environmental exploration attenuation mechanism is: In formula (10), σ0 is the initial standard deviation and ε is the decay rate.

Images