Parking task assignment and trajectory planning system based on multi-agent reinforcement learning

Abstract

The application provides a parking task allocation and trajectory planning system based on multi-agent reinforcement learning, which comprises a parking space allocation subsystem and a trajectory planning subsystem; after a vehicle enters a parking lot environment, the vehicle accesses the parking space allocation subsystem, corresponding parking spaces are allocated to multiple vehicles at the same time, so that the time for the vehicle to find a parking space after entering is avoided, and the parking efficiency and safety are improved; when the parking space allocation subsystem allocates parking spaces, the demand of task allocation is reflected in the design of a reward value, a shortest path reward and a surrounding parking space occupation influence reward are designed, and the influence of the length of the vehicle parking trajectory and the blocking of the surrounding environment after parking is considered; for the trajectory planning subsystem, when the optimal trajectory is obtained, the distance between the current position of the vehicle and the nearest obstacle is taken as one of the considered factors, the collision scene of the trajectory is further optimized, and under the premise of meeting the dynamics constraint, the intelligence of the decision is ensured, and the safety of driving is ensured.

Description

Technical Field

[0001] This invention belongs to the field of intelligent vehicle technology, and in particular relates to a parking task allocation and trajectory planning system based on multi-agent reinforcement learning. Background Technology

[0002] Autonomous vehicles have been under research for decades, capable of replacing humans in complex scenarios involving high density, long cycles, and high traffic volumes, possessing significant socio-economic value. Autonomous multi-agent parking technology enables multiple vehicles to park simultaneously within a parking area. Compared to traditional single-vehicle parking, this is faster and more efficient, reducing waiting time and congestion. Autonomous multi-agent parking technology allows for better utilization of parking resources, especially during peak hours, more effectively allocating parking spaces, avoiding waste and congestion, and optimizing parking resources. It also prevents parking accidents caused by human error, such as scratches and collisions, improving parking safety. Autonomous parking technology requires a high degree of cooperation and interaction between vehicles, which plays a crucial role in promoting the development and widespread adoption of autonomous driving technology.

[0003] Traditional path search-based vehicle task allocation and trajectory planning methods rely on occupancy grid maps for searching. Their computational complexity is highly dependent on map size and resolution, leading to real-time limitations in handling large-scale scenes and thus restricting their practical applications. Existing Automated Valet Parking (AVP) systems are based on single-agent trajectory planning methods, considering only the vehicle's own motion state, thus failing to optimize global parking space allocation and trajectory planning. Furthermore, the lack of global information in handling conflicts between multiple vehicle trajectories limits the effectiveness of conflict resolution. With the development of artificial intelligence and machine learning, reinforcement learning is increasingly being applied to autonomous driving tasks. Because it does not rely on pre-labeled datasets, it possesses strong generalization capabilities and can more effectively address special situations in the environment. However, it is typically applied to single-vehicle intelligence, failing to fully leverage the advantages of reinforcement learning in multi-agent domains. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a parking task allocation and trajectory planning system based on multi-agent reinforcement learning, which can significantly improve parking efficiency and safety.

[0005] A parking task allocation and trajectory planning system based on multi-agent reinforcement learning includes a parking space allocation subsystem operated by an intelligent parking lot and a trajectory planning subsystem operated by vehicles.

[0006] The parking space allocation subsystem uses a reinforcement learning-based PPO network to simultaneously allocate parking spaces to each vehicle entering the smart parking lot. When allocating parking spaces, priority is given to parking spaces with shorter vehicle parking trajectory lengths and fewer occupancy spaces in the surrounding area after parking.

[0007] After receiving parking space information from the parking space allocation subsystem, each vehicle's trajectory planning subsystem uses an improved MAPPO network to obtain its optimal path from its current position to the designated parking space. The optimal path is achieved by the vehicle performing two or more actions. When selecting the action for the current step, priority is given to actions that minimize the distance between the vehicle's current position and the designated parking space, minimize the deviation between the vehicle's current position and the pre-planned reference path, and maximize the distance between the vehicle's current position and the nearest obstacle.

[0008] Furthermore, the improved MAPPO network consists of an action network, an evaluation network, and a constraint value network, and the method for the improved MAPPO network to obtain any step of the optimal path for any vehicle is as follows:

[0009] S1: Using environmental information as the observation input to the action network of the current vehicle i, the action of the current vehicle i at the current k-th step is obtained. The environmental information includes the motion state s of the current vehicle i. vi =[x,y,v x ,v y [sinθ,cosθ], the state of the specified parking space corresponding to the current vehicle i (goal). i = [x,y,0,0,sinθ,cosθ], and the motion states of the 6 nearest vehicles around the current vehicle i: surr ve =[x,y,v x ,v y [,sinθ,cosθ],(e=0,1,2,3,4,5), [x,y] represents the coordinates of each vehicle in the parking environment coordinate system, [v x ,v y [x, y] represents the velocity of each vehicle along the x and y axes, and [sinθ, cosθ] represents the sine and cosine values ​​of the heading angle of each vehicle; at the same time, the action includes the direction angle and acceleration of the current vehicle i.

[0010] S2: Obtain the reward R for vehicle i at the current k-th step. ik =R goal +R ref , where R goal R represents the distance between the current position of vehicle i and the specified parking space. ref This represents the deviation between the current position of vehicle i and the pre-planned reference path;

[0011] S3: Use the actions and rewards of all vehicles at the current k-th step as input to the evaluation network of vehicle i to obtain the advantage function estimate of the evaluation network of vehicle i. And based on the advantage function estimate Constructing the loss function L of the evaluation network vi :

[0012]

[0013]

[0014] in, This represents the long-term reward calculated by the evaluation network for vehicle i at the current k-th step using the Bellman equation for all vehicles. This represents the action value function estimate made by the evaluation network of vehicle i at the current k-th step for all vehicles. Expressing expectations;

[0015] S4: Use the actions of all vehicles at the current k-th step and the obstacle collision cost of vehicle i as inputs to the constrained value network of vehicle i to obtain the advantage function estimate of the constrained value network of vehicle i. And based on the advantage function estimate Constructing the loss function L of the constrained value network vci :

[0016]

[0017] in, This represents the constraint cost estimate obtained based on the obstacle collision cost of the current vehicle i;

[0018] S5: Based on the advantage function estimate and the estimate of the advantage function Constructing a hybrid advantage function

[0019]

[0020] Where λ is the set weight;

[0021] S6: Based on the mixed advantage function The loss function L for constructing the action network of the current vehicle i CLIP :

[0022]

[0023] Where, r tiThis represents the ratio of the probability of the action taken by vehicle i at the current k-th step to the probability of the action taken at the (k-1)-th step, where ∈ is the set update change amount, clip() represents the clipping function, and min() represents taking the minimum value.

[0024] S7: Update the improved MAPPO network based on the loss functions of the action network, evaluation network, and constraint value network. Then, repeat steps S1 to S7 based on the updated improved MAPPO network until all loss functions converge or the set number of repetitions is reached, and obtain the final action of the current vehicle i at the current k-th step.

[0025] Furthermore, the obstacle collision cost (Cost) for vehicle i at present. dis The calculation method is as follows:

[0026]

[0027] Among them, D min Γ is the distance between the current vehicle i and the nearest obstacle. d k is the maximum collision distance threshold set by the system. pre ζ represents the total number of historical steps calculated up to the current k-th step, and ζ is the set step weight.

[0028] Furthermore, the distance R between the current vehicle i and the designated parking space goal The calculation method is as follows:

[0029] R goal =1-||s vi -goal i || / d max

[0030] Where, d max This indicates the maximum distance set.

[0031] The deviation R between the current vehicle i and the pre-planned reference path ref The calculation method is as follows:

[0032] R ref =1-D vi (s vi ,Path i ) / D max

[0033] Among them, D vi () represents the distance D from the current position of vehicle i to the pre-planned reference path. max Path is the set maximum deviation distance. iBased on a known map, the A* algorithm is used to search for an obstacle-free path from the current position of vehicle i to a specified parking space.

[0034] Furthermore, the training method for the PPO network is as follows:

[0035] A0: Initialize the sequence of vehicles to be assigned, List v = {0,1,2,…,i-1,i,i+1,…,m}, where m represents the number of vehicles waiting to be allocated parking spaces;

[0036] A1: Using parking lot environmental information as the observation input to the PPO network, the parking space number p assigned to each vehicle is obtained. i ;

[0037] A2: For the vehicle i to be assigned and the parking space p corresponding to vehicle i. i Calculate the reward value R i Where, if the allocated parking space p i If it is already occupied, then the reward value R i = -1, otherwise the reward value is calculated as follows:

[0038] R i =R path-i +R global-i

[0039] Among them, R path-i For the path reward of vehicle i, R global-i Let R be the global reward for vehicle i, and let R be the path reward. path-i The calculation method is as follows:

[0040] R path-i =-Path i / MaxPath

[0041] Where, Path i MaxPath represents the default value of the longest path in the known map, obtained by searching the A* algorithm based on a known map.

[0042] Global reward R global-i The calculation method is as follows:

[0043]

[0044] Among them, P l This represents the parking space p corresponding to vehicle i. i The surrounding five parking spaces, l = 0, ..., 4, w i_l This represents the parking space p corresponding to vehicle i. i The occupancy weights of the five surrounding parking spaces, if Pl If P is occupied, then l =0, otherwise P l =1;

[0045] A3: Let i = i + 1, and repeat step A2 until the reward value of all vehicles is obtained;

[0046] A4: Based on the reward value R for each vehicle i The sum ∑R i Construct a mean squared error loss function, and then update the PPO network based on the mean squared error loss function;

[0047] A5: Repeat steps A1 to A5 using the updated PPO network until the set number of repetitions is reached to obtain the final PPO network for parking space allocation.

[0048] Furthermore, the parking environment information input to the PPO network observation includes vehicle motion state information and parking space occupancy state information. The vehicle motion state information includes the coordinates [x, y] of each vehicle in the parking environment coordinate system, and the velocities [v] of each vehicle along the x and y axes. x ,v y The sine and cosine values ​​of the heading angles of each vehicle [sinθ, cosθ] are given, and the parking space occupancy status information is represented as follows:

[0049] [P0,P1,…,P n ,…,P N ]

[0050] Where N is the total number of parking spaces, and if parking space P n If P is occupied, then n =1, otherwise P n =0.

[0051] Beneficial effects:

[0052] 1. This invention provides a parking task allocation and trajectory planning system based on multi-agent reinforcement learning, including a parking space allocation subsystem and a trajectory planning subsystem. After a vehicle enters the parking lot environment, it connects to the parking space allocation subsystem, supporting simultaneous allocation of parking spaces for multiple vehicles. This avoids the time vehicles spend searching for parking spaces after entering the parking lot, improving parking efficiency and safety. When allocating parking spaces in the parking space allocation subsystem, the task allocation requirements are reflected in the reward value design. Shortest path rewards and rewards for the impact of surrounding parking space occupancy are designed, and the length of the vehicle's parking trajectory and the impact of surrounding environmental congestion after parking are considered. For the trajectory planning subsystem, when obtaining the optimal trajectory, the distance between the vehicle's current position and the nearest obstacle is considered as one of the factors, further optimizing the collision scenarios of the trajectory. Under the premise of complying with dynamic constraints, this ensures both the intelligence of the decision-making and the safety of driving.

[0053] 2. This invention provides a parking task allocation and trajectory planning system based on multi-agent reinforcement learning. In the original MAPPO network design, a collision constraint value network is added to improve trajectory safety constraints. Therefore, this invention ensures both intelligent decision-making and driving safety while adhering to dynamic constraints. Furthermore, this invention comprehensively considers the limited vehicle observation in parking scenarios, modeling parking trajectory planning as a partially observable Markov decision process. It utilizes global observation to evaluate the value of each vehicle's trajectory actions and optimizes the trajectory actions.

[0054] 3. This invention provides a parking task allocation and trajectory planning system based on multi-agent reinforcement learning. When obtaining the optimal trajectory, the system allocates a single-step reward value R for the vehicle. ik The calculation of the reference path pre-planned based on the A* algorithm is introduced into the design of the single-step reward function, which can accelerate the convergence of the training network. Attached Figure Description

[0055] Figure 1 This is a parking scenario diagram for the present invention;

[0056] Figure 2 This is a block diagram of the system scheme of the present invention;

[0057] Figure 3 This is a flowchart of the parking allocation process of the present invention;

[0058] Figure 4 This is a network diagram for weighted feature extraction of vehicles around the parking allocation subsystem of the present invention;

[0059] Figure 5 This is a design diagram for the parking space allocation weights in this invention;

[0060] Figure 6 This is a block diagram of the improved MAPPO network of the present invention;

[0061] Figure 7 This is a flowchart of the parking path planning process of the present invention;

[0062] Figure 8 This is a diagram illustrating the trajectory planning effect of the present invention. Detailed Implementation

[0063] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0064] like Figure 1As shown, in parking scenarios, multi-agent reinforcement learning can overcome the limitations of traditional methods on maps, coordinate parking space allocation among multiple agents, handle multi-vehicle trajectory conflicts, and maximize parking efficiency and safety. Based on this, this invention provides a parking task allocation and trajectory planning system based on multi-agent reinforcement learning, such as... Figure 2 As shown, it includes a parking space allocation subsystem operated by an intelligent parking lot and a trajectory planning subsystem operated by vehicles.

[0065] The parking space allocation subsystem uses a reinforcement learning-based PPO network to simultaneously allocate parking spaces to all vehicles entering the smart parking lot. When allocating parking spaces, priority is given to parking spaces with shorter vehicle parking trajectory lengths and fewer occupied surrounding parking spaces after parking.

[0066] After receiving parking space information from the parking space allocation subsystem, each vehicle's trajectory planning subsystem uses an improved MAPPO network to obtain its optimal path from its current position to the designated parking space. The optimal path is achieved by the vehicle performing two or more actions. When selecting the action for the current step, priority is given to actions that minimize the distance between the vehicle's current position and the designated parking space, minimize the deviation between the vehicle's current position and the pre-planned reference path, and maximize the distance between the vehicle's current position and the nearest obstacle.

[0067] In other words, this invention comprises two main parts: first, a multi-vehicle parking space allocation subsystem based on reinforcement learning; and second, a multi-agent trajectory planning subsystem based on conflict constraints. Each subsystem relies on a different type of entity. The reinforcement learning-based multi-vehicle parking space allocation subsystem is operated by the intelligent parking lot, allocating parking spaces to vehicles entering the parking environment. The conflict constraint-based multi-agent trajectory planning subsystem resides at the vehicle end, where the vehicle receives parking space assignments from the intelligent parking lot and actively plans a safe trajectory for parking. Both subsystems can be trained and deployed independently.

[0068] The following section first details the training method of the PPO network in the multi-vehicle parking space allocation subsystem based on reinforcement learning, such as... Figure 3 As shown, the specific steps include:

[0069] A0: Initialize the sequence of vehicles to be assigned, List v = {0,1,2,…,i-1,i,i+1,…,m}, where m represents the number of vehicles waiting to be allocated parking spaces;

[0070] A1: Using parking lot environmental information as the observation input to the PPO network, the parking space number p assigned to each vehicle is obtained. i The list of m vehicles to be assigned is represented as: List v ={s0,s1,s2,…,s i-1 ,si ,s i+1 ,…,s m}, where s i Let the motion state of vehicle i be represented by the feature [x, y, v]. x ,v y [sinθ, cosθ] represent the vehicle's [x, y] coordinates in the parking environment coordinate system, and its velocity [v] along the x and y axes, respectively. x ,v y The sine and cosine values ​​of the heading angle are [sinθ, cosθ]. Assume there are N parking spaces in the parking lot, the spaces are fixed, and the occupancy of the n spaces is [P0, P1, ..., P]. n ,…,P N If parking space P n If P is occupied, then n =1, otherwise P n =0, therefore, the environmental information used as the observation input includes the above two parts: vehicle movement status information and parking space occupancy status information. It should be noted that when calculating the characteristics of the observation information, one can use... Figure 4 The task allocation system shown uses a weighted feature extraction network for vehicles around the master vehicle. This network weights the features of vehicles around the master vehicle with the master vehicle's state to obtain weighted features, thereby increasing the network's scalability and enabling it to handle scenarios with different numbers of vehicles.

[0071] A2: For the vehicle i to be assigned and the parking space p corresponding to vehicle i. i Calculate the reward value R i Where, if the allocated parking space p i If it is already occupied, then the reward value R i = -1, otherwise the reward value is calculated as follows:

[0072] R i =R path-i +R global-i That is, the sum of path reward and global reward.

[0073] Among them, R path-i For the path reward of vehicle i, R global-i Let R be the global reward for vehicle i, and let R be the path reward. path-i The calculation method is as follows:

[0074] R path-i =-Path i / MaxPath

[0075] Where, Path iThis describes the obstacle-free path from the current position of vehicle i to a specified parking space, obtained using the A* algorithm based on a known map. MaxPath represents the preset value of the longest path in the known map, used to normalize R. path-i The path reward value indicates that the closer the route is to vehicle i, the higher the reward will be.

[0076] By analyzing the target parking space p i The status of the five surrounding parking spaces determines the global reward R. global-i The calculation method is as follows:

[0077]

[0078] Among them, P l This represents the parking space p corresponding to vehicle i. i The surrounding five parking spaces, l = 0, ..., 4, w i_l This represents the parking space p corresponding to vehicle i. i The occupancy weight of the five surrounding parking spaces, such as Figure 5 As shown, if P l If P is occupied, then l =0, otherwise P l =1; the global reward R global-i The desired parking space allocation subsystem is configured to allocate parking spaces to vehicle i with the least number of occupied spaces in the surrounding area, in order to reduce local congestion.

[0079] A3: Let i = i + 1, and repeat step A2 until the reward value of all vehicles is obtained;

[0080] A4: Based on the reward value R for each vehicle i The sum ∑R i Construct a mean squared error loss function, and then update the PPO network based on the mean squared error loss function;

[0081] A5: Repeat steps A1 to A5 using the updated PPO network until the set number of repetitions is reached to obtain the final PPO network for parking space allocation.

[0082] Furthermore, for vehicle i, the motion state of vehicle i is adjusted to the head of the queue, and all dynamic obstacle observations are adjusted as follows: {s i ,s i+1 ,…,s m ,s0,s1,s2,…,s i-1The current observation information is used as the basis for feature extraction of the current vehicle i and surrounding obstacle vehicles. An attention weighting mechanism is employed to compare the motion states of other obstacle vehicles with those of the main vehicle, learning the influence weights of obstacles on the current vehicle i. This weighted approach is used to process the influence of multiple obstacles, reducing network complexity and addressing scalability issues.

[0083] Based on parking space allocation, a trajectory planning subsystem based on conflict resolution is then used to obtain the optimal path for vehicles to reach designated parking spaces. Therefore, the improved MAPPO network in the trajectory planning subsystem is described in detail below to obtain the optimal path. Figure 6 As shown, the improved MAPPO network consists of an action network, an evaluation network, and a constraint value network. The method by which the improved MAPPO network obtains any step of the optimal path for any vehicle is as follows: Figure 7 As shown:

[0084] S1: For multiple vehicles waiting to park, the environmental information is used as the observation input to the action network of the current vehicle i to obtain the action of the current vehicle i at the current k-th step. The environmental information includes the motion state s of the current vehicle i. vi =[x,y,v x ,v y [sinθ,cosθ], the state of the specified parking space corresponding to the current vehicle i (goal). i = [x,y,0,0,sinθ,cosθ], and the motion states of the 6 nearest vehicles around the current vehicle i: surr ve =[x,y,v x ,v y [,sinθ,cosθ],(e=0,1,2,3,4,5), [x,y] represents the coordinates of each vehicle in the parking environment coordinate system, [v x ,v y [θ, cosθ] represents the velocity of each vehicle along the x and y axes, and [sinθ, cosθ] represents the sine and cosine values ​​of the heading angle of each vehicle. Simultaneously, the action includes the direction angle and acceleration [θ, a] of the current vehicle i, where the angle is limited to... Acceleration is limited to [-5m / s²] 2 5m / s 2 ].

[0085] S2: Obtain the reward R for vehicle i at the current k-th step. ik =R goal +R ref , where R goal R represents the distance between the current position of vehicle i and the specified parking space. refR represents the deviation between the current position of vehicle i and the pre-planned reference path; goal This indicates that the closer vehicle i is to the designated parking space, the higher the reward value R will be. ik The higher, R ref This represents the penalty for the current vehicle i deviating from the reference path obtained by the A* algorithm.

[0086] The distance R between the current vehicle i and the designated parking space goal The calculation method is as follows:

[0087] R goal =1-||s vi -goal i || / d max

[0088] Where, d max This represents the maximum distance set for normalizing R. goal ;

[0089] It should be noted that, since vehicle trajectories are consistent within a parking lot environment, this invention incorporates a reference trajectory into the reward function design to accelerate network convergence and guide vehicle trajectory planning. The deviation R between the current vehicle i and the pre-planned reference path provided by this invention... ref The calculation method is as follows:

[0090] R ref =1-D vi (s vi ,Path i ) / D max

[0091] Among them, D vi () represents the distance D from the current position of vehicle i to the pre-planned reference path. max The maximum deviation distance is set and used to normalize R. ref Path i Based on a known map, the A* algorithm is used to search for an obstacle-free path from the current position of vehicle i to a specified parking space.

[0092] S3: Use the actions and rewards of all vehicles at the current k-th step as input to the evaluation network of vehicle i to obtain the advantage function estimate of the evaluation network of vehicle i. And based on the advantage function estimate Constructing the loss function L of the evaluation network vi :

[0093]

[0094]

[0095] in, This represents the long-term reward calculated by the evaluation network for vehicle i at the current k-th step using the Bellman equation for all vehicles. This represents the action value function estimate made by the evaluation network of vehicle i at the current k-th step for all vehicles. Expressing expectations;

[0096] It should be noted that the input to the evaluation network includes overall observations of multiple vehicles, thereby enabling a comprehensive measurement of the action value of each vehicle.

[0097] S4: Use the actions of all vehicles at the current k-th step and the obstacle collision cost of vehicle i as inputs to the constrained value network of vehicle i to obtain the advantage function estimate of the constrained value network of vehicle i. And based on the advantage function estimate Constructing the loss function L of the constrained value network vci :

[0098]

[0099] in, This represents the obstacle collision cost (Cost) based on the current vehicle i. dis The obtained constraint cost estimate; where the obstacle collision cost is Cost. dis The calculation method is as follows:

[0100]

[0101] Among them, D min Γ is the distance between the current vehicle i and the nearest obstacle. d k is the maximum collision distance threshold set by the system. pre Let ζ be the total number of historical steps calculated up to the current k-th step, and let ζ be the set step weight. Therefore, the obstacle collision cost (Cost) is... dis The overall description is that the collision distance between the main vehicle and the obstacle is constrained; the closer the distance, the higher the cost.

[0102] S5: Based on the advantage function estimate and the estimate of the advantage function Constructing a hybrid advantage function

[0103]

[0104] Where λ is the set weight;

[0105] S6: Based on the mixed advantage function The loss function L for constructing the action network of the current vehicle i CLIP :

[0106]

[0107] Where, r ti This represents the ratio of the probability of the action taken by vehicle i in the current k-th step to the probability of the action taken in the (k-1)-th step, where ∈ is the set update change amount, clip() represents the clipping function, and min() represents taking the minimum value. It should be noted that the existence of the clipping function can limit the range of change of the action taken in the current k-th step, ensuring the stability of the action update.

[0108] S7: Update the improved MAPPO network based on the loss functions of the action network, evaluation network, and constraint value network. Then, repeat steps S1 to S7 based on the updated improved MAPPO network until all loss functions converge or the set number of repetitions is reached, and obtain the final action of the current vehicle i at the current k-th step.

[0109] It should be noted that when updating the improved MAPPO network based on the loss function, gradient descent or other optimization methods can be used to solve the problem.

[0110] like Figure 8 The diagram shows the trajectory planning effect of this invention. As can be seen, this invention first provides a multi-vehicle parking space allocation subsystem based on reinforcement learning. This subsystem relies on intelligent parking lots; vehicles enter the parking environment and are connected to the subsystem, supporting the simultaneous allocation of parking spaces for multiple vehicles. This avoids the time vehicles spend searching for parking spaces after entering, improving parking efficiency and safety. Secondly, this invention provides a multi-vehicle parking space allocation subsystem based on reinforcement learning, incorporating task allocation requirements into the reward design, and designing shortest path rewards as well as rewards for the impact of surrounding parking space occupancy. It also considers the length of the vehicle's parking trajectory and the impact of surrounding environmental congestion after parking. Finally, after establishing the above parking space allocation subsystem, this invention provides a multi-agent trajectory planning subsystem based on conflict constraints. This subsystem plans collision-free trajectories for multiple vehicles and multiple objectives simultaneously. In the original MAPPO network design, a collision constraint value network is added to improve trajectory safety constraints. In trajectory collision scenarios, under the premise of complying with dynamic constraints, both the intelligence of decision-making and the safety of driving are ensured. Taking into account the limited vehicle observation in parking scenarios, parking trajectory planning is modeled as a partially observable Markov decision process. Global observation is used to evaluate the value of each vehicle's trajectory actions and optimize the trajectory actions.

[0111] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A multi-agent reinforcement learning based parking task assignment and trajectory planning system, characterized in that, This includes a parking space allocation subsystem operated by a smart parking lot and a trajectory planning subsystem operated by vehicles; The parking space allocation subsystem uses a reinforcement learning-based PPO network to simultaneously allocate parking spaces to each vehicle entering the smart parking lot. When allocating parking spaces, priority is given to parking spaces with shorter vehicle parking trajectory lengths and fewer occupancy spaces in the surrounding area after parking. After receiving parking space information from the parking space allocation subsystem, each vehicle's trajectory planning subsystem uses an improved MAPPO network to obtain its optimal path from its current position to the designated parking space. The optimal path is achieved by the vehicle performing two or more actions. When selecting the action for the current step, priority is given to actions that minimize the distance between the vehicle's current position and the designated parking space, minimize the deviation between the vehicle's current position and the pre-planned reference path, and maximize the distance between the vehicle's current position and the nearest obstacle. The improved MAPPO network consists of an action network, an evaluation network, and a constraint value network. The improved MAPPO network obtains any action of any step in the optimal path for any vehicle as follows: S1: Using environmental information as the observation input to the action network of the current vehicle i, the action of the current vehicle i at the current k-th step is obtained. The environmental information includes the motion state of the current vehicle i. The current parking space status corresponding to vehicle i. [and the motion states of the 6 nearest vehicles around vehicle i:] , This represents the coordinates of each vehicle in the parking environment coordinate system. Indicates each vehicle along Velocity in the axial direction, This represents the sine and cosine values ​​of the heading angle for each vehicle; simultaneously, the action includes the direction angle and acceleration of the current vehicle i. S2: Obtain the reward for vehicle i at the current k-th step. ,in, This represents the distance between the current position of vehicle i and the specified parking space. This represents the deviation between the current position of vehicle i and the pre-planned reference path; S3: take all vehicles' actions and rewards at the current k-th step as the input of the critic network of the current vehicle i, and get the advantage function estimate value of the critic network of the current vehicle i , and construct the loss function of the critic network based on the advantage function estimate value :​ where, represents the long-term reward computed by the Bellman equation of the evaluation network of the current vehicle i for the reward of all vehicles at the current k-th step, represents the action-value function estimate made by the evaluation network of the current vehicle i for the action of all vehicles at the current k-th step, represents expectation; S4: take all vehicles' actions at current step k and the obstacle collision cost of current vehicle i as the input of the constraint value network of current vehicle i, and get the advantage function estimate value of the constraint value network of current vehicle i , and based on the advantage function estimate value , construct the loss function of the constraint value network : wherein, denotes a constraint cost estimate derived from the obstacle collision cost of the current vehicle i; S5: Based on the advantage function estimate and the estimate of the advantage function Constructing a hybrid advantage function : wherein, to set the weight; S6: according to the mixed advantage function constructing a loss function of the action network of the current vehicle i : wherein, denotes the ratio of the probability corresponding to the action taken by the current vehicle i at the current k-th step and the probability corresponding to the action taken at the k-1-th step, is a set update variation, denotes a clipping function, denotes taking the minimum value; S7: Update the improved MAPPO network based on the loss functions of the action network, evaluation network, and constraint value network. Then, repeat steps S1 to S7 based on the updated improved MAPPO network until all loss functions converge or the set number of repetitions is reached, and obtain the final action of the current vehicle i in the current k-th step.

2. The multi-agent reinforcement learning based parking task assignment and trajectory planning system of claim 1, wherein, a current vehicle i's obstacle collision cost is calculated as: in, The distance between the current vehicle i and the nearest obstacle. The maximum collision distance threshold is set. This represents the total number of historical steps calculated up to the current k-th step. The set step weight.

3. The multi-agent reinforcement learning based parking task assignment and trajectory planning system of claim 1, wherein, Distance between the current vehicle i and the designated parking space The calculation method is as follows: wherein, denotes the set maximum distance; deviation between the current vehicle i and the pre-planned reference path The calculation method is: wherein, represents the distance of the current position of the current vehicle i from the pre-planned reference path, is a set maximum deviation distance, is an obstacle-free path from the current position of the current vehicle i to the designated parking space obtained by searching using the A* algorithm based on the known map.

4. The multi-agent reinforcement learning based parking task assignment and trajectory planning system of any one of claims 1-3, wherein, The training method for the PPO network is as follows: A0: initialize the sequence of vehicles to be allocated, m represents the number of vehicles to be allocated to the parking space; A1 : taking parking lot environment information as the observation input of the PPO network, to obtain the parking space number allocated to each vehicle ; A2: For vehicles currently awaiting allocation and vehicles corresponding parking space Calculate the reward value Among them, if the allocated parking space If it is already occupied, then the reward value Otherwise, the reward value is calculated as follows: wherein, is a path reward for the vehicle is a global reward for the vehicle is a path reward for the vehicle is a global reward for the vehicle is calculated by: wherein, is an obstacle-free path from the current position of the current vehicle i to the designated parking space obtained by searching using the A* algorithm based on the known map, represents a preset value of the longest path in the known map; Global reward The method of calculating is: in, Indicates vehicle corresponding parking space The five parking spaces around it , Indicates vehicle corresponding parking space The occupancy weight of the five surrounding parking spaces, if If occupied, then ,otherwise ; A3: Let , repeat step A2 until the reward value of all vehicles is obtained; A4: the sum of the reward values corresponding to each vehicle construct a mean square error loss function, and update the PPO network according to the mean square error loss function;​ A5: Repeat steps A1 to A5 using the updated PPO network until the set number of repetitions is reached to obtain the final PPO network for parking space allocation.

5. The multi-agent reinforcement learning based parking task assignment and trajectory planning system of claim 4, wherein, The parking environment information input to the PPO network includes vehicle motion status information and parking space occupancy status information. Among them, the vehicle motion status information includes the coordinates of each vehicle in the parking environment coordinate system. Each vehicle along axial velocity The sine and cosine values ​​of the heading angle for each vehicle The parking space occupancy status information is shown as follows: where N is the total number of parking spaces, and if a parking space is occupied, then , , otherwise .

Images