A vehicle path planning method and system based on improved MADDPG algorithm

Abstract

The present invention relates to the field of path planning technology, and discloses a vehicle path planning method and system based on an improved MADDPG algorithm, comprising: converting a vehicle path planning problem into a MADDPG algorithm learning problem, and performing a transformation definition of the problem; improving the MADDPG algorithm including a safety officer network based on the characteristics of the safety officer vehicle path planning problem; designing the action state space for each vehicle; using a reward function to evaluate the behavior of each vehicle; using the MADDPG algorithm to train the value network and policy network of each vehicle; using the reward function to feed back data and updating it using the MADDPG algorithm; and setting rewards for avoiding collisions and rewards for distance to the destination. In the present invention, since the improved MADDPG algorithm comprehensively considers the influence of multiple factors, the length of the optimal path is relatively short. The beneficial effects of the present invention are: being able to cover most of the driving focus points in the environment, avoid obstacles and high-slope points, and obtain a higher total reward value.

Description

Technical Field

[0001] The present invention belongs to the technical field of path planning and relates to a vehicle path planning method and system based on an improved MADDPG algorithm. Background Art

[0002] Path planning is one of the main research areas of motion planning. Motion planning consists of path planning and trajectory planning. The sequence of points or curves connecting the starting and ending points is called a path, and the strategy for constructing a path is called path planning. Path planning has a wide range of applications in many fields. Applications in high-tech fields include: vehicle travel distance return; autonomous collision-free robot movement; obstacle avoidance and penetration flight of drones; cruise missile evasion of radar searches, defense against rebound attacks, and completion of penetration and blasting missions. Applications in daily life include: GPS navigation; GIS-based road planning; urban road network planning and navigation. Applications in decision-making management include: the Vehicle Routing Problem (VRP) in logistics management and similar resource management and resource allocation problems; routing problems in the field of communications technology, etc. Path planning methods can basically be used to solve any planning problem that can be topologically represented as a point-line network. Currently, most vehicle path planning algorithms are studied in the field of single-vehicle path planning.

[0003] The Chinese invention patent with publication number CN116358594A discloses a vehicle path planning method and device. A specific implementation of the method includes: in response to a vehicle path planning request, obtaining vehicle waypoint information and multiple planning targets; generating a first path set based on the vehicle waypoint information according to pre-set planning parameters; performing evolutionary processing and variable neighborhood operation processing on the first path set to generate a second path set; based on multiple planning targets, generating a vehicle path planning result through the first path set and the second path set. This implementation method can set multiple planning targets, and based on evolutionary processing and variable neighborhood operation processing, perform path planning for multiple planning targets, reduce the time complexity of path planning, and improve real-time responsiveness. However, the above patent still has the following problems during implementation:

[0004] The above patent discloses a vehicle path planning method and device. Although it reduces the time complexity of path planning and improves real-time responsiveness by setting multiple planning goals and performing path planning for multiple planning goals based on evolutionary processing and variable neighborhood operation processing, it still cannot solve the optimal path in complex environments and problems with many variables. Summary of the Invention

[0005] The purpose of the present invention is to provide a vehicle path planning method and system based on an improved MADDPG algorithm, which can cover most of the driving focus in the environment and avoid obstacles and high-slope points, thereby obtaining a higher total reward value.

[0006] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0007] In a first aspect, the present invention proposes a vehicle path planning method based on an improved MADDPG algorithm, comprising:

[0008] Convert the vehicle path planning problem into a MADDPG algorithm learning problem and define it;

[0009] Improve the MADDPG algorithm based on the characteristics of vehicle path planning problem;

[0010] Treat each vehicle as an intelligent agent and design the action state space for each vehicle;

[0011] Use a reward function to evaluate the actions of each vehicle;

[0012] Use the improved MADDPG algorithm to train the value network and policy network of each vehicle;

[0013] Set the reward function for avoiding collisions and the reward function for being close to the destination;

[0014] Based on the reward function for collision avoidance and the reward function for distance to the destination, a total reward function is derived. This reward value is fed back through the total reward function, and the improved MADDPG algorithm is used to update the actions of each vehicle. In this way, each agent obtains the maximum reward value through the reward and penalty function, and then collaboratively plans the paths of multiple agents to reach their respective destinations.

[0015] The problem characteristics include problems in reporting event points, road obstacles, and building area distribution.

[0016] Combined with the first aspect, we further transform the vehicle path planning problem into a MADDPG algorithm learning problem and define the specific steps as follows:

[0017] The vehicle driving area is rasterized, and the coordinates of each grid point and the corresponding grid attribute features are used as the state of the intelligent agent; the total action of the intelligent agent in the rasterized vehicle driving area is defined, the dimension of the output action is determined based on the total action, and the reward function is defined according to the grid attribute features of the vehicle driving area.

[0018] The state of the agent refers to the location information of the agent. The grid attribute feature refers to the grid size representing the spatial resolution and the value representing the attribute of the location.

[0019] In combination with the first aspect, the method for further improving the MADDPG algorithm is as follows:

[0020] The activation functions of the output layers of the value network and the policy network are modified to adaptive activation functions, and smoothness constraints are added to the adaptive activation functions.

[0021] In combination with the first aspect, further, the method for designing the action state space for each vehicle is:

[0022] For the speed information of vehicle i at time t The action set of all vehicles constitutes the action space of the entire system;

[0023] For the position information of the i-th vehicle at time t The state of the entire system is the aggregation of the states of all vehicles;

[0024] Where i represents the number of the i-th vehicle; T represents time, t = 1, 2,…, T.

[0025] Combined with the first aspect, the method of using the reward function to evaluate the behavior of each vehicle is as follows:

[0026] Count the distance between each vehicle and the destination while avoiding collisions, and derive a reward value based on the collision avoidance and distance to the destination in the overall reward-penalty function;

[0027] The closer each vehicle is to the destination, the greater the reward value it receives; if the distance between each vehicle and other vehicles is less than the minimum distance, a negative reward value is received.

[0028] Combined with the first aspect, the specific method of using the improved MADDPG algorithm to train the value network and policy network of each vehicle is as follows:

[0029] By using the improved MADDPG algorithm to train the value network and policy network of each vehicle, through multiple training and demonstrations, each vehicle can learn the optimal strategy in a multi-vehicle environment, which is beneficial for subsequent calculations.

[0030] Combined with the first aspect, further, the reward value is fed back through the total reward function, and the improved MADDPG algorithm is used to update the action of each vehicle as follows:

[0031] Let multiple vehicles choose the next action based on the current state, then conduct a comprehensive analysis of the vehicle actions and provide feedback based on the total reward function. Then use the improved MADDPG algorithm to obtain the reward value, and finally update the action of each vehicle.

[0032] The current state refers to the current vehicle speed information Location information As well as speed and position information of other agents.

[0033] In combination with the first aspect, further, a method for setting a reward function for avoiding vehicle collisions and a reward function for distance to a destination is as follows:

[0034] A smooth penalty function is introduced, setting rewards for avoiding vehicle collisions and rewards for distance to the destination. To simplify and increase the smoothness of the penalty for path deviation, a penalty term proportional to the path deviation is used, and a nonlinear term is introduced to prevent small deviations from resulting in large penalties. By introducing a smooth penalty function, the rewards and penalties can be prevented from changing dramatically as the distance between the two vehicles decreases, making the learning process smoother.

[0035] The expression of the smooth penalty function (i.e., the reward function for avoiding vehicle collisions) is as follows:

[0036]

[0037] In the above formula, R' collision represents the penalty value; α is the parameter that controls the decay speed and is a preset value; D safe is the safe distance, which is set based on experience; P i (t) represents the position of the vehicle itself; P j (t) represents the position of other vehicles; N represents the total number of N vehicles; i and j represent the vehicle numbers; i = 1, 2, ..., N, j = 1, 2, ..., N-1, i ≠ j; T represents time, t = 1, 2, ..., T; when the distance between two vehicles is greater than D safe When the distance between the two cars is less than D safe When , the penalty value increases;

[0038] The reward function for the distance to the destination is expressed as follows:

[0039]

[0040] In the above formula, R' path Indicates the reward value of the distance to the destination; P i,target represents the destination of the i-th vehicle; ||P i (t)-P i,target|| still represents the Euclidean distance between the position of vehicle i at time t and its target position on the path; γ is a positive scaling parameter, which is a regulation factor used to adjust the penalty intensity of the distance between the target position and the agent position, that is, to adjust the degree of influence of path deviation; the log part in the above formula is used to ensure that even if the distance error is small, deviation is allowed to exist, making the learning process smoother and more robust.

[0041] In a second aspect, the present invention provides a vehicle path planning system based on the MADDPG algorithm, comprising:

[0042] The problem conversion and definition module is configured to convert the vehicle path planning problem into a MADDPG algorithm learning problem and define it;

[0043] An algorithm improvement module is configured to improve the MADDPG algorithm according to the characteristics of the vehicle path planning problem;

[0044] An action state space design module is configured to regard each vehicle as an intelligent agent and perform action state space design for each vehicle;

[0045] An evaluation and determination module configured to evaluate the actions of each vehicle using a reward function;

[0046] A vehicle training module is configured to train the value network and policy network of each vehicle using an improved MADDPG algorithm;

[0047] A reward setting module is configured to set a reward function for avoiding collision and a reward function for distance to a destination;

[0048] The data update module is configured to obtain a total reward function based on the reward function for avoiding collisions and the reward function for distance to the destination, feedback the reward value through the total reward function, and use the improved MADDPG algorithm to update the action of each vehicle.

[0049] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the vehicle path planning method based on the improved MADDPG algorithm are implemented.

[0050] In a fourth aspect, the present invention provides a computer device, comprising:

[0051] memory for storing computer programs;

[0052] A processor is used to execute the computer program to implement the steps of the vehicle path planning method based on the improved MADDPG algorithm.

[0053] In a fifth aspect, the present invention provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the above-mentioned vehicle path planning method based on the improved MADDPG algorithm.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] (1) The present invention solves the problem of how to find the optimal path in a complex environment with many variables. It can cover most of the driving points in the environment and avoid obstacles and high-slope points, thereby obtaining a higher total reward value.

[0056] (2) The present invention is based on the improved MADDPG algorithm and comprehensively considers the influence of multiple factors. The optimal path planned is short and highly flexible. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 Schematic diagram of the process of the path planning method in Example 1;

[0058] Figure 2 A schematic diagram of area division in the path planning method of Example 1;

[0059] Figure 3 is the exploration degree graph of the agent in the path planning method of Example 1;

[0060] Figure 4 Schematic diagram of the neural network structure of the improved MADDPG algorithm in the path planning method of Example 1. DETAILED DESCRIPTION

[0061] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0062] The term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Additionally, the character " / " generally indicates an "or" relationship between the related objects.

[0063] Example 1

[0064] like Figure 1 As shown, the steps of the vehicle path planning method based on the improved MADDPG algorithm of this embodiment are as follows:

[0065] Step S1: Convert the vehicle path planning problem into a MADDPG algorithm learning problem and define it.

[0066] The vehicle driving area is rasterized, and the coordinates of each grid point and the corresponding grid attribute features are used as the state of the intelligent agent; the total action of the intelligent agent in the rasterized vehicle driving area is defined, the dimension of the output action is determined based on the total action, and the reward function is defined according to the grid attribute features of the vehicle driving area.

[0067] Step S2: Improve the MADDPG algorithm based on the characteristics of the vehicle path planning problem.

[0068] The activation functions of the output layers of the value network and the policy network are modified to adaptive activation functions, and smoothness constraints are added to the adaptive activation functions to improve the MADDPG algorithm.

[0069] Among them, the problem characteristics include problems in reporting event points, road obstacles, and building area distribution.

[0070] In step S3, each vehicle is regarded as an intelligent agent and the action state space is designed for each vehicle.

[0071] For the speed information of vehicle i at time t The action set of all vehicles constitutes the action space of the entire system;

[0072] For the position information of the i-th vehicle at time t The state of the entire system is the aggregation of the states of all vehicles;

[0073] Where i represents the number of the i-th vehicle; T represents time, t = 1, 2,…, T.

[0074] In step S4, the reward function is used to evaluate the actions of each vehicle.

[0075] Count the distance between each vehicle and the destination while avoiding collisions, and derive a reward value based on the collision avoidance and distance to the destination in the overall reward-penalty function;

[0076] The closer each vehicle is to the destination, the greater the reward value it receives; if the distance between each vehicle and other vehicles is less than the minimum distance, a negative reward value is received.

[0077] In step S5, the improved MADDPG algorithm is used to train the value network and policy network of each vehicle.

[0078] By using the improved MADDPG algorithm to train the value network and policy network of each vehicle, through multiple training and demonstrations, each vehicle can learn the optimal strategy in a multi-vehicle environment, which is beneficial for subsequent calculations.

[0079] The value network is used to estimate the expected return under a given state and action to improve the accuracy of the value function; the policy network is used to evaluate the action value of the agent under the current policy to maximize the action value.

[0080] Initialize the value network and policy network, and create a corresponding target value network for each policy network and value network. Initialize the experience replay buffer. Each vehicle executes actions according to the current policy, observes the results (rewards and new states), and stores this experience in the replay buffer. A certain number of experience tuples are randomly extracted from the replay buffer. For each experience tuple, the Q value of the next state is calculated using the target value network, and the target Q value is generated by combining the rewards. By keeping the parameters of the policy network fixed, the policy gradient is calculated, and the weights of the policy network are updated using the gradient ascent algorithm. The parameters of the target value network are regularly updated by taking a weighted average of the weights of the main network (i.e., the policy network and value network) and the weights of the target value network. Then, repeated training is performed to eventually produce a stable network.

[0081] Step S6: Set a reward function for avoiding collision and a reward function for distance to the destination.

[0082] A smooth penalty function is introduced, and rewards for avoiding vehicle collisions and distance to the destination are set. At the same time, in order to simplify and increase the smooth penalty for path deviation, a penalty term proportional to the path deviation is adopted, and a nonlinear term is introduced to avoid small deviations leading to large penalties. By introducing a smooth penalty function, it is possible to avoid sharp changes in reward and penalty values ​​due to the decrease in the distance between the two vehicles, which can make the learning process smoother.

[0083] The expression of the smooth penalty function (i.e., the reward function for avoiding vehicle collisions) is as follows:

[0084]

[0085] In the above formula, R' collision represents the penalty value; α is the parameter that controls the decay speed and is a preset value; D safe is the safe distance, which is set based on experience; P i (t) represents the position of the vehicle itself; P j (t) represents the position of other vehicles; N represents the total number of N vehicles; i and j represent the vehicle numbers; i = 1, 2, ..., N, j = 1, 2, ..., N-1, i ≠ j; T represents time, t = 1, 2, ..., T; when the distance between two vehicles is greater than Dsafe When the distance between the two cars is less than D safe , the penalty value increases.

[0086] The reward function for the distance to the destination is expressed as follows:

[0087]

[0088] In the above formula, R' path Indicates the reward value of the distance to the destination; P i,target represents the destination of the i-th vehicle; ||P i (t)-P i,target || still represents the Euclidean distance between the position of vehicle i at time t and its target position on the path; γ is a positive scaling parameter, which is a regulation factor used to adjust the penalty intensity of the distance between the target position and the agent position, that is, to adjust the degree of influence of path deviation; the log part in the above formula is used to ensure that even if the distance error is small, deviation is allowed to exist, making the learning process smoother and more robust.

[0089] The reward function is used to motivate the agent to optimize its own behavior, learn how to make optimal decisions independently in a multi-agent driving environment, and reach its destination by avoiding collisions with other agents.

[0090] In step S7, a total reward function is derived based on the collision avoidance reward function and the distance-to-destination reward function. This total reward function is used to provide feedback on reward values, and each vehicle's action is updated using the improved MADDPG algorithm. Specifically, multiple vehicles are asked to select their next action based on their current state. Afterwards, the vehicle actions are analyzed and feedback is provided based on the total reward function. Finally, the improved MADDPG algorithm is used to derive reward values, and each vehicle's action is updated.

[0091] The current state refers to the current vehicle speed information Location information As well as speed and position information of other agents.

[0092] In this way, each agent obtains the maximum reward value through the reward and punishment function, and then collaboratively plans the paths of multiple agents to reach their respective destinations.

[0093] More specific embodiments are as follows:

[0094] The path planning method of the present invention is used to realize the division of the vehicle driving area, so as to ensure the uniformity of the division of the vehicle driving area.

[0095] Before dividing the vehicle driving area, the vehicle driving area is first rasterized, and then the rasterized vehicle driving area is divided into the main driving area and the secondary driving area based on the reported event points, road obstacles, and building area and distribution, and the main driving area is divided into multiple sub-driving areas.

[0096] For example, the vehicle driving zone map is rasterized into a 42*32 grid map, and the actual size of each grid is about 3km*3km. Then, based on the reported event points, road obstacles, and building area and distribution, three main vehicle driving areas are divided. The reported event points, road obstacles, building area and distribution points of each main area account for approximately 0.34, 0.34, and 0.32, and the area ratio is approximately 1:1:1. Finally, the three main driving areas are subdivided into sub-driving areas, as shown in the following figure. Figure 2 As shown, Figure 2 The shaded area represents obstacles or buildings. The vehicle's driving path planning will be carried out in the sub-driving area.

[0097] Figure 3 This is the exploration degree graph of the agent in the vehicle path planning method, which is explored in the order from 1 to 9. Figure 4 This is a schematic diagram of the neural network architecture of the improved MADDPG algorithm. In the figure, a represents an action, s represents a state, and Minimize1 and Minimize2 represent two different loss functions that need to be minimized to train the value network and policy network. The minimization process is implemented using the gradient descent algorithm. The safety officer network refers to the value network and the decision network, responsible for the current decision. That is, it receives the state s of the environment as input and outputs the value (Q value) of action a. The supervisor network copies parameters from the value network and decision network to stabilize the learning process.

[0098] Because the improved MADDPG algorithm comprehensively considers the influence of multiple factors, the optimal path is relatively short, covers most driving priorities in the environment, and avoids obstacles and high-slope points, resulting in a high total reward value. Therefore, the driver path planning method based on the improved MADDPG algorithm provided in Example 1 considers multiple vehicles on the road as intelligent agents and performs path planning for multiple vehicles. This allows vehicles to plan paths based on the real-time status and actions of other vehicles, thereby achieving certain flexibility, feasibility, and rationality.

[0099] Example 2

[0100] The difference between Example 2 and Example 1 is that Example 2 regards the safety officer as an intelligent agent and plans the path of the safety officer in the work area, while Example 1 regards the vehicle as an intelligent agent and plans the vehicle's driving path.

[0101] Because each work area requires at least one assigned safety officer to oversee safety and other matters, they must alternate between traveling within their respective primary driving areas and following the planned paths within their respective sub-driving areas. Therefore, the path planning method of the present invention is used to implement the safety officer's driving path to ensure uniform and safe allocation of safety officers. The safety officer in the present invention is responsible for ensuring safety and can be either a human or a monitoring device. The safety officer is considered an intelligent agent, and the path planning method for the safety officer within the work park is identical to the vehicle path planning method.

[0102] Example 3

[0103] Based on the same inventive concept as Example 1, this example introduces a vehicle path planning system based on the MADDPG algorithm, including:

[0104] The problem conversion and definition module is configured to convert the vehicle path planning problem into a MADDPG algorithm learning problem and define it;

[0105] An algorithm improvement module is configured to improve the MADDPG algorithm according to the characteristics of the vehicle path planning problem;

[0106] An action state space design module is configured to regard each vehicle as an intelligent agent and perform action state space design for each vehicle;

[0107] An evaluation and determination module configured to evaluate the actions of each vehicle using a reward function;

[0108] A vehicle training module is configured to train the value network and policy network of each vehicle using an improved MADDPG algorithm;

[0109] A reward setting module is configured to set a reward function for avoiding collision and a reward function for distance to a destination;

[0110] The data update module is configured to obtain a total reward function based on the reward function for avoiding collisions and the reward function for distance to the destination, feedback the reward value through the total reward function, and use the improved MADDPG algorithm to update the action of each vehicle.

[0111] Example 4

[0112] Based on the same inventive concept as other embodiments, this embodiment introduces a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the vehicle path planning method based on the improved MADDPG algorithm are implemented.

[0113] Example 5

[0114] Based on the same inventive concept as other embodiments, this embodiment introduces a computer device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the above-mentioned vehicle path planning method based on the improved MADDPG algorithm.

[0115] Example 6

[0116] Based on the same inventive concept as other embodiments, this embodiment introduces a computer program product, including a computer program. When the computer program is executed by a processor, the steps of the vehicle path planning method based on the improved MADDPG algorithm are implemented.

[0117] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0118] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0121] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which are all protected by the present invention.

Claims

1. A vehicle path planning method based on an improved MADDPG algorithm, characterized in that: include: Convert the vehicle path planning problem into a MADDPG algorithm learning problem and define it; Improve the MADDPG algorithm based on the characteristics of vehicle path planning problem; Treat each vehicle as an intelligent agent and design the action state space for each vehicle; Use a reward function to evaluate the actions of each vehicle; Use the improved MADDPG algorithm to train the value network and policy network of each vehicle; Set the reward function for avoiding collisions and the reward function for being close to the destination; Based on the reward function for avoiding collisions and the reward function for distance to the destination, a total reward function is obtained. The reward value is fed back through the total reward function, and the improved MADDPG algorithm is used to update the action of each vehicle. The method for setting the reward function for avoiding vehicle collision and the reward function for distance to the destination is: A smooth penalty function is introduced, which sets the reward for avoiding vehicle collisions and the reward for the distance to the destination, and adopts a penalty term that is proportional to the path deviation. The expression of the smooth penalty function is as follows: ； In the above formula, Indicates the penalty value; It is the parameter that controls the decay speed and is the preset value; is the safe distance, which is set based on experience; Indicates the vehicle's own position; Indicates the position of other vehicles; N indicates that there are N vehicles in total; i and j are vehicle numbers; i=1,2,...,N, j=1,2,...,N-1, i≠j; T is time, t=1,2,...,T; when the distance between two vehicles is greater than When the distance between the two cars is less than When , the penalty value increases; The reward function for the distance to the destination is expressed as follows: ； In the above formula, Indicates the reward value of the distance to the destination; represents the destination of the i-th vehicle; Still means vehicle In time The Euclidean distance between the position of and the target position point on the path; is a positive scaling parameter.

2. The vehicle path planning method based on the improved MADDPG algorithm according to claim 1, characterized in that: The specific steps for converting the vehicle path planning problem into a MADDPG algorithm learning problem and defining it are as follows: The vehicle driving area is rasterized, and the coordinates of each grid point and the corresponding grid attribute features are used as the state of the intelligent agent; the total action of the intelligent agent in the rasterized vehicle driving area is defined, the dimension of the output action is determined based on the total action, and the reward function is defined according to the grid attribute features of the vehicle driving area.

3. The vehicle path planning method based on the improved MADDPG algorithm according to claim 1, characterized in that: The method to improve the MADDPG algorithm is: The activation functions of the output layers of the value network and the policy network are modified to adaptive activation functions, and smoothness constraints are added to the adaptive activation functions.

4. The vehicle path planning method based on the improved MADDPG algorithm according to claim 1, characterized in that: The method for designing the action state space for each vehicle is: For the speed information of vehicle i at time t ,The action set of all vehicles constitutes the action space of the entire system; For the position information of the i-th vehicle at time t , the state of the entire system is the set of states of all vehicles; Where i represents the number of the i-th vehicle; T represents time, t=1,2,…,T.

5. The vehicle path planning method based on the improved MADDPG algorithm according to claim 1, characterized in that: The reward function is used to evaluate the behavior of each vehicle: Count the distance between each vehicle and the destination while avoiding collisions, and derive a reward value based on the collision avoidance and distance to the destination in the overall reward-penalty function; The closer each vehicle is to the destination, the greater the reward value it receives; if the distance between each vehicle and other vehicles is less than the minimum distance, a negative reward value is received.

6. A vehicle path planning system based on an improved MADDPG algorithm, characterized in that: A vehicle path planning method based on an improved MADDPG algorithm for implementing any one of claims 1 to 5, comprising: The problem conversion and definition module is configured to convert the vehicle path planning problem into a MADDPG algorithm learning problem and define it; An algorithm improvement module is configured to improve the MADDPG algorithm according to the characteristics of the vehicle path planning problem; An action state space design module is configured to regard each vehicle as an intelligent agent and perform action state space design for each vehicle; An evaluation and determination module configured to evaluate the actions of each vehicle using a reward function; A vehicle training module is configured to train the value network and policy network of each vehicle using an improved MADDPG algorithm; A reward setting module is configured to set a reward function for avoiding collision and a reward function for distance to a destination; The data update module is configured to obtain a total reward function based on the reward function for avoiding collisions and the reward function for distance to the destination, feedback the reward value through the total reward function, and use the improved MADDPG algorithm to update the action of each vehicle.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the vehicle path planning method based on the improved MADDPG algorithm described in any one of claims 1 to 5 are implemented.

8. A computer device, characterized in that: include: memory for storing computer programs; A processor is used to execute the computer program to implement the steps of the vehicle path planning method based on the improved MADDPG algorithm according to any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the vehicle path planning method based on the improved MADDPG algorithm described in any one of claims 1 to 5 are implemented.

Images