An automatic driving path planning method, system, electronic device and medium

Abstract

The application provides an automatic driving path planning method and system, electronic equipment and a medium. The method comprises the following steps: an electronic device acquires road traffic data of a specific area, obtains an optimal joint action based on the road traffic data by using a space-time coupling matrix, controls traffic lights and vehicles in the specific area by using an optimal signal phase switching time and a vehicle driving path included in the optimal joint action, and the joint action space is formed by a Cartesian product of a signal control intelligent agent and a path planning intelligent agent to form a hybrid action space, and then the hybrid action space is combined with an Actor-Critic framework to obtain the joint action space. In this way, the decision-making process is completely based on objective data and is not affected by external factors such as weather, ensuring the stability of the control, and the traffic efficiency of vehicles in the specific area can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation systems and automatic driving intersection, and particularly relates to an automatic driving path planning method and system, an electronic device and a medium. BACKGROUND

[0002] Most traffic accidents are caused by human errors, such as fatigue driving, distraction, drunk driving, etc. The automatic driving system can more reliably comply with traffic rules and reduce the likelihood of accidents through sensors, artificial intelligence and precise control algorithms. Based on this, the automatic driving technology is gradually proposed.

[0003] However, although the current automatic driving technology is constantly improving, in complex road conditions and extreme weather conditions, the error of the Long Short-Term Memory (LSTM) prediction model for vehicle trajectory prediction reaches 1.5-2.3 meters. In addition, when the simulation environment is deployed to the real scene, due to the differences in complex environment and weather conditions, the effect of the model will appear attenuation, resulting in a decrease in vehicle speed, and thus causing the problem of reducing the traffic efficiency. SUMMARY

[0004] In order to overcome the above-mentioned defects of the automatic driving path planning in complex environment and weather conditions, the effect will appear attenuation, resulting in a decrease in vehicle speed, and thus causing the problem of low traffic efficiency, the present application provides an automatic driving path planning method, comprising:

[0005] Obtaining road traffic data of a specific area;

[0006] Based on the road traffic data combined with the space-time coupling matrix, the joint action space of the signal control agent and the path planning agent is solved to obtain the best joint action; the best joint action includes the best signal phase switching time and vehicle driving path in the specific area;

[0007] Using the best signal phase switching time and vehicle driving path to control the traffic signal and vehicle in the specific area;

[0008] The joint action space is formed by Cartesian product of the signal control agent and the path planning agent to form a hybrid action space, and then the hybrid action space is combined with the Actor-Critic framework to obtain.

[0009] Further, the construction process of the joint action space comprises:

[0010] According to the green light duration and switching transition time of the traffic signal, a discrete signal control agent is generated;

[0011] generate a continuous path planning agent according to a steering angle and an acceleration when a vehicle is driven;

[0012] form a hybrid action space by Cartesian product of the signal control agent and the path planning agent;

[0013] define a joint action in the hybrid action space as an output of an Actor network in an Actor-Critic framework;

[0014] define a road traffic state and the joint action as inputs of a Critic network in the Actor-Critic framework, and define a positioning of the joint action in the road traffic state as an output of the Critic network;

[0015] construct a joint action space according to the defined Actor-Critic framework.

[0016] Further, a pre-training process of the joint action space comprises:

[0017] take a road traffic state in a joint action space to be trained as a state quantity of an autonomous driving path planning scenario;

[0018] take a joint action in a hybrid action space as an action quantity of the autonomous driving path planning scenario;

[0019] based on the state quantity and the action quantity, perform collaborative iterative training on the joint action space to be trained by using an improved Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, to obtain a trained joint action space.

[0020] Further, the collaborative iterative training on the joint action space to be trained by using the improved MADDPG algorithm based on the state quantity and the action quantity to obtain the trained joint action space comprises:

[0021] based on the improved MADDPG algorithm, initialize parameters in the joint action space to be trained under the constraint of a space-time coupling matrix; the parameters include network parameters of the Actor-Critic framework;

[0022] based on the state quantity, generate a corresponding action quantity by using the Actor-Critic framework with the initialized parameters, and calculate a value estimation when the signal control agent and the path planning agent jointly execute the action quantity;

[0023] Based on the value estimate, the network parameters of the Critic network and the Actor network in the Actor-Critic framework are iteratively optimized to obtain a trained joint action space.

[0024] Further, the iterative optimization of the network parameters of the Critic network and the Actor network in the Actor-Critic framework based on the value estimate to obtain a trained joint action space comprises:

[0025] Based on the value estimate, a loss value is calculated using a mean square error loss function; and based on the loss value, the network parameters of the Critic network are optimized.

[0026] The joint execution of the action amount by the signal control agent and the path planning agent is simulated using road simulation technology to obtain simulated traffic data.

[0027] Based on the simulated traffic data, a reward value of the signal control agent and the path planning agent is calculated using a reward function; and based on the reward value, the network parameters of the Actor network are optimized.

[0028] Based on the optimized network parameters of the Critic network and the Actor network, corresponding action amounts are continuously generated until the iteration termination condition is met, and a trained joint action space is obtained.

[0029] Further, the reward function satisfies the following formula:

[0030]

[0031] wherein R total is the reward value, a is a preset weight, X is the overall traffic efficiency of the specific region, N is the number of vehicles in the specific region, x i is the actual driving time of the i-th vehicle in the driving data, is the expected driving time of the i-th vehicle in the driving data.

[0032] Further, before controlling the traffic signal and the vehicle in the specific region using the optimal signal phase switching time and the vehicle driving path, the method further comprises:

[0033] Based on the optimal signal phase switching time and the vehicle driving path in the optimal joint action, a conflict detection function is used to determine a traffic conflict detection result.

[0034] If the traffic conflict detection result indicates that there is a potential traffic conflict, a preset correction mechanism is used to correct the optimal joint action.

[0035] In another aspect, the present application also provides an automatic driving path planning system, comprising:

[0036] a traffic data collection module, configured to acquire road traffic data of a specific area;

[0037] a dynamic environment modeling module, configured to solve a joint action space of a signal control agent and a path planning agent based on the road traffic data combined with a space-time coupling matrix, to obtain an optimal joint action; the optimal joint action comprises an optimal signal phase switching time and a vehicle driving path in the specific area;

[0038] an intelligent decision-making module, configured to control traffic lights and vehicles in the specific area by using the optimal signal phase switching time and the vehicle driving path; the joint action space is obtained by forming a hybrid action space through Cartesian product of the signal control agent and the path planning agent, and then constructing the hybrid action space combined with an Actor-Critic framework.

[0039] Further, the dynamic environment modeling module is further configured to construct the joint action space by:

[0040] generating a discrete signal control agent according to a green light duration and a switching transition time in a traffic light;

[0041] generating a continuous path planning agent according to a steering angle and an acceleration during vehicle driving;

[0042] forming a hybrid action space through Cartesian product of the signal control agent and the path planning agent;

[0043] defining a joint action in the hybrid action space as an output of an Actor network in an Actor-Critic framework;

[0044] defining a road traffic state and the joint action as inputs of a Critic network in the Actor-Critic framework, and positioning the joint action taken under the road traffic state as an output of the Critic network;

[0045] constructing the joint action space according to the defined Actor-Critic framework.

[0046] Further, it further comprises a multi-objective optimization module, configured to train the joint action space by:

[0047] taking a road traffic state in the joint action space to be trained as a state quantity of an automatic driving path planning scene;

[0048] The joint action in the mixed action space is taken as the action quantity of the automatic driving path planning scene;

[0049] Based on the state quantity and the action quantity, the improved multi-agent deep deterministic policy gradient (MADDPG) algorithm is used to cooperatively iteratively train the joint action space to be trained, so as to obtain the trained joint action space.

[0050] Further, a multi-objective optimization module is specifically configured to initialize parameters in the joint action space to be trained based on the improved MADDPG algorithm under the constraint of the space-time coupling matrix; the parameters include network parameters of the Actor-Critic framework.

[0051] Based on the state quantity, the Actor-Critic framework with the initialized parameters is used to generate a corresponding action quantity, and a value estimation value of joint execution of the action quantity by the signal control agent and the path planning agent is calculated.

[0052] Based on the value estimation value, network parameters of the Critic network and the Actor network in the Actor-Critic framework are iteratively optimized to obtain the trained joint action space.

[0053] Further, the multi-objective optimization module is specifically configured to calculate a loss value by using a mean square error loss function based on the value estimation value; and optimize the network parameters of the Critic network based on the loss value.

[0054] The joint execution of the action quantity by the signal control agent and the path planning agent is simulated by using a road simulation technology to obtain simulation traffic data.

[0055] Based on the simulation traffic data, a reward function is used to calculate a reward value of the signal control agent and the path planning agent; and the network parameters of the Actor network are optimized based on the reward value.

[0056] The Critic network and the Actor network with the optimized network parameters continue to generate corresponding action quantities until an iteration termination condition is met, so as to obtain the trained joint action space.

[0057] Further, the intelligent decision-making module is further configured to determine a traffic conflict detection result by using a conflict detection function based on the optimal signal phase switching time and the vehicle driving path in the optimal joint action.

[0058] If the traffic conflict detection result is that there is a potential traffic conflict, a preset correction mechanism is used to correct the optimal joint action.

[0059] In another aspect, the present application also provides a computer device, comprising: one or more processors;

[0060] The processor is configured to store one or more programs;

[0061] When the one or more programs are executed by the one or more processors, the automatic driving path planning method described in any one of the above embodiments is implemented.

[0062] In another aspect, the present application also provides a computer readable storage medium, having a computer program stored thereon, wherein the computer program is executed to implement the automatic driving path planning method described in any one of the above embodiments.

[0063] Compared with the prior art, the present application has the following advantages:

[0064] The present application provides an automatic driving path planning method, system, electronic device and medium, the method comprising: an electronic device acquires road traffic data of a specific area, and based on the road traffic data, a time-space coupling matrix is used to obtain a best joint action, and the best joint action includes a best signal phase switching time and a vehicle driving path, which are used to control the traffic signal and the vehicle in the specific area, the joint action space is formed by Cartesian product of the signal control agent and the path planning agent to form a hybrid action space, and then the hybrid action space is combined with the Actor-Critic framework to obtain, in this way, the decision-making process is completely based on objective data and is not affected by external factors such as weather, ensuring the stability of the control, and the traffic efficiency of the vehicle in the specific area can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0065] Figure 1 FIG. 1 is a flowchart of the automatic driving path planning method of the present application;

[0066] Figure 2 FIG. 2 is a structural schematic diagram of the automatic driving path planning system of the present application;

[0067] Figure 3 FIG. 3 is a structural schematic diagram of the electronic device of the present application. DETAILED DESCRIPTION

[0068] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0069] Embodiment 1:

[0070] The automatic driving path planning method provided by the present application has a flowchart as shown in FIG. 1, which comprises: Figure 1

[0071] ​Step 101: acquire road traffic data of a specific area;

[0072] Step 102: based on the road traffic data, combine the space-time coupling matrix to solve the joint action space of the signal control agent and the path planning agent, and obtain the best joint action; the best joint action includes the best signal phase switching time and the vehicle driving path in the specific area;

[0073] Step 103: control the traffic signal and the vehicle in the specific area by using the best signal phase switching time and the vehicle driving path; the joint action space is formed by Cartesian product of the signal control agent and the path planning agent into a hybrid action space, and then the hybrid action space is combined with the Actor-Critic framework to build.

[0074] The automatic driving path planning method provided by the embodiment of the application is applied to an electronic device, which can be a personal computer (PC), a server or other intelligent device.

[0075] In order to accurately and effectively perform automatic driving path planning, the space-time coupling matrix is used in the embodiment of the application to obtain the best joint action, and the best signal phase switching time and the vehicle driving path included in the best joint action are used to control the traffic signal and the vehicle in the specific area. This way not only ensures the stability of the control by making the decision-making process completely based on objective data and not affected by external factors such as weather, but also improves the traffic efficiency of vehicles in the specific area.

[0076] In order to realize automatic driving path planning, the electronic device can first acquire road traffic data of a specific area. Specifically, multi-dimensional road traffic data including vehicle position, speed, traffic signal state and road congestion index can be acquired in real time through roadside sensors, vehicle-mounted devices and traffic monitoring systems.

[0077] In order to accurately and effectively perform path planning, the electronic device can establish a data preprocessing pipeline to perform time stamp alignment, noise filtering and format standardization processing on the acquired road traffic data.

[0078] Based on the multi-source fused road traffic data, the electronic device constructs a space-time coupling matrix to depict the space-time evolution characteristics of the traffic flow. The space-time coupling matrix quantitatively analyzes the traffic state correlation of each road section in the specific area at different time periods, and establishes a collaborative optimization model between signal control and vehicle path.

[0079] Specifically, the space-time coupling matrix describes the interaction of traffic elements from two dimensions of time and space: the time dimension reflects the timing characteristics such as signal cycle and phase difference, and the space dimension represents the topological characteristics such as road capacity and turning relationship. In the solving process, the electronic device can use a deep reinforcement learning algorithm to model the signal control agent and the path planning agent as collaborative decision-making subjects, and through iterative optimization of the joint action space, finally output the best joint action, which includes the signal phase switching time and the vehicle driving path in the specific area.

[0080] In the embodiment of the application, the electronic device adopts a collaborative optimization mechanism based on the Actor-Critic framework to construct a joint action space of signal control and path planning. Specifically, two independent but closely related action domains are defined: the signal control agent is responsible for the discrete action space, which is used to optimize the phase switching strategy of the traffic signal; the path planning agent is responsible for the continuous action space, which is used to dynamically adjust the driving trajectory of the vehicle. Through Cartesian product operation, the discrete and continuous action spaces are deeply integrated to form a unified hybrid action space. Based on the Actor-Critic framework, collaborative learning and decision-making of the two agents are realized: the Critic evaluates the global value of the joint action to provide the optimization direction for the Actor; the Actor optimizes the signal control strategy and the path planning strategy respectively. This design allows the electronic device to consider the changes of the traffic signal and the adjustment of the vehicle driving trajectory at the same time.

[0081] After obtaining the best joint action, the electronic device can control the traffic signal and the vehicle in the specific area using the best signal switching time and the vehicle driving path contained in the best joint action.

[0082] In the embodiment of the application, in order to ensure the smooth operation of the traffic system, a space-time coupling matrix is introduced to coordinate the signal phase switching time window and the vehicle expected arrival time. When selecting actions, considering the design of the joint action space, the signal control agent and the path planning agent need to work together to determine the best joint action. At this time, the space-time coupling matrix helps to determine whether these actions will cause potential conflicts and guides the signal control agent and the path planning agent to make necessary adjustments.

[0083] In the embodiment of the application, the automatic driving path planning is realized based on the multi-agent reinforcement learning algorithm.

[0084] In order to accurately and effectively construct the joint action space, on the basis of the above embodiment, in the embodiment of the application, the construction process of the joint action space includes:

[0085] According to the green light duration and switching transition time of the traffic signal, a discrete signal control agent is generated;

[0086] generate a continuous path planning agent according to a steering angle and an acceleration when a vehicle is driven;

[0087] form a hybrid action space by Cartesian product of the signal control agent and the path planning agent;

[0088] define a joint action in the hybrid action space as an output of an Actor network in an Actor-Critic framework;

[0089] define a road traffic state and the joint action as inputs of a Critic network in the Actor-Critic framework, and define a long-term return of taking the joint action in the road traffic state as an output of the Critic network;

[0090] construct a joint action space according to the defined Actor-Critic framework.

[0091] The construction process of the joint action space is based on deep fusion and collaborative optimization of multi-dimensional traffic elements. The electronic device can first generate a discrete signal control agent according to the green light duration and phase switching transition time in the traffic signal as core parameters. The agent can accurately describe the timing characteristics of signal timing and its influence on traffic flow. In the dimension of vehicle motion control, the electronic device takes the steering angle and acceleration as continuous control variables to generate a continuous path planning agent, which realizes fine modeling of the vehicle driving trajectory. Through Cartesian product operation, the discrete signal control agent and the continuous path planning agent are mathematically mapped to form a unified hybrid action space, thereby completely describing the collaborative relationship between traffic signals and vehicle motion. The hybrid action space is specifically a joint action in space.

[0092] In the specific implementation of the Actor-Critic framework, the electronic device defines the joint action in the hybrid action space as the output of the Actor network, which simultaneously generates a signal control strategy and a path planning strategy through policy optimization. At the same time, the electronic device takes the road traffic state and the joint action as inputs of the Critic network. The road traffic state includes traffic flow density, speed distribution, queue length, etc., and the global utility of the current strategy is evaluated through the value function. The output of the Critic network reflects the long-term return of taking the joint action in the road traffic state, which provides the gradient direction for the policy update of the Actor network. The electronic device can define the long-term return of taking the joint action in the road traffic state as the output of the Critic network. The electronic device constructs the joint action space according to the defined Actor-Critic framework.

[0093] In an example, the electronic device can define the signal control agent as a discrete action space A signal{P1, P2, …, P n} where each phase P i Including green duration (5-60 seconds) and switching transition time (3-5 seconds), the path planning agent can be defined as a continuous action space A vehicle ∈R 2 , R contains steering angle θ∈[-30°,+30°] and acceleration a∈[-3m / s,+3m / s]. The hybrid action space can be represented as A joint =A signal ×A vehicle , the dimension is expanded to n+2.

[0094] In order to obtain the joint action space, on the basis of the above embodiments, in the embodiment of the application, the pre-training process of the joint action space includes:

[0095] The road traffic state in the joint action space to be trained is taken as the state quantity of the automatic driving path planning scene;

[0096] The joint action in the hybrid action space is taken as the action quantity of the automatic driving path planning scene;

[0097] Based on the state quantity and the action quantity, the improved MADDPG algorithm is used for cooperative iterative training of the joint action space to be trained, to obtain the trained joint action space.

[0098] In the embodiment of the application, the pre-training process of the joint action space is realized by constructing a simulation environment for automatic driving path planning. The electronic device can first take the road traffic state in the joint action space to be trained as the state quantity in the automatic driving planning scene, and the road traffic state includes traffic flow density, signal phase, vehicle position and the like, and the state quantity is used to comprehensively describe the dynamic characteristics of the traffic scene. And the joint action in the hybrid action space can be defined as the action quantity of the automatic driving path planning scene, and the joint action is the combination of the signal control strategy and the vehicle path planning strategy, and the action quantity can be used as the decision output of the interaction between the agent and the environment.

[0099] Based on the state quantity and the action quantity, the improved MADDPG algorithm is used for cooperative iterative training of the joint action space to be trained, to obtain the trained joint action space. Wherein, in the training stage, the Critic network evaluates the value of the joint strategy by using the global state and action information, and guides the policy update of the Actor network; in the execution stage, each agent only depends on the local observation information to make independent decisions. Through multiple rounds of iterative training, the electronic device gradually optimizes the cooperative strategy of signal control and path planning, and finally obtains the joint action space which can efficiently cope with complex traffic scenes. This pre-training mechanism significantly improves the convergence speed and strategy stability.

[0100] To obtain the trained joint action space, based on the above embodiments, in the embodiments of the present application, based on the state quantity and the action quantity, the improved MADDPG algorithm is used to cooperatively iteratively train the joint action space to be trained, and the trained joint action space is obtained, including:

[0101] Based on the improved MADDPG algorithm, the parameters in the joint action space to be trained are initialized under the constraint of the space-time coupling matrix; the parameters include the network parameters of the Actor-Critic framework;

[0102] Based on the state quantity, the Actor-Critic framework with the initialized parameters is used to generate the corresponding action quantity, and the value estimation value when the signal control agent and the path planning agent jointly execute the action quantity is calculated;

[0103] Based on the value estimation value, the network parameters of the Critic network and the Actor network in the Actor-Critic framework are iteratively optimized, and the trained joint action space is obtained.

[0104] In the embodiments of the present application, the electronic device can initialize the parameters in the joint action space to be trained under the constraint of the space-time coupling matrix, and the parameters in the joint action space to be trained include the weights, biases and other parameters of the Critic network and the Actor network in the Actor-Critic framework.

[0105] The electronic device generates a joint action quantity including a signal control strategy and a path planning strategy according to the current traffic state by using the initialized Actor network, and evaluates the global value estimation value of the joint action by the Critic network, which comprehensively considers multiple indicators such as traffic efficiency, signal coordination and vehicle travel cost.

[0106] In the optimization phase, the electronic device can update the parameters of the Critic network and the Actor network based on the value estimation value through the gradient backpropagation algorithm or other algorithms: the Critic network improves the accuracy of the value estimation by minimizing the time difference error, and the Actor network optimizes the generation strategy of the joint action through the policy gradient method. Through multiple rounds of iterative training, the optimal strategy is gradually converged, and finally the joint action space that can efficiently coordinate signal control and path planning is obtained, providing a reliable solution for cooperative optimization in complex traffic scenarios.

[0107] In an example, the electronic device can periodically or every step copy the weights of the main network to the target network with a small learning rate to maintain the stability of the training.

[0108] In order to obtain the trained joint action space, based on the above embodiments, in the embodiments of the present application, the network parameters of the Critic network and the Actor network in the Actor-Critic framework are iteratively optimized based on the value estimate value, and the trained joint action space includes:

[0109] Based on the value estimate value, the loss value is calculated by using the mean square error loss function; and the network parameters of the Critic network are optimized based on the loss value;

[0110] The joint action amount of the signal control agent and the path planning agent is simulated by using the road simulation technology, and simulation traffic data is obtained;

[0111] Based on the simulation traffic data, the reward value of the signal control agent and the path planning agent is calculated by using the reward function; and the network parameters of the Actor network are optimized based on the reward value;

[0112] Based on the Critic network and the Actor network with optimized network parameters, the corresponding action amount is continuously generated until the iteration termination condition is met, and the trained joint action space is obtained.

[0113] In the embodiments of the present application, the electronic device can estimate the value of the current policy by using the Critic network, and calculate the difference between the predicted value and the actual value by using the mean square error loss function as the loss value. Based on this loss value, the parameters of the Critic network are updated. In an example, the gradient descent optimization algorithm can be used to update the parameters of the Critic network to improve the accuracy of its value estimation.

[0114] With the help of road simulation technology, the joint action amount of the signal control agent and the path planning agent is simulated, and simulation traffic data is obtained. This process can generate rich simulation traffic data, including vehicle flow, driving speed, signal light state and other key information.

[0115] Based on the obtained simulation traffic data, the electronic device can use the reward function to quantitatively evaluate the performance of the signal control agent and the path planning agent. The reward function considers traffic efficiency, safety, environmental protection and other dimensions to ensure that the behavior of the agent meets the needs of actual traffic management. According to the calculated reward value, the parameters of the Actor network are further optimized.

[0116] After obtaining the Critic network and the Actor network with optimized network parameters, the Critic network and the Actor network with optimized network parameters continue to generate corresponding action amount until the iteration termination condition is met, i.e. the trained joint action space is obtained.

[0117] In order to obtain the trained joint action space, on the basis of the above embodiments, in the embodiment of the application, the reward function satisfies the following formula:

[0118]

[0119] Wherein, R total is the reward value, alpha is a preset weight, X is the overall traffic efficiency of a specific area, N is the number of vehicles in the specific area, x i is the actual driving time of the i-th vehicle in the driving data, is the expected driving time of the i-th vehicle in the driving data.

[0120] The reward function provided by the embodiment of the application comprehensively considers factors such as global traffic efficiency and individual vehicle travel time deviation. This reward structure encourages the signal control agent and the path planning agent to not only pursue local optimal solutions, but also consider the overall performance.

[0121] In order to accurately and effectively perform automatic driving planning, on the basis of the above embodiments, in the embodiment of the application, before controlling the traffic signal and the vehicle in the specific area by using the optimal signal phase switching time and the vehicle driving path, the electronic device further comprises:

[0122] Based on the optimal signal phase switching time and the vehicle driving path in the optimal joint action, the conflict detection function is used to determine a traffic conflict detection result.

[0123] If the traffic conflict detection result is that there is a potential traffic conflict, a preset correction mechanism is used to correct the optimal joint action.

[0124] Before implementing the optimal signal phase switching time and the vehicle driving path to regulate the traffic signal and the vehicle in the specific area, the electronic device also needs to ensure smooth and safe traffic. Specifically, the electronic device can use the conflict detection function to identify potential traffic conflicts based on the determined optimal joint action, and determine a traffic conflict detection result.

[0125] If the traffic conflict detection result is that there is a potential traffic conflict, the electronic device can use a preset correction mechanism to correct the optimal joint action. This mechanism can quickly adjust the signal phase switching time and the vehicle driving path to avoid conflicts. Thus, the harmony and efficiency of the traffic system are ensured.

[0126] In the embodiment of the application, the electronic device uses the conflict detection function to identify possible conflicts, and triggers the corresponding correction mechanism when a conflict is detected, thereby ensuring traffic safety and efficiency.

[0127] Assume that at a crossroad, the traffic signal light plans a specific phase switching time according to the optimal joint action, and at the same time, the vehicle selects the optimal driving path according to the suggestion of the path planning agent. However, the conflict detection function finds that the straight vehicles in the east-west direction (the east-west direction described here is the actual east-west direction in the scene) and the left-turn vehicles in the north-south direction (the north-south direction described here is the actual north-south direction in the scene) (the left and right described here are the actual left and right in the scene) may have a cross conflict at a certain time point. The preset correction mechanism can be to extend the green light time of the east-west direction to let the straight vehicles pass through the intersection as soon as possible. Delay the start time of the left-turn green light in the north-south direction to avoid entering the intersection at the same time as the east-west direction vehicles.

[0128] In the embodiment of the application, the space-time coupling matrix provides the necessary constraint condition, so that the signal control agent and the path planning agent can fully consider the time dynamic characteristics of the traffic flow when making decisions; and the joint action space allows the signal control agent and the path planning agent to explore more complex cooperation modes. Combined with the improved MADDPG algorithm, the signal control agent and the path planning agent can effectively promote the collaborative optimization of the signal control agent and the path planning agent in a complex traffic scene.

[0129] The application discloses a traffic flow optimization and automatic driving path planning method based on reinforcement learning, which dynamically adjusts traffic signal light timing and automatic driving vehicle path planning by constructing an intelligent simulation system of a traffic environment and using a reinforcement learning algorithm. The method comprises a traffic data acquisition module, a dynamic environment modeling module, an intelligent decision-making module, and a multi-objective optimization module. Through a multi-agent reinforcement learning algorithm (MARL), the global optimization of traffic flow can be achieved in a complex traffic scene, congestion can be reduced, and the passing efficiency of automatic driving vehicles can be improved. The application is particularly suitable for the development of intelligent traffic management systems and automatic driving technologies, and provides an innovative solution for future traffic systems.

[0130] Embodiment 2:

[0131] Based on the same inventive concept, the application further provides an automatic driving path planning system, a structural schematic diagram of which is shown in Figure 2 and comprises:

[0132] The traffic data acquisition module 201 is configured to acquire road traffic data of a specific region.

[0133] The dynamic environment modeling module 202 is configured to solve a joint action space of a signal control agent and a path planning agent based on the road traffic data in combination with a space-time coupling matrix, and obtain an optimal joint action. The optimal joint action comprises optimal signal phase switching time and a vehicle driving path in the specific region.

[0134] The intelligent decision module 203 is configured to control the traffic signal and the vehicle in the specific area by using the optimal signal phase switching time and the vehicle driving path; the joint action space is formed by Cartesian product of the signal control agent and the path planning agent, and then the mixed action space is combined with an Actor-Critic framework to obtain the joint action space.

[0135] In a specific implementation, the dynamic environment modeling module 202 is further configured to construct the joint action space by:

[0136] generating a discrete signal control agent according to the green light duration and the switching transition time of the traffic signal;

[0137] generating a continuous path planning agent according to the steering angle and the acceleration of the vehicle during driving;

[0138] forming a mixed action space by Cartesian product of the signal control agent and the path planning agent;

[0139] defining a joint action in the mixed action space as an output of an Actor network in an Actor-Critic framework;

[0140] defining a road traffic state and the joint action as an input of a Critic network in the Actor-Critic framework, and defining a position of the joint action in the road traffic state as an output of the Critic network;

[0141] constructing the joint action space according to the defined Actor-Critic framework.

[0142] In a specific implementation, the method further includes a multi-objective optimization module 204 configured to train the joint action space by:

[0143] taking the road traffic state in the joint action space to be trained as a state quantity of an autonomous driving path planning scene;

[0144] taking the joint action in the mixed action space as an action quantity of the autonomous driving path planning scene;

[0145] performing collaborative iterative training on the joint action space to be trained based on the state quantity and the action quantity by using an improved multi-agent deep deterministic policy gradient (MADDPG) algorithm to obtain a trained joint action space.

[0146] In a specific implementation, the multi-objective optimization module 204 is specifically configured to initialize parameters in the joint action space to be trained under the constraint of the space-time coupling matrix based on the improved MADDPG algorithm; the parameters include network parameters of the Actor-Critic framework;

[0147] Based on the state quantity, the Actor-Critic framework with the initialized parameters is used to generate a corresponding action quantity, and a value estimation value of joint execution of the action quantity by the signal control agent and the path planning agent is calculated.

[0148] Based on the value estimation value, network parameters of the Critic network and the Actor network in the Actor-Critic framework are iteratively optimized to obtain a trained joint action space.

[0149] In a specific implementation, the multi-objective optimization module 204 is specifically configured to calculate a loss value by using a mean square error loss function based on the value estimation value; and optimize network parameters of the Critic network based on the loss value.

[0150] The joint execution of the action quantity by the signal control agent and the path planning agent is simulated by using a road simulation technology to obtain simulation traffic data.

[0151] Based on the simulation traffic data, a reward value of the signal control agent and the path planning agent is calculated by using a reward function; and network parameters of the Actor network are optimized based on the reward value.

[0152] Based on the Critic network and the Actor network with the optimized network parameters, corresponding action quantities are continuously generated until an iteration termination condition is met, and a trained joint action space is obtained.

[0153] In a specific implementation, the intelligent decision module 203 is further configured to determine a traffic conflict detection result by using a conflict detection function based on the best signal phase switching time and the vehicle driving path in the best joint action.

[0154] If the traffic conflict detection result is that there is a potential traffic conflict, the best joint action is corrected by using a preset correction mechanism.

[0155] Embodiment 3:

[0156] As Figure 3As shown, the present application also provides an electronic device, which can be a computer device, a single-chip microcomputer device, a smart mobile device, etc. The electronic device in the embodiment can include a processor, a memory, a transceiver component, etc. The memory, the processor and the transceiver component are connected through a bus; the memory can be used to store an execution program, and the exemplary execution program can include instructions; the processor is used to execute the instructions stored in the memory. The memory can also be used to store data, which can be called and / or modified when the instructions are executed.

[0157] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, and are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the storage medium to implement a corresponding method flow or a corresponding function, so as to implement the steps of the automatic driving path planning method in the above embodiment.

[0158] Embodiment 4:

[0159] Based on the same inventive concept, the present application also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in the electronic device, and is used to store programs and data. It can be understood that the storage medium here can include the built-in storage medium in the electronic device, and of course can also include the expansion storage medium supported by the electronic device. The storage medium provides a storage space, and the storage space stores the operating system of the terminal. Moreover, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk memory. The processor loads and executes one or more instructions stored in the storage medium, which can implement the steps of the automatic driving path planning method in the above embodiment.

[0160] Those skilled in the art will appreciate that embodiments of the application can be devised for a method, a system, or a computer program product. Accordingly, the present application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0161] The present application is described in reference to the flowchart and / or block diagrams of the method, apparatus (system) and computer program product according to embodiments of the application. It will be understood that each block of the flowchart and / or block diagrams, and combinations of blocks in the flowchart and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0162] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart and / or block diagram block or blocks.

[0164] Finally, it should be noted that the above-mentioned embodiments are merely used to illustrate the technical solutions of the present application, but not to limit the scope of protection of the present application. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that: after reading the present application, those skilled in the art can make various changes, modifications or equivalent replacements to the specific embodiments of the application, but these changes, modifications or equivalent replacements are all within the scope of protection of the claims of the application.

Claims

1. An automated driving path planning method, characterized by, The method comprises: acquiring road traffic data of a specific area; solving a joint action space of a signal control agent and a path planning agent based on the road traffic data combined with a space-time coupling matrix to obtain an optimal joint action; the optimal joint action comprises optimal signal phase switching time and vehicle driving path in the specific area; controlling traffic signals and vehicles in the specific area by using the optimal signal phase switching time and vehicle driving path; the joint action space is formed by Cartesian product of the signal control agent and the path planning agent into a hybrid action space, and then is constructed by combining an Actor-Critic framework; the solving of the joint action space of the signal control agent and the path planning agent based on the road traffic data combined with the space-time coupling matrix to obtain the optimal joint action comprises: based on multi-source fusion road traffic data, constructing a space-time coupling matrix to depict the space-time evolution characteristics of traffic flow; combining the space-time coupling matrix, solving the joint action space of the signal control agent and the path planning agent to obtain the optimal joint action.

2. The method of claim 1, wherein, the construction process of the joint action space comprises: generating a discrete signal control agent according to the green light duration and switching transition time of a traffic signal light; generating a continuous path planning agent according to the steering angle and acceleration of vehicle driving; forming a hybrid action space by Cartesian product of the signal control agent and the path planning agent; defining the joint action in the hybrid action space as the output of the Actor network in the Actor-Critic framework; defining the road traffic state and the joint action as the input of the Critic network in the Actor-Critic framework, and positioning the joint action taken under the road traffic state as the output of the Critic network; constructing the joint action space according to the defined Actor-Critic framework.

3. The method of claim 1 or 2, wherein, the pre-training process of the joint action space comprises: taking the road traffic state in the joint action space to be trained as the state quantity of an autonomous driving path planning scene; taking the joint action in the hybrid action space as the action quantity of the autonomous driving path planning scene; based on the state quantity and the action quantity, performing collaborative iterative training on the joint action space to be trained by using an improved multi-agent deep deterministic policy gradient (MADDPG) algorithm to obtain a trained joint action space.

4. The method of claim 3, wherein, the collaborative iterative training of the joint action space to be trained by using the improved MADDPG algorithm based on the state quantity and the action quantity to obtain the trained joint action space comprises: based on the improved MADDPG algorithm, initializing parameters in the joint action space to be trained under the constraint of the space-time coupling matrix; the parameters comprise network parameters of the Actor-Critic framework. generate corresponding action amounts by using the Actor-Critic framework with the initialization parameters based on the state amounts, and calculate a value estimation value when the signal control agent and the path planning agent jointly execute the action amounts; iteratively optimize network parameters of the Critic network and the Actor network in the Actor-Critic framework based on the value estimation value, to obtain a trained joint action space.

5. The method of claim 4, wherein, The iteratively optimizing network parameters of the Critic network and the Actor network in the Actor-Critic framework based on the value estimation value, to obtain a trained joint action space includes: based on the value estimation value, calculate a loss value by using a mean square error loss function; and optimize network parameters of the Critic network based on the loss value; simulate the joint execution of the signal control agent and the path planning agent by using a road simulation technology, to obtain simulation traffic data; based on the simulation traffic data, calculate a reward value of the signal control agent and the path planning agent by using a reward function; and optimize network parameters of the Actor network based on the reward value; continue to generate corresponding action amounts by using the Critic network and the Actor network with optimized network parameters, until an iteration termination condition is met, to obtain a trained joint action space.

6. The method of claim 5, wherein, The reward function satisfies the following formula: wherein R total is a reward value, a is a preset weight, X is the overall traffic efficiency of the specific area, N is the number of vehicles in the specific area, x i is the actual driving time of the i-th vehicle in the simulated traffic data, is the expected driving time of the i-th vehicle in the simulated traffic data.

7. The method of claim 1, wherein, Before the best signal phase switching time and the vehicle driving path are used to control the traffic signal and the vehicle in the specific region, the method further includes: based on the best signal phase switching time and the vehicle driving path in the best joint action, determine a traffic conflict detection result by using a conflict detection function; if the traffic conflict detection result indicates that there is a potential traffic conflict, correct the best joint action by using a preset correction mechanism.

8. An automated driving path planning system, characterized by, The method includes: a traffic data acquisition module, configured to acquire road traffic data of a specific region; a dynamic environment modeling module, configured to solve a joint action space of a signal control agent and a path planning agent based on the road traffic data and a space-time coupling matrix, to obtain a best joint action; the best joint action includes a best signal phase switching time and a vehicle driving path in the specific region; an intelligent decision-making module, configured to control a traffic signal and a vehicle in the specific region by using the best signal phase switching time and the vehicle driving path; the joint action space is obtained by forming a hybrid action space through a Cartesian product of the signal control agent and the path planning agent, and then constructing the hybrid action space by using an Actor-Critic framework; The solving of the joint action space of the signal control agent and the path planning agent based on the road traffic data and the space-time coupling matrix, to obtain the best joint action includes: based on multi-source fused road traffic data, construct a space-time coupling matrix to depict space-time evolution characteristics of a traffic flow; The joint action space of the signal control agent and the path planning agent is solved in combination with a space-time coupling matrix to obtain the optimal joint action.

9. An electronic device, comprising: The method comprises the following steps: at least one processor and a memory; the memory and the processor are connected through a bus; the memory is used for storing one or more programs; when the one or more programs are executed by the at least one processor, the automatic driving path planning method as claimed in any one of claims 1-7 is implemented.

10. A readable storage medium, characterized by, The program is executed on the computer, and the automatic driving path planning method as claimed in any one of claims 1-7 is implemented when the program is executed.

Images