Urban large-scale motor vehicle driving strategy generation method

Abstract

The invention discloses an urban large-scale motor vehicle driving strategy generation method based on dual reinforcement learning, and the method comprises the steps: dividing a training time length into N time slices, and carrying out the cross training of a Q network and an AC network for many times through combining SUMO simulation software, storing an environmental factor matrix and all vehicle intentions and an optimal action set of all road sections when the Q network is trained each time; all environmental factor matrixes and all vehicle intentions of all road sections are used as input, the corresponding optimal action set is used as output, the two full-connection layers are trained, a new road resistance return function is obtained to replace a road resistance return function in the trained Q network, and a motor vehicle driving strategy generation model is obtained; and after TMI control is implemented on traffic, acquiring an environmental factor matrix of each road section of the research area at the current moment in SUMO simulation software and intentions of all vehicles in all road sections as an intention set, and inputting the intention set into a motor vehicle driving strategy generation model to obtain a motor vehicle driving strategy.

Description

Technical Field

[0001] The invention relates to vehicle path planning technology, and in particular to a method for generating a driving strategy for large-scale motor vehicles in a city. Background Art

[0002] Traffic path planning is a key area in intelligent transportation systems, and its purpose is to find the optimal driving path in a given traffic network. With the development of real-time data and big data technology, the introduction of deep reinforcement learning technology has significantly improved the accuracy and efficiency of path planning. Reinforcement learning-based methods are used to learn the optimal driving strategy. Zheng Bolong et al. proposed an AS-AC dynamic online car-hailing path planning algorithm with an action sampling strategy, which optimizes the scheduling strategy by sensing the supply and demand distribution and the mismatch between supply and demand in the road network, and optimizes the vehicle's driving route over a long period of time. Wang et al. proposed a deep reinforcement learning path planning algorithm based on enhanced observation, and introduced a cyclic prediction module to enhance the efficiency of vehicle path planning in complex environments.

[0003] In order to predict the road network congestion after the implementation of variable traffic organization plans (Traffic Management Initiates, TMI), it is necessary to generate large-scale motor vehicle driving strategies. When the above-mentioned reinforcement learning is applied to TMI, there are the following problems: if traditional reinforcement learning is used to learn the motor vehicle driving strategy Π, and then the motor vehicle trajectory planning is carried out based on Π after the implementation of TMI, the real reward function of the motor vehicle is difficult to design; if inverse reinforcement learning is used to infer the reward function of the motor vehicle driving strategy model, the motor vehicle trajectory data is required, but the trajectory data cannot be obtained; because of the existence of these two problems, the existing methods cannot generate motor vehicle driving strategies after the implementation of TMI. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the method for generating urban large-scale motor vehicle driving strategies provided by the present invention solves the problem that the existing reinforcement learning method cannot generate motor vehicle driving strategies after TMI is implemented.

[0005] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is: A method for generating a large-scale urban motor vehicle driving strategy is provided, which comprises the following steps: S1. Divide the training time into N time slices, time slice variables k The initial value of is 1; S2, in k The environmental factor matrix of the road network topology of the study area output by SUMO simulation software and the road resistance reward function obtained by calculating the intentions of all vehicles on all road sections are used for each time segment. Q The network is trained; S3, adopt the k Time segments trained Q The vehicle execution actions predicted by the network are input into the SUMO simulation software to obtain the environmental factor matrix. The AC network is trained using the environmental factor matrix and the calculated intentions of all vehicles on all road sections. S4. When k Less than N , execute steps S2 and S3 for the next time segment until k = N ,save Q The environmental factor matrix and the intentions and optimal action sets of all vehicles on all road sections at each training of the network; S5, using all environmental factor matrices and all vehicle intentions on all road sections as inputs and the corresponding optimal action set as outputs, trains two fully connected layers to obtain a new road resistance reward function; S6. Use the new road resistance feedback function to replace the N The training time segment ends at Q The road resistance reward function in the network is used to obtain the motor vehicle driving strategy generation model; S7. When the traffic is controlled by TMI, the environmental factor matrix of each road section in the study area at the current moment is obtained in the SUMO simulation software, and the intentions of all vehicles in all road sections are calculated as the intention set; S8, merging the environmental factor matrix and the intention set in step S7 and inputting the result into a motor vehicle driving strategy generation model to obtain the motor vehicle driving strategy at the current moment; S9, determine whether the current moment is the predicted end moment, if so, end the algorithm, otherwise, input the motor vehicle driving strategy into the SUMO simulation software, and return to step S7.

[0006] The beneficial effects of the present invention are as follows: the goal of this scheme is to generate motor vehicle decision-making behaviors, and to make the macro traffic congestion phenomenon obtained by large-scale motor vehicles under the guidance of their own decision-making models through simulation consistent with the macro traffic congestion phenomenon in real scenes. Although motor vehicle driving strategies can also be generated based on traditional reinforcement learning methods, it is necessary to manually design the rewards obtained by motor vehicles after making decisions, and it is difficult to guide large-scale motor vehicles to obtain traffic congestion phenomena consistent with real scenes in a simulation environment through artificially designed rewards. On the other hand, if the motor vehicle reward function is learned based on traditional inverse reinforcement learning, a large amount of expert data is required. Such data requires precise video, radar and other collection methods, and trajectory-level motor vehicle data cleaning and processing are required, which is almost impossible to obtain with a large number of motor vehicles at the city level.

[0007] However, with the method of the present invention, the reward value of the motor vehicle in the reinforcement learning process is obtained by dynamic learning of the AC network. The goal of AC network learning is to enable the motor vehicle to make decisions and find paths based on the reward value output by the AC network. Finally, the congestion state obtained in the simulation environment is close to the congestion state in the real scene, thereby achieving the goal of reconstructing real traffic congestion.

[0008] Furthermore, step S2 further comprises: S21. Initialization t =( k -1) B +1, B is the total duration of each time segment; S22, using SUMO simulation software to obtain t The environmental factor matrix of each road section in the road network topology of the research area is constantly studied, and the intentions of all vehicles on all road sections in the road network topology are calculated to obtain the intention set; S23, the environmental factor matrix and the intention set are combined as the road network topology map to obtain the road resistance return function, and input it into Q The network is trained with an exploration rate of ε The greedy strategy selects the optimal action set of the vehicle; S24. Judgment t Is it equal to kB If yes, go to step S3, otherwise save the current training environment factor matrix and all vehicle intentions and optimal action sets of all road sections, and input the optimal action set into the SUMO simulation software. t = t +1, then return to step S22.

[0009] The beneficial effects of the above technical solution are: when traditional reinforcement learning trains the Q network, its road resistance reward function is fixed. Large-scale motor vehicles find their way through the fixed road resistance reward function, and the congestion state presented in the end may not meet the required congestion state. For example, large-scale motor vehicles will cause congestion on certain branch roads through a fixed road resistance reward function, but we hope that the path finding of large-scale motor vehicles can cause congestion on the main roads to reconstruct the actual congestion scene. If the S2 method is adopted, the path finding of large-scale motor vehicles can proceed towards the expected congestion mode.

[0010] Furthermore, the step S3 further comprises: S31, Initialization t =( k -1) B +1, Initialization AC The initial road network topology for network training is Q The network topology obtained by summing up at the first moment of each time segment ; S32, using the road network topology map Train the AC network and map the road network topology Enter the k The end time of the time segment training Q Network, get t The vehicle performs actions at the moment ; S33, use SUMO simulation software to perform actions , get the t +1 time environmental factor matrix for each road section , calculate the t +1 The intentions of all vehicles in all road sections at the moment to obtain the intention set ,right and Ask and take the first k In the time segment t +1 time road network topology ; S34, Judgment t Is it equal to k B, if yes, go to S35, otherwise update = ,make t = t After +1, return to step S32; S35. Judgment k Is it equal to N If yes, output and save all environmental factor matrices and all vehicle intentions and optimal action sets during training, and go to step S5. Otherwise, let k = k +1, and use SUMO vehicle simulation software to execute all the retained optimal action sets in chronological order, and then return to step S21.

[0011] The beneficial effects of the above technical solution are as follows: the traditional road resistance function is a fixed form function, such as the common road resistance function BPR function In actual situations, in addition to road traffic, more road influencing factors must be considered. On the other hand, our goal is to use S3 to obtain a large number of motor vehicles to find their way through the road resistance function, and finally present a congestion state that conforms to the actual scene, which may not necessarily meet the required congestion state. Common road resistance functions, such as the BPR function, cannot meet this requirement.

[0012] Furthermore, the expression of the optimal action set is: ,

[0013] in, and Respectively k In the time segment t The optimal action set selected by the vehicle at each moment and the set of all actions that can be selected by the vehicle; For the k In the time segment t Always training Q network; for Network parameters; a 1. a 2 and a n Represents vehicles 1, 2 and n possible actions, including left turn, right turn, going straight and U-turn.

[0014] The beneficial effect of the above technical solution is that the optimal action set of each motor vehicle is connected in series to form the path-finding trajectory of the motor vehicle. The optimal action set can guide the trajectory allocation of a large number of motor vehicles, so that during the simulation process, a large number of motor vehicles can present a congestion state that conforms to the actual scenario based on their own optimal action set. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 The flowchart is a method for generating large-scale urban motor vehicle driving strategies.

[0016] Figure 2 The figure is a simple example diagram of a road network topology map; (a) is the road network topology map without TMI control; (b) is the road network topology map with TMI control.

[0017] Figure 3 The road network diagram in the designed experiment; (a) is the real road network map of a certain area in Chongqing, and (b) is the SUMO road network map.

[0018] Figure 4 Figure 2 is a road network diagram near a primary school; (a) is the road network topology diagram after TMI control is implemented according to Scheme 1; (b) is the road network topology diagram after TMI control is implemented according to Scheme 2.

[0019] Figure 5 The actual road network topology diagram of Scheme 1 (left) and the topology diagrams before and after the implementation of TMI control (upper right is before control, lower right is after control).

[0020] Figure 6 The following is a simplified diagram of the actual road network topology of Scheme 2 (left) and the topology diagrams before and after the implementation of TMI control (the upper right is before the control, and the lower right is after the control).

[0021] Figure 7This is a curve chart of the global number of vehicles from 0 to 400 s using two different driving strategies: deterministic shortest path and random shortest path in an environment based on the original map.

[0022] Figure 8~Figure 11 This is a comparison chart of the model reward curves for different driving strategies based on the original map environment; specifically: Figure 8 The reward curve generated in 100~200s under the deterministic shortest path driving strategy; Fig. 9 The reward curve generated in 200~300s under the deterministic shortest path driving strategy; Fig.10 This is the reward curve generated in 100~200s under the random shortest path driving strategy; Fig.11 This is the reward curve generated in 200~300s under the random shortest path driving strategy.

[0023] Figure 12~Figure 19 This is a congestion prediction simulation diagram under different TMI schemes; specifically: Fig.12 This is the congestion simulation diagram of the deterministic shortest path benchmark data corresponding to Scheme 1; Fig.13 This is the deterministic shortest path congestion prediction graph corresponding to scheme 1; Fig.14 This is the congestion simulation diagram of the deterministic shortest path benchmark data corresponding to Scheme 2; Fig.15 This is the deterministic shortest path congestion prediction graph corresponding to scheme 2; Fig.16 This is the congestion simulation diagram of the random shortest path benchmark data corresponding to Scheme 1; Fig.17 This is the random shortest path congestion prediction graph corresponding to scheme 1; Fig.18 This is the congestion simulation diagram of the random shortest path benchmark data corresponding to Scheme 2; Fig.19 This is the random shortest path congestion prediction graph corresponding to Scheme 2.

[0024] Fig. 20 and Fig.21 The congestion reduction experiment results of different time slice sizes and the congestion prediction experiment results of the model under different TMI implementation plans are shown in the figure. Fig. 20 The congestion reduction experiment results for the time slice size; Fig.21 Experimental results of congestion prediction for time slice size. DETAILED DESCRIPTION

[0025] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0026] refer to Figure 1 , Figure 1 A flow chart of a method for generating a large-scale urban motor vehicle driving strategy is shown; Figure 1 As shown, the method S includes steps S1 to S9.

[0027] In step S1, the training duration is divided into N time slices, time slice variables k The initial value of is 1; In step S2, k The environmental factor matrix of the road network topology of the study area output by SUMO simulation software and the road resistance reward function obtained by calculating the intentions of all vehicles on all road sections are used for each time segment. Q The network is trained; the environmental factor matrix includes at least the one-hot encoding of the road section, the length of the road section, the number of vehicles on the road section, the average vehicle speed, the number of stationary vehicles, the current travel time and the lane occupancy rate.

[0028] The road network topology diagram of this scheme is a road network topology structure with road sections as nodes G :This solution focuses on simulated vehicle path planning, and in order to represent the changes in the road network after the implementation of TMI, the urban road network is represented as a directed topological structure using road segments as nodes to better represent the connectivity between road segments. Representing the urban road network as a directed topological structure with weights G =( V , E , A ), V is a node set, representing all the road segments in the road network. E is the set of links between nodes ( i , j ), indicating a slave node i To Node j To the edge, where i , j ∈ V , A is the adjacency matrix of the topological graph.

[0029] In order to facilitate the understanding of the road network topology, this scheme Figure 2 A simple example of a road network topology diagram is given in the paper. The actual road network is converted into Figure 2 (a) Road network topology diagram, each node has a road resistance value u i , at this time, vehicles can travel from section 1 to section 8 and section 10. If TMI is implemented, vehicles are prohibited from traveling from section 1 to section 10, and the connection between section 1 and section 10 will be disconnected. Figure 2 (b) as shown.

[0030] In one embodiment of the present invention, step S2 further comprises: S21. Initialization t =( k -1) B +1, B is the total duration of each time segment; S22, using SUMO simulation software to obtain t The environmental factor matrix of each road section in the road network topology of the research area is constantly studied, and the intentions of all vehicles on all road sections in the road network topology are calculated to obtain the intention set; S23, the environmental factor matrix and the intention set are combined as the road network topology map to obtain the road resistance return function, and input it into Q The network is trained with an exploration rate of ε The greedy strategy selects the optimal action set of the vehicle; S24. Judgment t Is it equal to kB If yes, go to step S3, otherwise save the current training environment factor matrix and all vehicle intentions and optimal action sets of all road sections, and input the optimal action set into the SUMO simulation software. t = t +1, then return to step S22.

[0031] When implemented, the expression of the optimal action set selected by this scheme is: ,

[0032] in, and Respectively k In the time segment t The optimal action set selected by the vehicle at each moment and the set of all actions that can be selected by the vehicle; For the k In the time segment t Always training Q network; for Network parameters; a 1. a 2 and a n Represents vehicles 1, 2 and n possible actions, including left turn, right turn, going straight and U-turn.

[0033] During implementation, when the scheme preferably trains the Q network, the update method of the loss function includes: S231, using the road resistance reward function and the reward function to calculate thek In the time segment t Obstacles and rewards at every moment: ,

[0034] in, For the k In a time segment t The road resistance set of all road sections at the moment; For AC network; For the k In a time segment t Road network topology at the moment; The observed global state space; Parameters of the Actor network; are the parameters of the Critic network; For the k In the time segment t Rewards for all vehicles at all times; n For the k In the time segment t The total number of vehicles at the time; , and Respectively i The vehicles pass through the first section, the second path and the j The road blockage of the road section, , and belong ; S232, use SUMO vehicle simulation software to execute the optimal action set , get the k In the time segment t +1 time environmental factor matrix for each road section , calculate the k In the time segment t +1 The intention set of all road segments at the moment ; S233, yes and Ask and take the first k In the time segment t +1 time road network topology , and according to the road network topology and rewards ,renew Q The loss function of the network loss :

[0035] in, is the mean square error loss function; γ is the discount rate.

[0036] In step S3, the k Time segments trained Q The vehicle execution actions predicted by the network are input into the SUMO simulation software to obtain the environmental factor matrix. The AC network is trained using the environmental factor matrix and the calculated intentions of all vehicles on all road sections. In one embodiment of the present invention, step S3 further comprises: S31, Initialization t =( k -1) B +1, Initialization AC The initial road network topology for network training is Q The network topology obtained by summing up at the first moment of each time segment ; S32, using the road network topology map Train the AC network and map the road network topology Enter the k The end time of the time segment training Q Network, get t The vehicle performs actions at the moment ; S33, use SUMO simulation software to perform actions , get the t +1 time environmental factor matrix for each road section , calculate the t +1 The intentions of all vehicles in all road sections at the moment to obtain the intention set ,right and Ask and take the first k In the time segment t +1 time road network topology ; S34, Judgment t Is it equal to k B, if yes, go to S35, otherwise update = ,make t = t After +1, return to step S32; S35. Judgment k Is it equal to N If yes, output and save all environmental factor matrices and all vehicle intentions and optimal action sets during training, and go to step S5. Otherwise, let k = k+1, and use SUMO vehicle simulation software to execute all the retained optimal action sets in chronological order, and then return to step S21.

[0037] In step S4, when k Less than N , execute steps S2 and S3 for the next time segment until k = N ,save Q The environmental factor matrix and the intentions and optimal action sets of all vehicles on all road sections at each training of the network; In step S5, all environmental factor matrices and all vehicle intentions on all road sections are used as inputs, and the corresponding optimal action set is used as output. The two fully connected layers are trained to obtain a new road resistance reward function:

[0038] in, and They are the network parameters of the first and second layers in the two fully connected layers respectively; and There are two fully connected layers respectively; Adopting a new road resistance reward function Replace the step S231 .

[0039] The output of the AC network is the real-time road resistance of each road section in the road network during the simulation process. The training goal of the AC network is to have large-scale motor vehicles find their way according to the road resistance output by the AC network, and finally present a congestion state that conforms to the actual scene. Obtaining road resistance through the above technical route and then guiding large-scale motor vehicles to find their way is more conducive to simulating congestion phenomena that conform to the actual situation.

[0040] In step S6, a new road resistance feedback function is adopted Replace N The training time segment ends at Q Path resistance reward function in the network , and obtain the motor vehicle driving strategy generation model; In step S7, when the traffic is controlled by TMI, the environmental factor matrix of each road section in the study area at the current moment is obtained in the SUMO simulation software, and the intentions of all vehicles in all road sections are calculated as the intention set; In step S8, the environmental factor matrix and the intention set in step S7 are combined and input into the motor vehicle driving strategy generation model to obtain the motor vehicle driving strategy at the current moment; In step S9, it is determined whether the current moment is the predicted termination moment. If so, the algorithm is terminated. Otherwise, the vehicle driving strategy is input into the SUMO simulation software and the process returns to step S7.

[0041] In one embodiment of the present invention, AC When the network is trained, update AC Parameters of the Actor Network in the Network And the parameters of the Critic network The methods include: S321, calculate the k Time segment t Roadblocks of the moment and rewards ; S322, will Stored as a sample in the experience replay pool D , and in the experience replay pool D Select a preset number of samples and calculate the TD error of the samples:

[0042] in, For the k In the time segment t Extracted at any time m The average TD error of samples; γ is the discount rate; is the state value function, i.e., the Critic network; are the parameters of the Critic network; S323, according to TD error ,renew AC Parameters of the Actor Network in the Network And the parameters of the Critic network : ,

[0043] in, is the loss function of the Actor network; is the strategy of the Actor network, that is, in state s t Next select action a t probability; is the loss function of the Critic network, Express expectations.

[0044] In this scheme, the road resistance is AC Network acquisition, AC The Actor network and Critic network of the network are both two-layer graph convolutional networks. AC The methods for calculating the road resistance of the network include: A two-layer graph convolutional network is used to encode the road network topology and road section information to obtain the characteristics of the road section and the rest of the nodes connected to it. The two-layer GCN model is expressed as:

[0045]

[0046] in, is the activation function; and They are the first layer GCN neural network function and the double layer GCN neural network function respectively; A is the adjacency matrix of the road network topology; In the Actor network, the node features processed by the graph convolution layer are transformed through two fully connected layers to generate the mean and standard deviation :

[0047]

[0048] in, is the first fully connected layer function; is a fully connected layer; Based on mean and standard deviation The normal distribution sampling of is used to determine the road resistance of the road network.

[0049] In the Critic network, the node features processed by the graph convolution layer are further generated into state values ​​by global average pooling and two fully connected layers. V , used to evaluate the quality of the Actor network:

[0050]

[0051] in, is the first layer global average pooling function; Global average pooling neural network.

[0052] Since simple neural networks are difficult to represent the topological relationship of the road network, in order to better process the graph structure data of the traffic network, the Actor function and Critic function of this solution are both implemented by graph neural networks. Due to the strong dependencies between traffic road networks, traditional convolutional neural networks can obtain local spatial features, but can only be used for Euclidean space, such as images, regular networks, etc., and the complex urban road network as a non-Euclidean structure graph makes it impossible for traditional convolution to correctly reflect its spatial connection relationship. The emergence of the graph convolutional network model GCN can solve this problem. In the Actor network and Critic network of this solution, double-layer GCN is used to model the road network.

[0053] In one embodiment of the present invention, the method for calculating the intention of all vehicles on each road segment includes: Build k In the time segment t The time-segment-vehicle dynamic graph is defined as G I =( V I , E I ), node collection V I ={ V R , V V}, V R and V V are all road segment nodes and all vehicle nodes respectively. E I It means connecting each road segment with all vehicles on the road segment; Will G I Break it down into each road segment j Self-image G j =( V j , E j ), V j ={ j , V j,v} is the road segment j , V j,v For the road section j All vehicles on board; Using the GNN network to map the self-image of each road section G j Get the vehicle intention and all connected vehiclesV j,v Intent Matrix S j , converge to the road section j , get the aggregation state :

[0054] in, is the GNN neural network, S j Point to the road section j The intention matrix of all vehicles on the network includes the road section that the vehicle will reach next and the destination road section of the vehicle.

[0055] According to the aggregation state , calculate the intention of all vehicles on each road segment:

[0056] in, is the intention of all vehicles on road segment j at time t; Separate t - o , t - o +1, t -1 time section j The intention of all vehicles on board; is any linear function.

[0057] If the calculation of road weights only relies on current road information, potential future information cannot be taken into account. This solution introduces a vehicle intention extraction model into the framework to obtain the driving intentions of all vehicles on a certain road section, such as the vehicle's next driving direction and desired destination, etc., to help the model better understand the dynamic changes of traffic flow and provide forward-looking information for the calculation of road section weights.

[0058] In order to verify the effectiveness of the urban large-scale motor vehicle driving strategy generation method provided by this scheme, the following is an explanation combined with the designed experiment: Simulation environment design The experiment will use a commonly used traffic simulation software SUMO to simulate dynamic traffic, including the movement and trajectory changes of vehicles, and the extraction of current information on urban roads.

[0059] The model interacts with SUMO by calling the API interface provided by SUMO software, achieving close integration with traffic simulation software. By extracting road section information in the SUMO environment in real time, including but not limited to key parameters such as the current average road speed, occupancy rate, and number of vehicles, the dynamic changes of the traffic network are accurately captured, the topology weight is calculated and updated at each time step, and the optimal path of the SUMO simulated vehicle from the current location to the destination is dynamically adjusted.

[0060] This experiment uses a road network and vehicle dataset from a certain area in Chongqing. The dataset consists of two parts: a. Urban road network data, which is the real road data from a central area in a certain district in Chongqing. There are 114 roads and 45 intersections in the area. Figure 3 As shown in (a), it is converted into a SUMO road network graph, as Figure 3 (b) as shown; b. Vehicle data, which is the vehicle passing data from 8:30 to 9:00 am on August 21, 2023. After preprocessing, the origin / destination data (Origin / Destination, O / D) of all vehicles in this time period are obtained.

[0061] Motor vehicle driving strategy design Since it is impossible to directly obtain the congestion conditions of all road sections, it is necessary to synthesize the congestion conditions of the road sections within the time period through the real OD of the vehicle. This solution mainly uses two methods to synthesize congestion data: BPR road resistance function deterministic shortest path method

[32] , Random Shortest Path Method

[33] .

[0062] a. BPR road resistance function determined shortest path method: Based on the known vehicle OD, the BPR road resistance function is used Calculate the actual driving time of a road section under different conditions t , as the weight of the road network topology, where t 0 is the travel time in free flow state, Q is the current traffic flow, C is the capacity of the road, α and β is a model parameter, which is calibrated as recommended by the Federal Highway Administration α =0.15, β =4, when the vehicle reaches the intersection, the Dijkstra algorithm is used to calculate the shortest time path from the current position to the destination to dynamically update the vehicle's driving path.

[0063] b. Stochastic shortest path method based on stochastic user-equilibrium (SUE): Taking into account the heterogeneity of travelers' behavioral preferences, this method does not use the shortest path based on road resistance as a determining factor when selecting a path, but assumes that it obeys random distribution, thereby simulating travelers' different perceived costs of the path. When using the BPR function to calculate the actual travel time of each path t Afterwards, based on t Calculate the three shortest paths from the vehicle's current location to the destination as alternative paths. At this time, the total travel time cost of each alternative path is in Is the path j Previous i The travel time of each road segment and the deterministic utility of the alternative paths , calculate the selection probability of each alternative path:

[0064] The vehicle passes through probability to select a path from three alternative paths, thus simulating the inaccuracy of travelers' behavior.

[0065] Through the above two methods, the simulated vehicle can dynamically find the path under the original road network conditions, and the model is trained in this environment to verify the effectiveness of the D2RL framework, that is, whether the congestion of the road network can be restored under the original road network conditions.

[0066] TMI Solution Design To further verify the applicability of the model in the TMI implementation scenario, this experiment will implement the TMI solution in the road network. Figure 4 There is frequent congestion near a primary school, so a TMI solution is designed to address this congestion.

[0067] Option 1: If Figure 5 As shown, in response to the congestion caused by Road 3 and Road 5, measures such as prohibiting left turns on Road 1 and Road 7 are taken to try to alleviate the congestion on Road 3 and Road 5. At this time, the connection between Road 1 and Road 3 and Road 7 and Road 5 in the road network topology diagram should be disconnected.

[0068] Option 2: If Figure 6 As shown, for the congestion on Road 2, it is speculated that the vehicles on Roads 23 and 4 converge on Road 2, causing the traffic volume to increase. Therefore, it is attempted to prohibit left turns on Road 23; at the same time, prohibit right turns on Road 7 to reduce the traffic volume on Road 4 to alleviate the congestion on Road 2. At this time, the connection between Road 23 and Road 2 and between Road 7 and Road 4 in the road network topology diagram should be disconnected.

[0069] Benchmark Data Synthesis Benchmark data: Based on the above vehicle driving strategy and traffic maps under different TMI conditions, the motor vehicle is driven to drive at each time step. t Get each road segment i Number of vehicles , in the vehicle j Get the total travel time after reaching the destination ,Will and As the benchmark data (Ground Truth Data).

[0070] D2RL output data: The driving strategies of motor vehicles trained based on the D2RL framework and the traffic maps under different TMI conditions guide the driving of motor vehicles. During the training process, the data is collected at each time step. t Get each road segment i Number of vehicles , in the vehicle j Get the total travel time after reaching the destination As shown in Table 1.

[0071] Table 1

[0072] Evaluation indicator design This experiment mainly evaluates the simulation effect from two aspects: the accuracy of vehicle travel time and the accuracy of road congestion restoration. The specific description of the indicators is as follows: a. Accuracy of the number of vehicles on a road section: The accuracy of the number of vehicles on a road section refers to the number of vehicles within a certain period of time. T , road section i exist The difference between the number of vehicles in a time step and the number of synthetic vehicles is used to evaluate the accuracy of the number of vehicles on a road section using the mean absolute error (MAE) and root mean square error (RMSE). The formula is as follows:

[0073]

[0074] in n is the total number of samples, for t Time section i The number of synthetic vehicles, for t Time section i The number of simulated vehicles.

[0075] b. Vehicle total travel time accuracy: Vehicle travel time accuracy refers to the difference between the simulated vehicle travel time and its synthetic travel time. It mainly evaluates whether the vehicle can reach the destination at a specific time. The accuracy directly reflects the effect of the simulation model in reproducing actual traffic conditions. It is evaluated using mean absolute error (MAE) and root mean square error (RMSE). The formula is as follows:

[0076]

[0077] Where m is the total number of vehicle samples, For vehicles j The composite travel time is Simulate travel times for vehicles.

[0078] Comparison Models To verify the effectiveness of the framework, this experiment compares the framework of this solution with the following model.

[0079] (1) ACNoGCN_DQN: Path Impedance Learning : Delete the GCN network in the original framework AC algorithm; vehicle path finding : DQN algorithm; uses the time slicing algorithm (i.e. divides the total time into several time segments).

[0080] (2) AC_Dijkstra: Path Impedance Learning : Same Actor_Critic algorithm as the original framework; vehicle pathfinding : Replace the DQN in the original framework with a simple Dijkstra minimum path-blocking path-finding algorithm; use a time slicing algorithm (i.e. divide the total time into several time segments).

[0081] (3) DQN_CTDE: Road Block Learning : None; Vehicle pathfinding :The vehicle directly uses the DQN algorithm to find its way based on the road network conditions; it uses the time slicing algorithm.

[0082] Algorithm parameter design The model of this solution is set up as follows: Vehicle pathfinding model (Q network): Adam algorithm is used as the optimizer, and the learning rate is 0.001; Path resistance learning model (AC network): Both the Actor network and the Critic network use the Adam algorithm as the optimizer, with a learning rate of 0.001. The experiments are conducted on Pytorch, using an Intel Xeon CPU E5-2690 v3 @ 2.60GHz2.60 GHz, 128-GB RAM and a GeForce RTX 3090 24G GPU.

[0083] Based on the original map environment, two different driving strategies, deterministic shortest path and random shortest path, are used respectively. SUMO simulation is performed continuously for 300 seconds in the same time period. Fig.13 As shown, the blue line corresponds to the curve graph generated by the deterministic shortest path, and the red line corresponds to the curve graph generated by the random shortest path. Since the first 100 seconds is the simulated vehicle startup period, the original vehicle data path is used to complete the simulation from 0 to 100 seconds without training; the D2RL framework is used for model training from 100 to 300 seconds, and 300 to 400 seconds is the simulated vehicle dissipation period without training.

[0084] Congestion restoration experiment of original road network conditions (RQ2) In order to verify the congestion restoration effect of the model, a congestion restoration comparison experiment of different driving strategies in the original map environment was set up. The benchmark data are and . Use the D2RL framework to train in 100-300 seconds, setting the time slice size to 100s, as shown in Figure 8~Figure 12 As shown, the rewards calculated using the reward calculation method of this scheme are calculated by Figure 8~Figure 12 It can be seen that D2RL performs best, with the reward curve rising rapidly and stabilizing at a high value; the reward curves of ACNoGCN_DQN and AC_Dijkstra models rise steadily, but the final cumulative effect is lower than that of this algorithm; the DQN_CTDE model has a poor effect, and the curve does not rise significantly. Since the training of subsequent stages depends on the strategies and actions of the previous stages, errors accumulate. Fig. 9 and Fig.11 The training effect will be lower than Figure 8 and Fig.10 . Tables 2 and 3 show the congestion reduction effects of different models on different benchmark data. Compared with other models, D2RL achieves the best performance on both datasets.

[0085] Table 2 Different models on benchmark data Congestion reduction results on

[0086] Table 3 Different models on benchmark data Congestion reduction results on

[0087] The experimental results show that the DQN_CTDE method, which only uses DQN for vehicle pathfinding, has poor training effect due to the lack of effective learning and optimization of road network impedance in the model; the ACNoGCN_DQN method deletes the GCN network in the AC algorithm, and the model cannot obtain global road network information, which affects the overall performance of the model; the AC_Dijkstra method uses the Dijkstra algorithm for minimum road resistance pathfinding, which is not as flexible and efficient as the DQN based on deep learning in dynamic and complex environments.

[0088] At the same time, due to the benchmark data exist The randomness of data is increased on the basis of The effect is worse than The algorithm of this scheme comprehensively utilizes the AC algorithm, DQN algorithm and GCN network, and uses time slicing to deal with the catastrophic forgetting problem. Through road resistance learning and vehicle path finding, congestion reduction can achieve the best effect.

[0089] Congestion prediction experiment under TMI implementation scenario (RQ3) In order to verify the congestion prediction effect under the TMI implementation scenario, the existing model is applied to the implementation scenarios of Scheme 1 and Scheme 2, and the congestion prediction experiment is completed based on two different driving strategies: deterministic shortest path and random shortest path. The benchmark data of Scheme 1 are and , the benchmark data of Scheme 1 are and .

[0090] Table 4 Different models on benchmark data Congestion reduction results on

[0091] Table 5 Different models on benchmark data Congestion reduction results on

[0092] Table 6 Different models on benchmark data Congestion reduction results on

[0093] Table 7 Different models on benchmark data Congestion reduction results on

[0094] Tables 4 and 5 show the congestion prediction results of two types of synthetic data under different models in the implementation scenario of Solution 1. Tables 6 and 7 show the congestion prediction results of two types of synthetic data under different models in the implementation scenario of Solution 2. The prediction effect of D2RL is significantly higher than that of other models.

[0095] Figure 12~Figure 19 The congestion prediction simulation results based on different schemes and different data synthesis methods using D2RL are demonstrated. It can be seen that the model has good performance in different TMI implementation scenarios and can effectively cope with the problem of changes in road network scenarios.

[0096] Experiment with time slice size (i.e. dividing the total time into multiple time slices) Since the time slice size can have a significant impact on the performance and behavior of the model, time slices of different sizes are designed to test the model restoration performance and the prediction performance after TMI implementation.

[0097] Fig. 20 and Fig.21 The congestion reduction experiment results of different time slice sizes and the congestion prediction experiment results of the model under different TMI implementation plans are shown in Figure 2. Fig. 20 It can be seen that the smaller the time slice, the better the results. Fig.21 In the prediction experiment, a smaller time slice will lead to a worse model effect. This is because the smaller the time slice, the more the model overfits. At the same time, the larger the time slice, the worse the restoration and prediction effects, leading to catastrophic forgetting in the model. When the time slice is 100s, the overall performance of the model is the best.

[0098] In summary, the above designed experiments verify that this solution can simultaneously learn the driving strategy and road resistance feedback of motor vehicles. It verifies that large-scale motor vehicle pathfinding can restore congestion phenomena that conform to the actual situation in both the original scenario and the TMI implementation scenario.

Claims

1. A method for generating a large-scale urban motor vehicle driving strategy, characterized in that: Includes steps: S1. Divide the training time into N time slices, time slice variables k The initial value of is 1; S2, in k The environmental factor matrix of the road network topology of the study area output by SUMO simulation software and the road resistance reward function obtained by calculating the intentions of all vehicles on all road sections are used for each time segment. Q The network is trained; S3, adopt k Time segments trained Q The vehicle execution actions predicted by the network are input into the SUMO simulation software to obtain the environmental factor matrix. The AC network is trained using the environmental factor matrix and the calculated intentions of all vehicles on all road sections. S4. When k Less than N , execute steps S2 and S3 for the next time segment until k = N ,save Q The environmental factor matrix and the intentions and optimal action sets of all vehicles on all road sections at each training of the network; S5, using all environmental factor matrices and all vehicle intentions on all road sections as inputs and the corresponding optimal action set as outputs, trains two fully connected layers to obtain a new road resistance reward function; S6. Use the new road resistance feedback function to replace the N The training time segment ends at Q The road resistance reward function in the network is used to obtain the motor vehicle driving strategy generation model; S7. When the traffic is controlled by TMI, the environmental factor matrix of each road section in the study area at the current moment is obtained in the SUMO simulation software, and the intentions of all vehicles in all road sections are calculated as the intention set; S8, merging the environmental factor matrix and the intention set in step S7 and inputting the result into a motor vehicle driving strategy generation model to obtain the motor vehicle driving strategy at the current moment; S9, determine whether the current moment is the predicted end moment, if so, end the algorithm, otherwise, input the motor vehicle driving strategy into the SUMO simulation software, and return to step S7.

2. The method for generating urban large-scale motor vehicle driving strategies according to claim 1, characterized in that: Step S2 further comprises: S21. Initialization t =( k -1) B +1, B is the total duration of each time segment; S22, using SUMO simulation software to obtain t The environmental factor matrix of each road section in the road network topology of the research area is constantly studied, and the intentions of all vehicles on all road sections in the road network topology are calculated to obtain the intention set; S23, the environmental factor matrix and the intention set are combined as the road network topology map to obtain the road resistance return function, and input it into Q The network is trained with an exploration rate of ε The greedy strategy selects the optimal action set of the vehicle; S24. Judgment t Is it equal to kB If yes, go to step S3, otherwise save the current training environment factor matrix and all vehicle intentions and optimal action sets of all road sections, and input the optimal action set into the SUMO simulation software. t = t +1, then return to step S22.

3. The method for generating urban large-scale motor vehicle driving strategies according to claim 2, characterized in that: The step S3 further comprises: S31, Initialization t =( k -1) B +1, Initialization AC The initial road network topology for network training is Q The network topology obtained by summing up at the first moment of each time segment ; S32, using the road network topology map Train the AC network and map the road network topology Enter the k The end time of the time segment training Q Network, get t The vehicle performs actions at the moment ; S33, using SUMO simulation software to perform actions , get the t +1 time environmental factor matrix for each road section , calculate the t +1 The intentions of all vehicles in all road sections at the moment to obtain the intention set ,right and Ask and take the first k In the time segment t +1 time road network topology ; S34, Judgment t Is it equal to k B, if yes, go to S35, otherwise update = ,make t = t After +1, return to step S32; S35. Judgment k Is it equal to N If yes, output and save all environmental factor matrices and all vehicle intentions and optimal action sets during training, and go to step S5. Otherwise, let k = k +1, and use SUMO vehicle simulation software to execute all the retained optimal action sets in chronological order, and then return to step S21.

4. The method for generating urban large-scale motor vehicle driving strategies according to claim 2, characterized in that: The expression of the optimal action set is: ， in, and Respectively k In the time segment t The optimal action set selected by the vehicle at each moment and the set of all actions that can be selected by the vehicle; For the k In the time segment t Always training Q network; for Network parameters; a 1. a 2 and a n Represents vehicles 1, 2 and n possible actions, including left turn, right turn, going straight and U-turn.

5. The method for generating urban large-scale motor vehicle driving strategies according to claim 4, characterized in that: When training the Q network, the update method of its loss function includes: S231, using the road resistance reward function and the reward function to calculate the k In the time segment t Obstacles and rewards at every moment: ， in, For the k In a time segment t The road resistance set of all road sections at the moment; For AC network; For the k In a time segment t Road network topology at the moment; The observed global state space; Parameters of the Actor network; are the parameters of the Critic network; For the k In the time segment t Rewards for all vehicles at all times; n For the k In the time segment t The total number of vehicles at the time; , and Respectively i The vehicles pass through the first section, the second path and the j The road blockage of the road section, , and belong ; S232, use SUMO vehicle simulation software to execute the optimal action set , get the k In the time segment t +1 time environmental factor matrix for each road section , calculate the k In the time segment t +1 The intention set of all road segments at the moment ; S233, yes and Ask and take the first k In the time segment t +1 time road network topology , and according to the road network topology and rewards ,renew Q The loss function of the network loss : in, is the mean square error loss function; γ is the discount rate.

6. The method for generating urban large-scale motor vehicle driving strategies according to claim 3 is characterized in that: AC When the network is trained, update AC Parameters of the Actor Network in the Network And the parameters of the Critic network The methods include: S321, calculate the k Time segment t Roadblocks of the moment and rewards ; S322, will Stored as a sample in the experience replay pool D , and in the experience replay pool D Select m samples, calculate the TD error of the samples: in, For the k In the time segment t Extracted at any time m The average TD error of samples; γ is the discount rate; is the state value function, i.e., the Critic network; are the parameters of the Critic network; S323, according to TD error Calculate the loss function of the actor network and the critic function, and use the loss function to update AC Parameters of the Actor Network in the Network And the parameters of the Critic network : ， in, is the loss function of the Actor network; is the strategy of the Actor network, that is, in state s t Next select action a t probability; is the loss function of the Critic network, Express expectations.

7. The method for generating urban large-scale motor vehicle driving strategies according to claim 5 or 6, characterized in that: The road resistance adopts AC Network acquisition, AC The Actor network and Critic network of the network are both two-layer graph convolutional networks. AC The methods for calculating the road resistance of the network include: A two-layer graph convolutional network is used to encode the road network topology and road section information to obtain the characteristics of the road section and the rest of the nodes connected to it. The two-layer GCN model is expressed as: in, is the activation function; and They are the first layer GCN neural network function and the double layer GCN neural network function respectively; A is the adjacency matrix of the road network topology; In the Actor network, the node features processed by the graph convolution layer are transformed through two fully connected layers to generate the mean and standard deviation : in, is the first fully connected layer function; is a fully connected layer; Based on mean and standard deviation Normal distribution sampling is used to determine the road resistance of the road network; In the Critic network, the node features processed by the graph convolution layer are further generated into state values ​​by global average pooling and two fully connected layers. V , used to evaluate the quality of the Actor network: in, is the first layer global average pooling function; Global average pooling neural network.

8. The method for generating a large-scale urban motor vehicle driving strategy according to any one of claims 1 to 6, characterized in that: The method to calculate the intention of all vehicles on each road segment includes: Build k In the time segment t The time-segment-vehicle dynamic graph is defined as G I =( V I , E I ), node collection V I ={ V R , V V }, V R and V V are all road segment nodes and all vehicle nodes respectively. E I It means connecting each road segment with all vehicles on the road segment; Will G I Break it down into each road segment j Self-image G j =( V j , E j ), V j ={ j , V j,v } is the road segment j , V j,v For the road section j All vehicles on board; Using the GNN network to map the self-image of each road section G j Get the vehicle intention and all connected vehicles V j,v Intent Matrix S j , converge to the road section j , get the aggregation state : in, is the GNN neural network, S j Point to the road section j The intention matrix of all vehicles on the network includes the road section that the vehicle will reach next and the destination road section of the vehicle. According to the aggregation state , calculate the intention of all vehicles on each road segment: in, is the intention of all vehicles on road segment j at time t; Separate t - o , t - o +1, t -1 time section j The intention of all vehicles on board; is any linear function.

9. The method for generating urban large-scale motor vehicle driving strategies according to claim 5, characterized in that: The new road resistance reward function The expression is: in, and They are the network parameters of the first and second layers in the two fully connected layers respectively; and There are two fully connected layers respectively; Adopting a new road resistance reward function Replace the step S231 .

10. The method for generating a large-scale urban motor vehicle driving strategy according to any one of claims 1 to 6 and 9, characterized in that: The environmental factor matrix includes at least one-hot coding of the road section, the length of the road section, the number of vehicles on the road section, the average speed of vehicles, the number of stationary vehicles, the current travel time and the lane occupancy rate.

Images