Large road network multi-vehicle path load balancing method based on reinforcement learning graph neural network

Abstract

The invention provides a large road network multi-vehicle path load balancing method based on a reinforcement learning graph neural network, and the method comprises the steps: obtaining road network data in a real scene, carrying out the modeling of a road network, and representing the modeling as a graph structure; splitting the graph structure into a plurality of sub-graph structures, wherein each sub-graph structure corresponds to a sub-road network; constructing and training a reinforcement learning adaptive graph neural network; for each sub-road network of the road network, calling the trained reinforcement learning adaptive graph neural network to generate a corresponding road selection probability in parallel, combining the generated road selection probabilities of the sub-road networks according to the topological structure of the original road network, and recovering the road selection probabilities of all the roads in the original road network into complete road selection probabilities in the original road network; and realizing road load balancing based on the selection probability of each road in the complete original road network. In a large-scale road network scene, efficient path re-planning is realized, the road network load is reduced in real time, the problem of road congestion is solved, and the traffic efficiency is improved.

Description

Technical Field

[0001] The artificial intelligence algorithm to which the present invention belongs specifically relates to three technical fields: graph neural network, reinforcement learning and load balancing. Background Art

[0002] In the real world, with the continuous advancement of urbanization, road traffic congestion is becoming increasingly serious. To address this problem, various road network optimization methods and traffic management strategies have been proposed.

[0003] However, in large-scale road network scenarios, traditional methods often face challenges of high computational complexity, poor real-time performance, and poor adaptability. Summary of the invention

[0004] This application proposes a large-scale road network multi-vehicle load balancing method based on reinforcement learning graph neural network, aiming to solve the road congestion problem in real scenarios.

[0005] The present invention is based on the idea of ​​decentralized transmission of data streams in load balancing, analogizes network nodes to traffic intersections, and transmission lines to lanes, and is applied to traffic dispersion tasks in real traffic scenarios. By training a reinforcement learning adaptive graph neural network, it adapts to road network graph structures of different sizes. At the application level, by dividing a large road network into small sub-road networks, and calling the trained reinforcement learning adaptive graph neural network to generate the corresponding road selection probabilities in parallel, the generated sub-road selection probabilities are merged according to the topological structure of the original graph, and restored to the complete road selection probabilities in the large road network. The vehicle adjusts the driving path in real time according to the road selection probability of the complete large road network, thereby realizing efficient path re-planning, ultimately achieving the effect of load balancing, and significantly shortening the planning time.

[0006] Technical solution: The method for multi-vehicle path load balancing in a large road network based on reinforcement learning graph neural network includes the following steps: Step 1: Obtain road network data in real scenarios, model the road network, and represent it as a graph structure; Step 2: Split the graph structure into multiple subgraph structures, each subgraph structure corresponds to a sub-network; Step 3: Build and train a reinforcement learning adaptive graph neural network; Step 4: For each sub-network of the road network, call the trained reinforcement learning adaptive graph neural network to generate the corresponding road selection probability in parallel, merge the generated sub-network road selection probabilities according to the topological structure of the original road network, and restore the road selection probabilities of each road in the complete original road network; Step 5: Based on the selection probability of each road in the complete original road network, road load balancing is achieved.

[0007] Beneficial effects of the present invention: The present invention can achieve efficient path re-planning in large-scale road network scenarios, reduce road network load in real time, solve road congestion problems, and improve traffic efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 This is a schematic diagram of selecting subgraphs and radiating them through geographical distribution according to an embodiment.

[0009] Figure 2 This is a schematic diagram of the segmentation, reorganization and restoration process of the embodiment.

[0010] Figure 3 Schematic diagram of the training principle of embedded graph neural network for reinforcement learning.

[0011] Figure 4 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0012] Graph neural network is a machine learning model specially designed for processing graph structured data. It can effectively learn the representation of nodes and edges in the graph. In road network optimization and traffic management, the road network is regarded as a graph, where nodes represent intersections and edges represent roads connecting intersections. By training graph neural networks, it is possible to generate the probability of road selection for vehicles to choose, thereby achieving more intelligent and dynamic path planning, and optimizing the vehicle's driving route according to real-time traffic conditions.

[0013] Reinforcement learning is a machine learning method that aims to learn the optimal decision-making strategy through the interaction between the agent and the environment. In this invention, the reinforcement learning method is applied to train an embedded model that can guide the path selection of multiple vehicles based on real-time road congestion data and road network weights. By continuously interacting with the environment, the embedded model can learn the optimal path selection strategy that adapts to different road congestion conditions and make real-time adjustments in the real-time environment.

[0014] Load balancing is a key technology for balancing traffic transmission tasks. In the present invention, load balancing is applied to reduce the load of large road networks and solve the road congestion problem in real scenarios. By dispersing the traffic of vehicles to different roads, it is possible to achieve rational use of resources and optimization of network performance. The application of network load balancing can effectively improve the road capacity and traffic efficiency.

[0015] The present invention realizes a large-scale road network multi-vehicle path load balancing method based on reinforcement learning graph neural network by comprehensively using three technologies: graph neural network, reinforcement learning and network load balancing, thereby reducing the road network load, solving road congestion problems and improving traffic efficiency.

[0016] The technical solution provided by the present application will be further described below in conjunction with specific embodiments and accompanying drawings. The advantages and features of the present application will become more apparent with the following description.

[0017] Example A multi-vehicle path load balancing method for a large road network based on a reinforcement learning graph neural network includes the following steps: Step 1: Obtain road network data in real scenarios, model the road network, and represent it as a graph structure.

[0018] 1.1 Collect road network data in real scenarios, including the longitude and latitude of intersections, the starting and ending intersections of roads and their corresponding longitude and latitude positions.

[0019] 1.2 Taking intersections as nodes and roads as edges, the road network is modeled as a graph structure, represented as ,in: is a node set, including intersection coordinate information; is the edge set, The elements in it correspond to each road, including the starting node and the ending node of the road.

[0020] Step 2: Split the graph structure into multiple subgraph structures, each subgraph structure corresponds to a sub-network.

[0021] The central node of each subgraph is selected according to the degree of dispersion of the geographical location. By calculating the distance between each node and the central node, it is determined whether the current node is within the coverage of the central node to determine the effective range of the subgraph.

[0022] Since the action space in reinforcement learning corresponds to the probability of road selection, and the probability of road selection is related to the number of roads, the action space is related to the number of edges in the graph structure. are closely related, so the larger the graph structure, The larger the value, the corresponding calculation speed increases linearly. Therefore, by dividing the graph structure into small subgraphs, calculating the road selection probability of each subgraph in parallel and then reorganizing it, the calculation efficiency can be improved.

[0023] The specific method of graph structure splitting is as follows: according to the geographical distribution, the central node of the static subgraph is selected, and the scope of the subgraph is determined by radiating from the central node to the surrounding area. Figure 1 As shown in the figure. Let the central node be I, and the relationship between it and its adjacent node J includes the distance between them and the radiation range. Depending on the size of the original image, the distance between the central node I and the adjacent node J is Need to adjust accordingly, radiation range The number of reference nodes also needs to be changed accordingly. and actual map size , the distance between the central node I and the adjacent node J and radiation range for: Among them, γ is the map range adjustment parameter, λ is the radiation range adjustment parameter, and “[]” is the rounding function. That is: The distance between the central node I and the adjacent node J The map range tuning parameter γ is multiplied by the number of nodes and the square root of the map size; Radiation range It is the map range adjustment parameter λ multiplied by the rounded number of hops of half the distance (the number of hops is the minimum number of nodes that a node needs to pass through to reach the current node).

[0024] like Figure 2 As shown, the map is divided into four sub-maps according to geographical regions, namely Figure 1 ,son Figure 2 ,son Figure 3 ,son Figure 4 At this time, since the edges connecting the subgraphs do not have corresponding structures, Figure 1 ,son Figure 2 ,son Figure 3 ,son Figure 4 As the fifth subgraph nodes, their connecting edges are regarded as the edges of the fifth subgraph. At this time, the connection relationship can be abstracted into a new subgraph, namely subgraph 5.

[0025] Step 3: Build and train a reinforcement learning adaptive graph neural network The size and structure of each subgraph are different, so it is necessary to design a reinforcement learning adaptive graph neural network to meet the input of subgraph structures of different sizes and corresponding road congestion levels.

[0026] 3.1 Build a reinforcement learning adaptive graph neural network.

[0027] like Figure 3 As shown in the figure, the reinforcement learning adaptive graph neural network includes two graph convolution layers, namely the first graph convolution layer GraphConv1, the second graph convolution layer GraphConv2, and a fully connected layer FC. The activation functions of the first graph convolution layer GraphConv1 and the second graph convolution layer GraphConv2 are ReLU, and the activation functions of the second graph convolution layer GraphConv2 and the fully connected layer FC are ReLU.

[0028] Reinforcement learning adaptive graph neural network, whose input is the road congestion data of the subgraph, and whose output is the subgraph edge selection (lane) probability. When the road congestion is higher, the corresponding road selection probability will be lower, thereby reducing the load of the road.

[0029] In the embodiment, the first graph convolution layer GraphConv1, the convolution layer input size is the embedding layer vector size, the output size is 128; the second graph convolution layer GraphConv2, the input size is 128, the output size is the action vector Size; Fully connected layer FC, the input size is the action vector Size, the output size is the action vector Size. Reinforcement learning adaptive graph neural network, through the input vector Generate a set of action vectors , Choose probability for the road.

[0030] 3.2 Training reinforcement learning adaptive graph neural network.

[0031] The reinforcement learning PPO (Proximal Policy Optimization) algorithm is used to train the reinforcement learning adaptive graph neural network and its evaluation network. Figure 3 The embedding vector of the implicit subgraph structure information and lane congestion information is The input is sent to the reinforcement learning adaptive graph neural network, and after passing through the first graph convolution layer GraphConv1, the second graph convolution layer GraphConv2, and the fully connected layer, an output is generated, namely the road selection probability corresponding to the subgraph.

[0032] Specifically, the training of the reinforcement learning adaptive graph neural network is implemented based on the simulator (system). The road network corresponding to the subgraph structure is mapped into the simulator (system), and vehicles are generated in the simulator (system) to simulate the vehicle driving process. During the vehicle driving process, the simulator (system) reads the current congestion level of each lane in real time. The congestion level of the lane is calculated as follows: number of vehicles × vehicle length / road length, 0 indicates an empty lane, and 1 indicates that the lane is occupied by vehicles. The lane congestion level changes in real time with the changes in the traffic simulation in the simulator (system).

[0033] 3.2.1 Start the simulator, create a vehicle, initialize the current lane information and vehicle information, obtain the corresponding lane congestion information, and embed it into the current time vector .

[0034] 3.2.2 At each time step t, obtain the current graph structure and the corresponding road congestion level, and generate the vector corresponding to time t Input reinforcement learning adaptive graph neural network and output the current moment , which is the road selection probability. The process includes: 3.2.2.1 Obtain the subgraph corresponding to the sub-road network and its corresponding road congestion information, and embed it into a vector ,Will As the feature of all lanes, it is represented as a state in the reinforcement learning algorithm. , s represents the state, and t represents the current time step.

[0035] 3.2.2.2 Generating Actions through Action Networks (Reinforcement Learning Adaptive Graph Neural Networks) , that is, the road selection probability of each lane in the subgraph.

[0036] 3.2.2.3 In the simulator (system), when the vehicle obtains the current road selection probability, in each subgraph, the path of each vehicle itself is updated, and the road corresponding to the current vehicle driving section is obtained. Choose the next road to travel, thereby modifying your trajectory in real time.

[0037] 3.2.2.4 Based on reward Iterate and update the parameters of the reinforcement learning adaptive graph neural network.

[0038] 3.2.3 Repeat step 3.2.2 for N time steps (500 time steps in the example) to complete a full set of training. 3.2.4 Repeat steps 3.2.2 and 3.2.3 for M sets (200 sets in the example), save the parameters of the reinforcement learning adaptive graph neural network, and complete the training.

[0039] The following is the state in 3.2.2 , action And reward function reward Detailed definition of .

[0040] The State: As reinforcement learning input, for different graph structures, the embedded vector As state input, the reinforcement learning algorithm needs to obtain road congestion information of graphs of different sizes, so it is unified into vectors of the same size. In order to meet the needs of reinforcement learning training.

[0041] The Action: as the reinforcement learning output, is related to the number of edges Same size output , corresponding to the road selection probability.

[0042] The function design of the Reward consists of three parts, namely the first part reward, the second part reward, and the third part reward: The first part of the reward is determined by the congestion level of the lane. The congestion level of the lane (side) is defined as the ratio of the total length of all vehicles to the total length of the lane, denoted as . The value reflects the congestion level of lane i, i∈E, E is the edge set. For example, if a car is 2m long, there are 50 cars in total, and the lane length is 200m, then The larger the congestion value, the higher the congestion level. Therefore, the first part of the reward is recorded as The second part, reward, is determined by the BPR value. In traffic science, the BPR value is an impedance function, which is Determine. Among them, T is the actual travel time of the current lane, T0 is the free flow speed travel time of the lane, D is the number of vehicles in the current lane, and C is the vehicle capacity of the current lane. a and b are parameters, and the general values ​​are a=0.15 and b=4. The larger the value of T, the greater the impedance of the road, and the longer the travel time. This application measures the value of the second reward function by calculating the T-T0 value, let is the T-T0 value of road i, and the second part reward is The third part, reward, has a value related to whether the path selected by the vehicle is the shortest path edge. Check whether the next road selected by the vehicle in the current situation is an edge on the shortest path. If so, set its reward value to 1, otherwise to 0. H is the set of vehicles, Indicates vehicle v Whether the currently selected path is the reward value of the shortest path edge.

[0043] Combining the above three reward parts, the final reward value function is: Step 4: For each sub-network of the road network, call the trained reinforcement learning adaptive graph neural network to generate the corresponding road selection probabilities in parallel, merge the generated sub-network road selection probabilities according to the topological structure of the original large road network, and restore them to the complete road selection probabilities in the original large road network.

[0044] After splitting, since the sub-graph corresponds to local road network information, it needs to be reassembled. According to the results of local calculation after splitting into sub-graphs (i.e., the sub-graph road selection probability), the road selection probability corresponding to the original large graph is composed to meet the vehicle's pathfinding task on the complete graph and balance the degree of road congestion.

[0045] At different times, the probability of selecting a road in the original complete road network will change. Since the degree of road congestion will change according to the number of vehicles on the current road at different times, it will affect the input of the reinforcement learning adaptive graph neural network, and the corresponding edge selection probability will also change accordingly. The reinforcement learning adaptive graph neural network will identify the corresponding edge congestion degree and generate edge selection probability. For example, if the edge is too congested, the edge selection probability will decrease, and vice versa.

[0046] like Figure 2 As shown in the figure, after the reinforcement learning adaptive graph neural network processes the subgraphs, the edge selection probabilities of each subgraph are obtained as [1a, 1b, 1c, 1d, 1e, 1f, 1g], [2a, 2b, 2c, 2d, 2e, 2f, 2g], [3a, 3b, 3c, 3d, 3e, 3f, 3g], [4a, 4b, 4c, 4d, 4e], [5a, 5b, 5c, 5d, 5e, 5f, 5g, 5h]. They are reassembled to correspond to the edge selection probability of each subgraph in the original large graph. The vehicle will select a road according to the road selection probability corresponding to the complete road network as the actual driving path of the vehicle.

[0047] Step 5: Achieve road load balancing. After each vehicle obtains the road selection probability corresponding to the complete road network, it refers to the shortest path in space to make a detour or choose the next road to travel on the original route, which not only meets its own travel needs, but also reduces the overall road traffic load and improves road traffic efficiency.

[0048] When the vehicle arrives at an intersection, the road selection probabilities of all roads connected to the current intersection are obtained, and then the next road to be entered is selected. The next road to be entered is selected based on two influencing factors: Influencing factor 1: Whether the next road belongs to the shortest path of the corresponding vehicle in the space of the original graph.

[0049] Influencing factor 2: Whether the next road is the road with the highest probability of being selected from the edge connected to the current intersection.

[0050] When there are fewer vehicles, the road with the shortest spatial distance is selected as the road to be entered according to the influence shadow 1, thereby reducing the vehicle's driving resource consumption; When there are many vehicles and the road is congested, the road with the highest probability of being selected from the edge connected to the current intersection is selected according to the influence shadow 2, that is, the road with the lowest congestion level is selected as the road to be entered.

[0051] The above description is only a description of the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Any changes or modifications made by any person skilled in the art based on the above disclosed technical contents shall be deemed as equivalent effective embodiments and shall fall within the scope of protection of the technical solution of the present application.

Claims

1. A large-scale road network multi-vehicle path load balancing method based on reinforcement learning graph neural network, characterized in that: The steps include: Step 1: Obtain road network data in real scenarios, model the road network, and represent it as a graph structure; Step 2: Split the graph structure into multiple subgraph structures, each subgraph structure corresponds to a sub-network; Step 3: Build and train a reinforcement learning adaptive graph neural network; Step 4: For each sub-network of the road network, call the trained reinforcement learning adaptive graph neural network to generate the corresponding road selection probability in parallel, merge the generated sub-network road selection probabilities according to the topological structure of the original road network, and restore the road selection probabilities of each road in the complete original road network; Step 5: Based on the selection probability of each road in the complete original road network, road load balancing is achieved.

2. The method for load balancing multiple vehicle paths in a large road network based on reinforcement learning graph neural network according to claim 1, characterized in that: The step 1 comprises: 1.1 Collect road network data in real scenarios, including the longitude and latitude of intersections, the starting and ending intersections of roads and their corresponding longitude and latitude positions; 1.2 With intersections as nodes and roads as edges, the road network is modeled as a graph structure, represented as ,in: is a node set, including intersection coordinate information; is the edge set, The elements in it correspond to each road, including the starting node and the ending node of the road.

3. The method for load balancing multiple vehicle paths in a large road network based on reinforcement learning graph neural network according to claim 1, characterized in that: The step 2 is to select the central node of each subgraph according to the dispersion degree of the geographical location, and to determine whether the current node is within the coverage range of the central node by calculating the distance between each node and the central node, so as to determine the effective range of the subgraph; Let the central node be I, and the relationship between it and its adjacent node J includes the distance between them and the radiation range; the number of reference nodes and actual map size , the distance between the central node I and the adjacent node J and radiation range for: Among them, γ is the map range adjustment parameter, λ is the radiation range adjustment parameter, and "[]" is the rounding function.

4. The method for load balancing multiple vehicle paths in a large road network based on reinforcement learning graph neural network according to claim 1, characterized in that: Step 3, The reinforcement learning adaptive graph neural network includes two graph convolution layers, namely a first graph convolution layer GraphConv1, a second graph convolution layer GraphConv2, and a fully connected layer FC; the activation functions of the first graph convolution layer GraphConv1 and the second graph convolution layer GraphConv2 are ReLU, and the activation functions of the second graph convolution layer GraphConv2 and the fully connected layer FC are ReLU; The reinforcement learning adaptive graph neural network has its input as the road congestion data of the subgraph and its output as the subgraph road selection probability.

5. The method for load balancing multiple vehicle paths in a large road network based on reinforcement learning graph neural network according to claim 1, characterized in that: Step 3, The training of the reinforcement learning adaptive graph neural network is realized based on the simulator; the road network corresponding to the subgraph structure is mapped into the simulator, and vehicles are generated in the simulator to simulate the vehicle driving process; during the vehicle driving process, the simulator reads the current congestion level of each lane in real time, and the lane congestion level changes in real time with the changes in the traffic flow simulation in the simulator; The training process includes: 3.2.1 Start the simulator, create a vehicle, initialize the current lane information and vehicle information, obtain the corresponding lane congestion information, and embed it into the current time vector ; 3.2.2 At each time step t, obtain the current graph structure and the corresponding road congestion level, and generate the vector corresponding to time t , input reinforcement learning adaptive graph neural network, output the current moment , that is, the road selection probability, the process includes: 3.2.2.1 Obtain the subgraph corresponding to the sub-road network and its corresponding road congestion information, and embed it into a vector ,Will As the feature of all lanes, it is represented as a state in the reinforcement learning algorithm. , s represents the state, t represents the current time step; 3.2.2.2 Generating Actions via Reinforcement Learning Adaptive Graph Neural Networks , that is, the road selection probability of each lane in the subgraph; 3.2.2.3 In the simulator, when the vehicle obtains the current road selection probability, in each subgraph, the path of each vehicle itself is updated, and the road corresponding to the current vehicle driving section is obtained. Choose the next road to travel, thereby modifying its trajectory in real time; 3.2.2.4 Based on reward Iterate and update the parameters of the reinforcement learning adaptive graph neural network; 3.2.3 Repeat step 3.2.2 N times to complete a full set of training; 3.2.4 Repeat steps 3.2.2 and 3.2.3 M times to save the reinforcement learning adaptive graph neural network parameters.

6. The method for load balancing multiple vehicle paths in a large road network based on reinforcement learning graph neural network according to claim 5, characterized in that: The function design of the Reward consists of three parts, namely the first part reward, the second part reward, and the third part reward: The first part, reward, is determined by the congestion level of the lane; the congestion level of the lane is defined as the ratio of the total length of all vehicles to the total length of the lane, denoted as ; The value reflects the congestion level of lane i, i∈E, E is the edge set; The larger the congestion value, the higher the congestion level. The first part of the reward is recorded as The second part, reward, is determined by the BPR value, which is an impedance function, given by Determine; where T is the actual travel time of the current lane, T0 is the free flow speed travel time of the lane, D is the number of vehicles in the current lane, C is the vehicle capacity of the current lane, and a and b are parameters; the larger the value of T, the greater the impedance of the road, and the longer the travel time; by calculating the T-T0 value, measure the value of the second reward function, let is the T-T0 value of road i, and the second part reward is The third part, reward, has a value related to whether the path selected by the vehicle is the shortest path edge; detect whether the next road selected by the vehicle in the current situation is an edge on the shortest path, if so, set its reward value to 1, otherwise to 0; H is the set of vehicles, Indicates vehicle v Whether the currently selected path is the reward value of the shortest path edge: Combining the above three reward parts, the final reward value function is: 。 7. The method for load balancing multiple vehicle paths in a large road network based on reinforcement learning graph neural network according to claim 1, characterized in that: The step 6 is specifically implemented as follows: When the vehicle arrives at an intersection, the road selection probabilities of all roads connected to the current intersection are obtained, and then the next road to be entered is selected; the next road to be entered is selected based on two influencing factors: Influencing factor 1: whether the next road belongs to the shortest path of the corresponding vehicle in the space of the original image; Influencing factor 2: whether the next road is the road with the highest probability of being selected from the edge connected to the current intersection; When there are fewer vehicles, the road with the shortest spatial distance is selected as the road to be driven into according to the influence shadow 1; When there are many vehicles and the road is congested, the road with the highest probability of being selected from the edge connected to the current intersection is selected according to the influence shadow 2, that is, the road with the lowest congestion level is selected as the road to be entered.

Images