Cross-domain vehicle path optimization method based on multi-agent reinforcement learning

Abstract

The application is based on a multi-agent reinforcement learning algorithm, and proposes a cross-domain vehicle path optimization method based on multi-agent reinforcement learning. The application divides the original road network in the traffic system into multiple regions, and models each region as an agent. The agent plans a cross-domain path for the vehicle in the region, and can learn an adaptive and cooperative strategy through a reinforcement learning algorithm, thereby avoiding traffic congestion and achieving lower driving time of the vehicle. The application uses artificial intelligence technology to provide a new solution for the multi-vehicle path finding problem on the road network. Through the reinforcement learning algorithm to train the path selection strategy, an adaptive and cooperative path optimization algorithm can be provided to avoid traffic congestion, improve the throughput of the road network, and reduce the average driving time of the vehicle. The application has wide application prospects and is expected to be applied in the fields of traffic travel, goods transportation, autonomous traffic, etc.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of multi-vehicle path planning optimization. BACKGROUND

[0002] The shortest path finding problem aims to plan the path for multiple vehicles in a road network with the shortest distance or travel time between different origins and destinations. Such path finding problem has important application value in traditional fields such as traffic travel and goods transportation, and emerging fields such as unmanned driving and autonomous transportation. When planning the path for vehicles, if the cooperation between vehicles is not considered, the paths for multiple vehicles will be selected to contain the same road segments, causing traffic congestion and affecting the travel time of individual vehicles. Existing works mainly focus on multi-vehicle path planning in a small range of area, and when considering a large-scale road network and a large number of vehicles, the existing methods have deficiencies in solving performance and efficiency, and poor scalability.

[0003] In actual traffic road network, road segments, intersections and traffic flow will present aggregation phenomenon in space, for example, the road network distribution near some main business districts and residential areas is often more dense, and the traffic flow into / out of these areas is larger.

[0004] The current shortest path finding method can be divided into two categories:

[0005] 1. Directly solving the complete path from the origin to the destination, this kind of method calculates the path with the shortest distance or the shortest travel time when starting from the origin to the destination according to the static road segment length information or the road condition information when the vehicle starts from the origin by using traditional shortest path algorithms such as Dijkstra algorithm and A* algorithm. This kind of method is simple to calculate, but lacks consideration of cooperation between vehicles. The paths selected for multiple vehicles with similar origins and destinations will contain multiple same road segments, causing congestion on these road segments. At the same time, this kind of method cannot adaptively adjust the path of the vehicle according to the change of road conditions during the travel of the vehicle.

[0006] 2. The solving method based on the next hop node updates the path of the vehicle and reselects the next hop node according to the road condition after the vehicle arrives at an intersection. This kind of method trains the path selection strategy for the vehicle or the intersection by using reinforcement learning algorithm, selects the next hop node for the vehicle itself or the vehicle close to the intersection according to the current traffic condition, and can dynamically adjust the path selection result according to the road condition. There is also a method of modeling the interaction between vehicles by using multi-agent reinforcement learning, but this kind of method needs to update the path of the vehicle too frequently, and the number of agents increases significantly with the increase of the number of vehicles and the size of the road network, which has poor solving efficiency and scalability. SUMMARY

[0007] In order to solve the problem of the existing method in the multi-start and end point path finding problem, the multi-agent reinforcement learning technology is applied to the solving of the problem, the original road network is divided into multiple regions through a graph partitioning algorithm, and each region is regarded as an agent. The regional agent can obtain the start point and end point information of the vehicle in the region, and select the next region to be reached by the vehicle and the path to reach the region. Through the multi-agent reinforcement learning technology, the cooperation between the regional agents can be realized, and when selecting the path, the mutual influence between the vehicles is considered, and different paths are selected for the vehicles close to the end point, so that the congestion of part of the road caused by selecting the same path is avoided.

[0008] Technical scheme:

[0009] The multi-agent reinforcement learning based cross-domain vehicle path optimization method divides the original road network in the traffic system into multiple regions, and models each region as an agent, the agent plans a cross-domain path for the vehicle in the region, and learns an adaptive and cooperative strategy through a reinforcement learning algorithm. Cooperation between multiple vehicles is realized, and the deficiency of the existing intra-domain vehicle path planning method in scalability is solved.

[0010] The present application provides a new solution for the multi-vehicle path finding problem in the road network by using artificial intelligence technology, trains the path selection strategy through a reinforcement learning algorithm, can provide an adaptive and cooperative path optimization algorithm, avoids traffic congestion, improves the throughput of the road network, and reduces the average driving time of the vehicle. The present application has wide application prospects, and is expected to be applied in the fields of traffic travel, cargo transportation, autonomous traffic, etc. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 It is a whole structure schematic diagram of the multi-agent reinforcement learning based cross-domain vehicle path optimization algorithm.

[0012] Figure 2 It is a cross-domain path finding decision agent framework schematic diagram. DETAILED DESCRIPTION

[0013] The present application divides the road network into multiple regions, models each region as an agent, and the agent plans a cross-domain path for the vehicle in the region, and can learn an adaptive and cooperative strategy through a reinforcement learning algorithm, so as to avoid traffic congestion and realize lower driving time of the vehicle.

[0014] The algorithm technology of the present application is applied to the vehicle navigation system, which brings a more intelligent method to the field of traffic control, and is expected to realize the cooperation between multiple regions and multiple vehicles, so as to avoid traffic congestion, realize higher throughput and lower average driving time.

[0015] The technical solutions provided by the present application will be further described below with specific embodiments and their accompanying drawings. The advantages and features of the present application will be more apparent in light of the following description.

[0016] The present application divides the original road network into multiple regions by a graph partitioning algorithm, and takes each region as an agent, models the multi-vehicle path finding problem as a partially observable Markov process, and learns the path finding strategy of each region agent through a reinforcement learning algorithm. Experiments are conducted on the widely used traffic simulator SUMO, and then applied to actual vehicle navigation applications.

[0017] The following detailed description of each design part and application deployment.

[0018] I. Road network partitioning and multi-agent reinforcement learning definition

[0019] The present application first divides the road network into M regions by a graph partitioning algorithm, and models each region as an agent. The agent selects the cut edge for the vehicle in the region to leave the region, as well as the path within the region.

[0020] Taking the path finding from the starting point to the end point as an example (represented by each vehicle node), as shown in Figure 1 (a): when the path finding vehicle starts from the starting point of region R1, the agent of region R1 plans the inter-domain path R1→R2 through inter-domain multi-agent communication, selects the cut edge v1→v4 to enter the adjacent region R2, and plans the intra-domain path from to v1. The agent corresponding to region R2 plans the inter-domain path R2→R3, selects the cut edge v7→v 10 to enter region R3, and plans the intra-domain path v4→v7 within region R2. The agent of region R3 finally plans the intra-domain path from v 10 →v d to the end point , and obtains the complete path from the starting point to the end point .

[0021] Road network partitioning:

[0022] The present application applies the classic graph partitioning algorithm (METIS algorithm) to divide the road network into M regions with similar sizes and as few cut edges as possible.

[0023] Multi-agent reinforcement learning definition:

[0024] Based on the results of region partitioning (forming M agents), the present application defines the multi-vehicle path finding problem as a partially observable Markov decision process, which is defined as follows.

[0025] • Agent: model each region as an individual agent, all agents form a set

[0026] When a vehicle l leaves a region or enters a new region, vehicle l sends the trip information to the relevant region agent R i , agent R i plans the path for vehicle l (including inter-regional path and intra-regional path) and sends the path calculation result to vehicle l.

[0027] • Action: when vehicle l enters the road network or reaches the cut edge of the entering region, the region agent needs to take action,

[0028] For example, in (a), when the vehicle enters the green region, the region agent R1 needs to take action Figure 1 for the vehicle's path request For each vehicle query request, region agent R i selects an edge from the set of cut edges

[0029] of the region as the inter-regional path selection, which determines the next region that the vehicle will enter and the edge through which the vehicle will enter the next region. Assume that at the t-th time step, agent R i receives a set of path query requests For each query request agent R i takes action

[0030] • Observation: the observation information of each region agent R i can be divided into two parts, the observation of the road network environment

[0031] and the observation of vehicle query requests

[0032] First, the observation of the road network environment includes intra-regional road segment state information and estimation of road state information of neighboring regions, including average driving speed and total number of vehicles in neighboring regions. The road network environment, the information of the road network is represented as where F is the input feature dimension, the features of each cut edge include the coordinates of the start and end points of the edge, the length of the edge, the average speed and total number of vehicles of the edge, and the average driving speed and total number of vehicles of the region connected by the edge.​

[0033] Secondly, the characteristics of each vehicle route request are represented as follows: Where F' represents the vehicle route request input features, including the coordinates of the vehicle's current location and destination, and the vehicle's arrival time. Time estimation for strip cutting and The distance from the cut edge to the vehicle's destination is a feature defined by the regional agent R. i It is calculated based on the origin and destination information sent by the vehicle. At each time step, R... i receive Path query request, agent R i The observation of the k-th query request at time t is represented as follows: R will act as an intelligent agent i The policy network π i ( Figure 2 )enter.

[0034] • Global state: The global state s at time t t This state is represented by the connection of features of all cut edges. This state representation contains the observation information of M agents. At the same time, the features of the cut edges contain the road condition information of these key road segments and the average driving speed information of each area.

[0035] • Policy: Given an agent R i Observations Agent R i The policy network π i Output probability distribution

[0036] This probability distribution represents the choice of each action. Based on the probability distribution, the intelligent system will select cross-domain actions for the vehicle. A cut edge in the middle, agent R i The policy network π i

[0037] Structure as Figure 2 As shown.

[0038] • Reward: Agent R i Regarding path requests Local reward is defined as Where, t′

[0039] For vehicles to enter the action section The time step.

[0040] For example, in Figure 1 In (a), when the vehicle arrives at node v1, the regional agent R2 selects road segment v7→v 10This is the action segment for the vehicle. The reward for this action is the inverse of the travel time from node v1 to node v7. When the vehicle takes a longer time to reach the action segment, the agent receives a smaller reward. This reward design is consistent with minimizing the total travel time for the vehicle to complete the journey.

[0041] II: Intelligent Agent Framework Design Figure 2 (As shown)

[0042] To enable cooperation among vehicles and avoid traffic congestion caused by routes planned for multiple vehicles containing the same road segments, this invention designs an intelligent agent framework, which includes: road network feature extraction, vehicle route request feature extraction, and vehicle action output.

[0043] That is, agent R i The policy network π i ,include:

[0044] (1) Road network feature extraction

[0045] The road network feature extraction process aims to extract the features of the traffic road network. Based on the road network division results, this invention will connect the regional intelligent agent R. i The set of cut edges is represented as The characteristics of each cut edge are: It includes the location information of the road segment corresponding to the cut edge, traffic congestion information, and traffic condition information of the connecting area of ​​the road segment. Then, this invention encodes the input features of each cut edge through a multilayer perceptron network to obtain the latent vector representation. By The latent vector representation of the cut edges allows us to obtain the region agent R. i Feature extraction for road networks

[0046] (2) Vehicle route request feature extraction

[0047] When planning routes for vehicles, the origin and destination information of the vehicles also needs to be considered. At time t, agent R... i The received set of vehicle route requests is For agent R at time t i The route request received from the kth vehicle Features It includes the vehicle's current location and the target location information. This invention encodes this information into a vector using a multilayer perceptron network. To consider the same time region R i The present invention will address the mutual influence between multiple vehicles. Each vehicle route request feature is input into a gated recurrent unit (GRU) for feature extraction, i.e. The last hidden vector of the GRU unit is taken as the vector representation of the multiple vehicle query requests, that is

[0048] (3) Vehicle action output

[0049] Through the road network encoding process and the vehicle path request encoding process, three vector representations can be obtained: the road network vector representation The path request encoding of the current vehicle And the path request encoding of multiple vehicles Taking the three vector representations as inputs, the agent generates a probability distribution of a selected cut edge And selects the cut edge according to the probability.

[0050] Specifically, the output process first uses a multi-layer perception to map the vector representation to The scores of the edges, uses an action mask operation to adjust the scores of infeasible actions (e.g., back to the last region) to -∞, and then converts the edge scores to the probability of selecting The edge as a cross-domain action, that is

[0051]

[0052] According to the vector representations And The agent can identify vehicle path requests with similar locations. If the number of such requests is small, the agent tends to assign the highest probability (close to 1.0) to the cut edge that can provide the shortest travel time for the request. Otherwise, the agent will also assign high probabilities to other cut edges. Thus, it can avoid planning too many vehicles on the same road segment and achieve cooperation between vehicles. The present application realizes the mapping from road network features, vehicle path request features to cut edge scores through a multi-layer perception (MLP), and the higher the score, the higher the probability of the corresponding cut edge. Through the mask mechanism, the output probability of an infeasible action is 0.

[0053] III. State value estimation Figure 1 and training algorithm

[0054] The present application uses an Actor-Critic framework to train regional agents for multi-start and multi-end path planning problems. Since the time of arrival of vehicles at each region is different, at time step t, some agents do not need to make actions. In order to solve the non-stationary problem of multi-agent reinforcement learning in the asynchronous decision-making scenario, the present application defines the global state as the observation information of all cut edges and proposes an asynchronous trajectory collection mechanism. Whenever an agent R iWhen an action is performed, the state transition record of that action is uploaded to the central server for training the policy network and value network. The agent R... i A state transition is Represents agent R i Based on the observation information at time t Make an action The global state at this time is Then at time t′, the vehicle completes its action. And receive a reward Global state transition to

[0055] To learn the path selection strategy of regional agents, this invention uses the Multi-Agent PPO algorithm (MAPPO) for training. Figure 1 (c)

[0056] For each state transition The advantage function of an action can be calculated. and discount value Where γ is the discount parameter and V is the state value function.

[0057] Then, by minimizing the loss function L(θ) i Optimize the policy network of actors (π) i The parameter θ i Loss function L(θ) i The definition is as follows:

[0058]

[0059] in, To represent the calculation of expectation, θ i For the parameters of the policy network, For the old strategy used to collect trajectories, the clip function will... Clipping to the interval [1-∈, 1+∈] avoids overly large updates for the strategy. The value range of the clip parameter ∈ is (0, 1).

[0060] The critic network parameters φ are optimized by minimizing the squared error. The loss function L(φ) is defined as follows:

[0061]

[0062] in, This indicates the calculation of the expectation, where φ is the parameter of the critic network.

[0063] IV. Experimental Verification

[0064] The road network containing 100 nodes is synthesized, the road network is divided into 4 regions, each region contains 25 intersections and 84 road segments, and 4 cut edges are connected. Assuming that the maximum driving speed of vehicles on each road segment is 13.89 m / s, randomly generate traffic flow between two non-adjacent regions, and simulate by SUMO.

[0065] 4.1 Comparison method

[0066] The throughput (total number of vehicles reaching the destination), average travel time (seconds) and average carbon dioxide emissions (kilograms) of each vehicle when leaving the road network of the proposed method are counted in 100 episodes (each episode is simulated for 600 seconds).

[0067] In order to verify the effectiveness of the proposed method, four existing comparison methods are selected as follows:

[0068] Random: the regional agent randomly selects a cut edge connected as the path selection action between regions,

[0069] The path selection method in the region is the same as the proposed method asyn-MARL.

[0070] SP: select the shortest path through the cut edge as the action between regions.

[0071] SPFR: assuming that the traffic speed information of the road segment in the region and the rough speed information of the connected region can be observed when selecting the path for the vehicle, the vehicle selects the next intersection according to the accurate speed information or estimated speed information of the road segment every time it reaches the traffic intersection.

[0072] AN: an existing multi-agent reinforcement learning algorithm, which trains a DQN network to select the next intersection for vehicles near the intersection.

[0073] 4.2 Experimental results

[0074] The above methods are used for experiments respectively, and the throughput, average travel time and average carbon dioxide emissions are counted. The effect comparison is as follows:

[0075] Method Throughput Average travel time (seconds) Average carbon dioxide emissions (kg) Random 320.30 313.503 0.807 SP 738.02 158.837 0.416 SPFR 784.34 150.864 0.384 AN 764.15 168.967 0.444 asyn-MARL 936.94 125.551 0.352

[0076] The experimental results show that the method proposed in the present application is superior to the existing shortest path method and multi-agent reinforcement learning method in multiple indicators.

[0077] Five, application deployment

[0078] The traffic condition information of all road segments in the region can be obtained by the regional server, and the traffic condition information of the neighboring region can also be obtained by accessing the cloud server, including the number of vehicles and the average speed in the neighboring region. When a vehicle departs from or arrives at a region, the vehicle's current position information and target position are sent to the server of the region. The strategy network of the agent deployed by the regional server (the network structure is shown in Figure 2 The traffic condition information of all road segments in the region can be obtained by the regional server, and the traffic condition information of the neighboring region can also be obtained by accessing the cloud server, including the number of vehicles and the average speed in the neighboring region. When a vehicle departs from or arrives at a region, the vehicle's current position information and target position are sent to the server of the region. The strategy network of the agent deployed by the regional server (the network structure is shown in

[0079] During the training process, when a vehicle completes the action of the regional agent and is about to leave the region, the reward information of the action, i.e., the negative value of the time required to complete the action, is sent to the corresponding regional server. Each regional server uploads the road condition observation information, all vehicle path request information, vehicle path planning results, and action reward information in the region to the central server. The central server estimates the state value according to the information of all agent and environment interactions using the centralized training method (as shown in Figure 1 The central server estimates the state value according to the global state information and the observation information of the regional agent for each vehicle path request, and uses it to update the strategy network of the agent. The updated regional agent strategy network parameters are returned to each regional server for estimating the state value. The centralized critic network used for estimating the state value is updated by the reward information obtained. This network structure is only used in the training phase. In the execution process, the regional agent only needs to obtain the observation information of the regional traffic condition and the path request information of the vehicle, and exchange a small amount of data with the central server, i.e., upload the traffic condition information of the region and obtain the traffic condition information of the neighboring region.

[0080] The above description is only a description of the preferred embodiments of the present application, and is not any limitation on the scope of the present application. Any modification or modification made by any person skilled in the art according to the technical content disclosed above shall be regarded as an equivalent effective embodiment, and shall fall within the scope of the technical solution protected by the present application.

[0081] References:

[0082] 1. F. Arasteh, S. SheikhGarGar, and M. Papagelis. Network-aware multiagent

[0083] reinforcement learning for the vehicle navigation problem. In Proceedings of the 30th

[0084] International Conference on Advances in Geographic Information Systems. Association for Computing Machinery. (2022)

[0085] 2. P. A. Lopez, M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Fl¨otter¨od, R. Hilbrich, L. L¨ucken, J. Rummel, P. Wagner, and E. Wiessner. Microscopic traffic simulation

[0086] using sumo. In International Conference on Intelligent Transportation Systems, pages

[0087] 2575–2582. (2018)

[0088] 3. G. Karypis. Metis: Unstructured graph partitioning and sparse matrix ordering system.

[0089] Technical report. (1997)

[0090] 4. J. Yin, W. Rao, Q. Zhao, C. Zhang, and P. Hui. Learn to optimize the constrained

[0091] shortest path on large dynamic graphs. IEEE Trans. Mob. Comput., 23(3):2456–2469, (2024)

[0093] 5. C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. M. Bayen, and Y. Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Advances in Neural Information Processing Systems 35. (2022)

[0094] Information Processing Systems 35. (2022)

[0095] 6. K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 1724-1734. (2014)

[0096] 7. Z. Wang and S. Wang. Xrouting: Explainable vehicle rerouting for urban road congestion avoidance using deep reinforcement learning. In 2022 IEEE International Smart Cities Conference, pages 1-7. (2022)

[0097] 8. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. (2017)

Claims

1. A cross-domain vehicle path optimization method based on multi-agent reinforcement learning, characterized in that, The original road network in the traffic system is divided into multiple regions, and each region is modeled as an agent, which plans the cross-region path for vehicles in the region, and learns an adaptive and cooperative strategy through reinforcement learning algorithm; The first division part: divide the road network into M regions, model each region as an agent, and the agent selects the cut edge for the vehicle to leave the region and the path within the region; The second definition part: The multi-vehicle path finding problem is defined as an observable Markov decision process, and the multi-agent reinforcement learning is defined as follows: Agent: model each area as a separate agent, and all the agents form a set , ; when a vehicle leaves one area or enters a new area, the vehicle will send the trip information to the relevant area agent , the agent will plan a path for the vehicle and send the path calculation result to the vehicle , the path includes the path between areas and the path inside the area; Action: When a vehicle enters a road network or reaches a cut-edge of an entering area, the area agent needs to make an action, for each vehicle query request, the area agent selects an edge from the cut-edge set of the area as the inter-area path selection, this selection determines the next area that the vehicle will enter and the edge through which the vehicle enters the next area; suppose that at the t-th time step, the agent receives a set of path query requests , for each query request , the agent makes an action ; Observation: each regional agent Observation information is divided into two parts, the observation of road network environment and the observation of vehicle query request ; first, the observation of road network environment includes the road segment state information within the region and the estimation of road state information of neighbor regions, which includes the average driving speed and the total number of vehicles of neighbor regions; the road network environment represents the information of road network as , where F is the input feature dimension, the feature of each cut edge contains the coordinates of the start and end points of the edge, the length of the edge, the average vehicle speed and the total number of vehicles of the edge, and the average driving speed and the total number of vehicles of the edge connecting region; second, the feature of each vehicle path request is represented as , where F' is the input feature of vehicle path request, including the coordinates of the current position and the end point of the vehicle, the time estimate of the vehicle to reach cut edges, and the distance of cut edges to reach the end point of the vehicle, which is obtained by the regional agent according to the start and end point information sent by the vehicle; at each time step, receiving path query requests, the agent observation of the th query request at time t is represented as , which will be input into the policy network of the agent ; Global state: will be Global state at time t The state is represented as the concatenation of the features of all cut edges, which contains the observation information of M agents. Meanwhile, the features of cut edges contain the road condition information of these key links and the average travel speed information of each region. Policy: given an observation of the environment , the policy network of the agent outputs a probability distribution over actions , where the probability distribution represents the probability of selecting each action , and the agent selects a cross-domain action for the vehicle, i.e., a cut edge , according to the probability distribution. Rewards: Agents Regarding path requests The local reward is defined as where, is the time step at which the vehicle enters the action segment The reward design and minimizing the total travel time of the vehicle's completed trip are consistent; The third application part: Deploy a regional server in each region to plan paths for vehicles within the region and communicate with vehicles within the region, and deploy a central server to train algorithms and communicate with regional servers; Intelligent agent Policy network of the intelligent agent including road network feature extraction, vehicle path request feature extraction, and vehicle action output, wherein; Road network feature extraction: The road network feature extraction process aims to extract the features of the traffic road network and, based on the road network division results, connect regional intelligent agents. The set of cut edges is represented as The characteristics of each cut edge are: It includes the location information of the road segment corresponding to the cut edge, traffic congestion information, and traffic condition information of the connecting area of ​​the road segment; the input features of each cut edge are encoded through a multilayer perceptron network to obtain the latent vector representation. By The latent vector representation of each cut edge is used to obtain the region agent. Feature extraction for road networks ; Vehicle path request feature extraction: At time t, the intelligent agent The received set of vehicle route requests is For the agent at time t The first received Route requests for individual vehicles , Features It contains information about the vehicle's current location and the target location; this information is encoded into a vector using a multilayer perceptron network. ;Will Each vehicle route request feature is input into a gated recurrent unit (GRU) for feature extraction, i.e. The last hidden vector of the GRU unit is used as the vector representation of multiple vehicle query requests, that is... ; Vehicle action output: Three vector representations are obtained through the road network encoding process and the vehicle path request encoding process: a road network vector representation , a path request encoding of the current vehicle , and path request encodings of a plurality of vehicles ; taking the three vector representations as inputs, the intelligent agent generates a probability distribution of a selected cut edge , and selects the cut edge according to the probability; First, a multilayer perceptron is used to map the vector representation to... The score for the cut edge is calculated using an action mask operation, adjusting the score of infeasible actions to a lower value. Then, the edge scores are converted into selection scores using the softmax function. The probability of a cut edge as a cross-domain action, i.e. According to the vector representation And The agent identifies vehicle path requests with similar locations.

2. The multi-agent reinforcement learning-based cross-region vehicle path optimization method of claim 1, wherein, intelligent agent A state transition is Represents intelligent agents exist Based on observation information Make an action The global state at this time is , and then At that moment, the vehicle completed the maneuver. and receive a reward The global state transition is .

3. The multi-agent reinforcement learning-based cross-region vehicle path optimization method of claim 1, wherein, For each state transition , the advantage function and discounted value of the action are computed, where is a discount parameter, is a state value function; Then, the parameters of the policy network of the actor are optimized by minimizing the loss function L(θa) The parameters of the policy network of the actor are optimized by minimizing the loss function L(θa) defined as follows: wherein, denotes the computation of the expectation, are parameters of the policy network, is the old policy used to collect the trajectories, the clip function will clip to the interval to avoid too large updates to the policy, the clip parameter is in the range ; Optimizing critic network parameters by minimizing squared error , loss function is defined as: wherein, represents the computation of the expectation, is the parameter of the critic network.

4. The multi-agent reinforcement learning-based cross-region vehicle path optimization method of claim 1, wherein, The application part is specifically: The regional server obtains traffic condition information of all road segments in the region, and obtains traffic profile information of neighboring regions by accessing the cloud server, including the number of vehicles and the average speed in the neighboring region; When a vehicle departs from or arrives at a region, it sends the current position information and target position of the vehicle to the server of the region, and the strategy network of the agent deployed by the regional server takes the path request of the vehicle and the condition information of the road network as input, and outputs the cross-region path planning of the vehicle, i.e. the next region and the cut edge to the next region, according to the cross-region path planning result, the regional server plans the intra-regional path to the target cut edge according to the starting point and endpoint of the vehicle according to the current traffic condition, and finally sends the calculated path to the vehicle, i.e. the sequence of road segments to the next region; when the vehicle reaches the next region along the path, it sends a path request to the server of the new region, which plans the next stage path for it. In the training process, when the vehicle completes the action of the regional agent and is about to leave the region, the reward information of the action, i.e. the negative value of the time required to complete the action, is sent to the corresponding regional server. Each regional server uploads the road condition observation information within the region, all vehicle path request information, vehicle path planning results and action reward information to the center server. The center server trains the cross-domain path planning strategy of the agent using the centralized training method according to the information of all agent and environment interactions. The center server estimates the state value according to the global state information and the observation information of the regional agent for each vehicle path request, and uses it to update the agent's policy network. The updated regional agent policy network parameters are returned to each regional server for estimating the state value. The centralized critic network used for estimating the state value is updated by the reward information obtained. This network structure is only used in the training stage. In the execution process, the regional agent only needs to obtain the observation information of the regional traffic condition and the path request information of the vehicle, and exchange data through the center server, i.e. upload the traffic condition information of the region and obtain the traffic condition information of the neighboring region.

Images