Cross-domain vehicle path optimization method based on multi-agent reinforcement learning

Abstract

The invention provides a cross-domain vehicle path optimization method based on multi-agent reinforcement learning based on a multi-agent reinforcement learning algorithm. According to the method, the original road network in the traffic system is divided into a plurality of areas, each area is modeled as an intelligent agent, the intelligent agents plan cross-domain paths for vehicles in the areas, and adaptive and cooperative strategies can be learned through a reinforcement learning algorithm, so that traffic congestion is avoided, and lower driving time of the vehicles is realized. A new solution is provided for the multi-vehicle routing problem on the road network by utilizing the artificial intelligence technology, a path selection strategy is trained through a reinforcement learning algorithm, a self-adaptive and cooperative path optimization algorithm can be provided, traffic congestion is avoided, the throughput rate of the road network is improved, and the average driving time of the vehicles is shortened. The method is wide in technical application prospect and is expected to be applied to the fields of transportation, cargo transportation, autonomous traffic and the like.

Description

Technical Field

[0001] The invention relates to the field of multi-vehicle path planning optimization. Background Art

[0002] The shortest path routing problem aims to plan the shortest path between different starting points and end points of multiple vehicles in a road network. This type of routing problem has important application value in traditional fields such as transportation and freight transportation, as well as emerging fields such as unmanned driving and autonomous transportation. When planning paths for vehicles, if the cooperation between vehicles is not considered, multiple vehicles will choose paths that include the same road section, causing traffic congestion, which in turn affects the travel time of individual vehicles. Existing work mainly focuses on multi-vehicle path planning in a small area. When considering the actual large-scale road network and a large number of vehicles, the existing methods have deficiencies in solution performance and efficiency, and poor scalability.

[0003] In actual traffic networks, road sections, intersections, and traffic flows tend to cluster in space. For example, the road network distribution near some major commercial and residential areas is often more dense, and the traffic volume flowing into / out of these areas is larger.

[0004] At present, the shortest path finding methods can be divided into two categories:

[0005] 1. Directly solve the complete path from the starting point to the end point. This type of method uses traditional shortest path algorithms such as Dijkstra algorithm and A* algorithm to calculate the path with the shortest distance from the starting point to the end point or the shortest driving time at the time of departure based on static road section length information or road condition information when the vehicle departs from the starting point. This type of method is simple to calculate, but lacks consideration of cooperation between vehicles. The paths selected for multiple vehicles with similar starting points and end points will contain multiple identical road sections, causing congestion on these sections. At the same time, this type of method cannot make adaptive adjustments to the vehicle's path according to changes in road conditions during the vehicle's driving process.

[0006] 2. The next-hop node-based solution method updates the path for the vehicle and reselects the next-hop node based on the road conditions every time the vehicle arrives at an intersection. This type of method uses a reinforcement learning algorithm to train a path selection strategy for the vehicle or intersection, and selects the next-hop node for the vehicle itself or the vehicle near the intersection based on the current traffic conditions. It can dynamically adjust the path selection results according to the road conditions. There are also multi-agent reinforcement learning methods to model the interaction between vehicles, but this type of method requires too frequent updates of vehicle paths, and the number of agents increases significantly with the number of vehicles and the scale of the road network, resulting in poor solution efficiency and scalability. Summary of the invention

[0007] In order to solve the problems existing in the existing methods in solving the path-finding problem with multiple starting points and ending points, the present invention applies multi-agent reinforcement learning technology to solve this problem. The original road network is divided into multiple regions through a graph partitioning algorithm, and each region is used as an agent. The regional agent can obtain the starting point and ending point information of the vehicle in the region, and select the next area to be reached and the path to reach the area for the vehicle. Through multi-agent reinforcement learning technology, cooperation between regional agents can be achieved, and when selecting paths, different paths will be selected for vehicles with close endings, taking into account the mutual influence between vehicles, thereby avoiding congestion of some sections of the road caused by all choosing the same path.

[0008] Technical solution:

[0009] The cross-domain vehicle path optimization method based on multi-agent reinforcement learning divides the original road network in the traffic system into multiple regions, and models each region as an agent. The agent plans cross-domain paths for vehicles in the region and learns adaptive and cooperative strategies through reinforcement learning algorithms. It realizes cooperation among multiple vehicles and solves the shortcomings of existing intra-domain vehicle path planning methods in terms of scalability.

[0010] The present invention uses artificial intelligence technology to provide a new solution to the multi-vehicle path-finding problem on the road network. By training the path selection strategy through the reinforcement learning algorithm, it can provide an adaptive and cooperative path optimization algorithm to avoid traffic congestion, improve the throughput rate of the road network, and reduce the average driving time of vehicles. The technology of the present invention has broad application prospects and is expected to be applied in the fields of transportation, cargo transportation, autonomous transportation, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Schematic diagram of the overall architecture of the cross-domain vehicle path optimization algorithm based on multi-agent reinforcement learning.

[0012] Figure 2 Schematic diagram of the cross-domain path-finding decision-making agent framework. DETAILED DESCRIPTION

[0013] The present invention divides the road network into multiple areas and models each area as an intelligent agent. The intelligent agent plans cross-domain paths for vehicles in the area and can learn adaptive and cooperative strategies through reinforcement learning algorithms, thereby avoiding traffic congestion and achieving lower vehicle travel time.

[0014] Applying the algorithm technology of the present invention to the vehicle navigation system brings a more intelligent method to the field of traffic control, which is expected to achieve collaboration between multiple areas and multiple vehicles, thereby avoiding traffic congestion and achieving higher throughput and lower average travel time.

[0015] The technical solution provided by the present application will be further described below in conjunction with specific embodiments and accompanying drawings. The advantages and features of the present application will become more apparent with the following description.

[0016] The present invention divides the original road network into multiple regions through a graph partitioning algorithm, and uses each region as an intelligent agent. The multi-vehicle routing problem is modeled as a partially observable Markov process, and the routing strategy of each regional intelligent agent is learned through a reinforcement learning algorithm. Experiments are conducted on the widely used traffic simulator SUMO, and then applied to actual vehicle navigation applications.

[0017] The following is a detailed description of each design part and application deployment.

[0018] 1. Road Network Division and Multi-Agent Reinforcement Learning Definition

[0019] The present invention first divides the road network into M areas using a graph partitioning algorithm, and models each area as an intelligent agent, which selects the cut edges leaving the area and the paths within the area for vehicles in the area.

[0020] Starting point To the end Take the path finding of as an example (represented by each vehicle node), such as Figure 1 As shown in (a): The path-seeking vehicle starts from the starting point of area R1 At the start, the agent in region R1 plans and decides the inter-region path R1→R2 through inter-region multi-agent communication, enters the adjacent region R2 by selecting the cut edge v1→v4, and plans to go from The agent corresponding to region R2 plans to find the path between regions R2→R3 and selects the cutting edge v7→v 10 Enter region R3 and plan the path from v4 to v7 in region R2. The agent in region R3 finally plans to go from v 10 →v d Path within the domain, reaching the end point Get the starting point To the end The full path.

[0021] Road network division:

[0022] The present invention applies a classic graph partitioning algorithm (METIS algorithm) to divide the road network into M areas of similar size and with the least number of cut edges possible.

[0023] Multi-agent reinforcement learning definition:

[0024] Based on the result of area division (forming M intelligent agents), the present invention defines the multi-vehicle routing problem as a partially observable Markov decision process, which is specifically defined as follows.

[0025] Agent: Each region is modeled as a separate agent, and all agents form a collection

[0026] When vehicle l leaves from one area or enters a new area, vehicle l sends the trip information to the relevant regional agent R i , Agent R i A path will be planned for vehicle l (including paths between areas and paths within areas) and the path calculation results will be sent to vehicle l.

[0027] Action: When vehicle l enters the road network or reaches the cut edge of the entry area, the agent in the area needs to take action.

[0028] For example, Figure 1 In (a), when the vehicle enters the green area, the regional agent R1 needs to take action for the vehicle’s path request. For each vehicle query request, the regional agent R i From the cut edge collection of the region

[0029] Select an edge as the path selection between regions. This choice determines the next region that the vehicle is about to enter and the edge through which it enters the next region. Suppose at the tth time step, the agent R i The received path query request set is For each query request Agent R i Make a move

[0030] Observation: Each regional agent R i Observation information It can be divided into two parts: observation of the road network environment

[0031] and observations for vehicle query requests

[0032] First, observation of the road network environment It includes the road status information within the area and the estimation of the road status information of the neighboring area, and the road status information of the neighboring area includes the average driving speed and the total number of vehicles in the neighboring area. Where F is the input feature dimension, and the features of each cut edge include the coordinates of the start and end points of the edge, the length of the edge, the average speed and total number of vehicles on the edge, and the average speed and total number of vehicles in the area connected by the edge.

[0033] Secondly, each vehicle route request feature is expressed as Where F' is the input feature of the vehicle path request, including the coordinates of the vehicle's current position and destination, the vehicle's arrival The time estimation of the cutting edge and The distance from the cut edge to the vehicle’s destination is characterized by the regional agent R i It is calculated based on the start and end point information sent by the vehicle. At each time step, R i receive path query request, agent R i The observation of the k-th query request at time t is expressed as As the intelligent agent R i The policy network π i ( Figure 2 )enter.

[0034] Global state: The global state s at time t t It is represented as the connection of the features of all the cut edges. This state representation contains the observation information of M agents. At the same time, the features of the cut edges contain the road condition information of these key sections and the average driving speed information of each area.

[0035] Strategy: Given an agent R i Observation Agent R i The policy network π i Output probability distribution

[0036] This probability distribution represents the choice of each action The probability of the intelligent experience will select cross-domain actions for the vehicle according to this probability distribution, that is, A cut edge in the agent R i The policy network π i

[0037] Structure Figure 2 shown.

[0038] Reward: Agent R i About route requests The local reward is defined as Among them, t′

[0039] For vehicles entering the action section time step.

[0040] For example, in Figure 1 In (a), when the vehicle arrives at node v1, the regional agent R2 selects the road segment v7→v 10As the action segment of the vehicle. The reward for this action is the inverse of the time it takes the vehicle to reach node v7 from node v1. When the vehicle takes a longer time to reach the action segment, the agent will receive a smaller reward. This reward design is consistent with minimizing the total travel time of the vehicle to complete the journey.

[0041] 2. Agent Framework Design ( Figure 2 (shown)

[0042] In order to achieve cooperation between vehicles and avoid traffic congestion caused by the paths planned for multiple vehicles containing the same road section, the present invention designs an intelligent agent framework, which includes: road network feature extraction, vehicle path request feature extraction and vehicle action output.

[0043] That is, agent R i The policy network π i ,include:

[0044] (1) Road network feature extraction

[0045] The road network feature extraction process aims to extract the characteristics of the traffic road network. According to the results of the road network division, the present invention connects the regional intelligent body R i The set of cut edges is expressed as The characteristics of each cutting edge are It contains the location information of the road section corresponding to the cut edge, traffic congestion information and traffic condition information of the road section connection area. Then, the present invention encodes the input features of each cut edge through a multi-layer perceptron network to obtain a latent vector representation. By The hidden vector of the cut edge represents the connection, and the regional agent R i Feature extraction for road networks

[0046] (2) Vehicle route request feature extraction

[0047] When planning a path for a vehicle, it is also necessary to consider the vehicle’s starting and ending points. i The received vehicle route request set is For agent R at time t i The route request received for the kth vehicle Features Contains the current position and target position information of the vehicle. The present invention encodes it into a vector through a multi-layer perceptron network In order to consider the region R at the same time i The mutual influence between multiple vehicles, the present invention will The vehicle path request features are input into the gated recurrent unit (GRU) for feature extraction, namely The last latent vector of the GRU unit is used as the vector representation of multiple vehicle query requests, that is,

[0048] (3) Vehicle motion output

[0049] Through the road network encoding process and the vehicle path request encoding process, three vector representations can be obtained: road network vector representation The route request code of the current vehicle and multiple vehicles route request encoding Taking these three vector representations as input, the agent will generate a selection cut edge The probability distribution of , and select the cutting edge according to the probability.

[0050] Specifically, the output process first uses a multi-layer perceptron to map the vector representation into The score of the cut edge is adjusted to -∞ using the action mask operation, and then the edge score is converted to a selection through the softmax function. The probability of cutting edges as cross-domain actions is

[0051]

[0052] According to the vector representation and The agent can identify vehicle path requests with similar locations. If the number of such requests is small, the agent tends to assign the highest probability (close to 1.0) to the cut edge that can provide the shortest travel time for the request. Otherwise, the agent will also assign higher probabilities to other cut edges. This can avoid planning too many vehicles on the same road section and achieve collaboration between vehicles. The present invention uses a multi-layer perceptron (MLP) to implement nonlinear transformation to achieve the mapping from road network features, vehicle path request features to cut edge scores. The higher the score, the higher the probability corresponding to the cut edge. Through the mask mechanism, the output probability of an inoperable action is 0.

[0053] 3. State Value Estimation Figure 1 (c)) and training algorithm

[0054] This paper uses the Actor-Critic framework to train regional agents. For the path planning problem with multiple starting points and multiple destinations, since the time when vehicles arrive at each area is different, at time step t, some agents do not need to take action. In order to solve the non-stationarity problem of multi-agent reinforcement learning in asynchronous decision-making scenarios, this paper defines the global state as the observation information of all cut edges and proposes an asynchronous trajectory collection mechanism. Whenever the agent R iWhen an action is taken, the state transition record of this action will be uploaded to the central server for training the policy network and value network. i A state transition of Represents the agent R i At time t, according to the observed information Make a move The global status at this time is Then at time t′, the vehicle completes the action And get rewards The global state transition is

[0055] In order to learn the path selection strategy of the regional agent, the present invention uses the multi-agent PPO algorithm (MAPPO) for training ( Figure 1 (c)).

[0056] For each state transition The advantage function of an action can be calculated and discount value Among them, γ is the discount parameter and V is the state value function.

[0057] Then, by minimizing the loss function L(θ i ) Optimize the Actor’s policy network π i The parameter θ i , the loss function L(θ i ) is defined as follows:

[0058]

[0059] in, represents the expected value, θ i are the parameters of the policy network, For the old strategy used to collect trajectories, the clip function will Clip to the interval [1-∈, 1+∈] to avoid excessive updates to the strategy. The value range of the clip parameter ∈ is (0, 1).

[0060] The critic network parameter φ is optimized by minimizing the squared error. The loss function L(φ) is defined as:

[0061]

[0062] in, represents the computational expectation, and φ is the parameter of the critic network.

[0063] IV. Experimental Verification

[0064] The present invention synthesizes a road network containing 100 nodes, which is divided into 4 regions, each of which contains 25 intersections and 84 road sections, connected by 4 cut edges. Assuming that the maximum speed of vehicles on each road section is 13.89 m / s, the traffic flow between two non-adjacent regions is randomly generated and simulated by SUMO.

[0065] 4.1 Comparison Method

[0066] The present invention statistically calculates the throughput (the total number of vehicles arriving at the destination), the average travel time (seconds), and the average carbon dioxide emissions (kg) of each vehicle when leaving the road network in 100 episodes (the simulation duration of each episode is 600 seconds).

[0067] In order to verify the effectiveness of the proposed method, four existing comparison methods are selected as follows:

[0068] Random: The regional agent randomly selects a connected cut edge as the path selection action between regions.

[0069] The path selection method inside the region is the same as the proposed method asyn-MARL.

[0070] SP: Select the cut edge that the path with the shortest length passes through as the action between regions.

[0071] SPFR: Assuming that the traffic speed information of the internal road segments and the rough speed information of the connecting areas can be observed when selecting a route for the vehicle, each time the vehicle arrives at a traffic intersection, it selects the next intersection based on the accurate speed information of the road segment or the estimated speed information.

[0072] AN: An existing multi-agent reinforcement learning algorithm that trains a DQN network to select the next intersection for vehicles approaching an intersection.

[0073] 4.2 Experimental Results

[0074] The above methods were used to conduct experiments and the throughput, average driving time and average carbon dioxide emissions were calculated. The results are compared as follows:

[0075] method Throughput Average driving time (seconds) Average CO2 emissions (kg) Random 320.30 313.503 0.807 SP 738.02 158.837 0.416 SPFR 784.34 150.864 0.384 AN 764.15 168.967 0.444 asyn-MARL 936.94 125.551 0.352

[0076] Experimental results show that the method proposed in this invention is superior to the existing shortest path method and multi-agent reinforcement learning method in multiple indicators.

[0077] 5. Application Deployment

[0078] The graph is divided into multiple regions through the graph partitioning algorithm (METIS algorithm), and servers are deployed in each region to plan routes for vehicles in the region. The regional server can obtain traffic condition information of all road sections in the region, and can also obtain traffic profile information of neighboring regions by accessing the cloud server, including the number of vehicles and average speed in neighboring regions. When a vehicle leaves or arrives at a region, it will send the current location information and target location of the vehicle to the server in the region. The strategy network of the intelligent agent deployed by the regional server (network structure as shown in Figure 1) Figure 2 As shown in the figure, the vehicle's path request and road network status information are taken as input, and the cross-domain path planning of the vehicle is output, that is, the next area of ​​the vehicle and the cut edge to reach the next area. According to the cross-domain path planning results, the regional server will plan the intra-domain path to the target cut edge according to the starting and ending points of the vehicle in the area and the current traffic conditions, and finally send the calculated path to the vehicle, that is, the sequence of road sections to reach the next area. When the vehicle arrives at the next area along the path, it sends a path request to the server of the new area, and the new regional agent plans the path for the next stage.

[0079] During the training process, when a vehicle completes the action of a regional agent and is about to leave the area, it will send the reward information of the action to the corresponding regional server, that is, the negative value of the time required to complete this action. Each regional server will upload the road condition observation information within the area, all vehicle path request information, vehicle path planning results and action reward information to the central server. The central server will use the centralized training method (such as Figure 1 c) The cross-domain path planning strategy for training agents, the central server estimates the state value based on the global state information and the observation information of each vehicle path request from the regional agent, and uses it to update the agent's policy network, and returns the updated regional agent policy network parameters to each regional server. The centralized critic network used to estimate the state value is updated by the reward information obtained. This network structure is only used in the training phase. During the execution process, the regional agent only needs to obtain the observation information of the internal traffic conditions of the region and the path request information of the vehicle, and exchange a small amount of data with the central server, that is, upload the traffic profile information of the region and obtain the traffic profile information of the neighboring region.

[0080] The above description is only a description of the preferred embodiments of the present application, and is not intended to limit the scope of the present application. Any changes or modifications made by any person skilled in the art based on the above disclosed technical contents shall be deemed as equivalent effective embodiments and shall fall within the scope of protection of the technical solution of the present application.

[0081] References:

[0082] 1.F.Arasteh,S.SheikhGarGar,and M.Papagelis.Network-aware multiagent

[0083] reinforcement learning for the vehicle navigation problem.InProceedings of the 30th

[0084] International Conference on Advances in Geographic InformationSystems.Associationfor Computing Machinery.(2022)

[0085] 2.P.A.Lopez,M.Behrisch,L.Bieker-Walz,J.Erdmann,Y.-P.Fl¨otter¨od,R.Hilbrich,L.L¨ucken,J.Rummel,P.Wagner,and E.Wiessner.Microscopic trafficsimulation

[0086] using sumo.In International Conference on Intelligent TransportationSystems,pages

[0087] 2575–2582.(2018)

[0088] 3.G.Karypis.Metis:Unstructured graph partitioning and sparse matrixordering system.

[0089] Technical report.(1997)

[0090] 4.J.Yin,W.Rao,Q.Zhao,C.Zhang,and P.Hui.Learn to optimize theconstrained

[0091] shortest path on large dynamic graphs.IEEE Trans.Mob.Comput.,23(3):2456–2469, (2024)

[0093] 5.C.Yu,A.Velu,E.Vinitsky,J.Gao,Y.Wang,A.M.Bayen,and Y.Wu.Thesurprisingeffectiveness of PPO in cooperative multi-agent games.In Advancesin Neural

[0094] Information Processing Systems 35.(2022)

[0095] 6.K.Cho,B.van Merrienboer,C,.G¨ulc,ehre,D.Bahdanau,F.Bougares,H.Schwenk,and Y.Bengio.Learning phrase representations using RNN encoder-decoder forstatistical machine translation.In Proceedings of the 2014Conference on EmpiricalMethods in Natural Language Processing,pages 1724–1734.(2014)

[0096] 7.Z.Wang and S.Wang.Xrouting:Explainable vehicle rerouting for urbanroadcongestion avoidance using deep reinforcement learning.In 2022 IEEEInternationalSmart Cities Conference,pages 1–7.(2022)

[0097] 8.J.Schulman,F.Wolski,P.Dhariwal,A.Radford,and O.Klimov.Proximalpolicyoptimization algorithms.arXiv preprint arXiv:1707.06347.(2017)。

Claims

1. A cross-domain vehicle path optimization method based on multi-agent reinforcement learning, characterized in that: The original road network in the transportation system is divided into multiple areas, and each area is modeled as an intelligent agent. The intelligent agent plans cross-domain paths for vehicles in the area and learns adaptive and cooperative strategies through reinforcement learning algorithms.

2. The cross-domain vehicle path optimization method based on multi-agent reinforcement learning as claimed in claim 1, characterized in that: The first division part: divide the road network into M areas, model each area as an intelligent agent, and the intelligent agent selects the cut edge leaving the area and the path inside the area for the vehicles in the area; Second definition part: The multi-vehicle routing problem is defined as an observable Markov decision process, and the multi-agent reinforcement learning is defined as follows: Agent: Each region is modeled as a separate agent, and all agents form a collection When vehicle l leaves from one area or enters a new area, vehicle l sends the trip information to the relevant regional agent R i , Agent R i A path will be planned for vehicle 1 and the path calculation result will be sent to vehicle 1, wherein the path includes the path between areas and the path within the area; Action: When vehicle l enters the road network or reaches the cut edge of the entry area, the regional agent needs to take action. For each vehicle query request, the regional agent R i From the cut edge collection of the region Select an edge as the path selection between regions. This choice determines the next region that the vehicle is about to enter and the edge through which it enters the next region. Suppose at the tth time step, the agent R i The received path query request set is For each query request Agent R i Make a move Observation: Each regional agent R i Observation information It is divided into two parts: observation of the road network environment and observations for vehicle query requests First, observation of the road network environment The road network environment includes the road segment status information within the area and the estimation of the road status information of the neighboring area, wherein the road status information of the neighboring area includes the average driving speed and the total number of vehicles in the neighboring area; the road network environment represents the road network information as Where F is the input feature dimension. The features of each cut edge include the coordinates of the start and end points of the edge, the length of the edge, the average speed and total number of vehicles on the edge, and the average speed and total number of vehicles in the area connected by the edge. Secondly, the feature of each vehicle route request is expressed as Where F' is the input feature of the vehicle path request, including the coordinates of the vehicle's current position and destination, the vehicle's arrival The time estimation of the cutting edge and The distance from the cut edge to the vehicle’s destination is characterized by the regional agent R i It is calculated based on the starting and ending point information sent by the vehicle; at each time step, R i receive path query request, agent R i The observation of the k-th query request at time t is expressed as As the agent R i The policy network π i enter; Global state: The global state s at time T t It is represented as the connection of all the features of the cut edges. This state representation contains the observation information of M agents. At the same time, the features of the cut edges include the road condition information of these key sections and the average driving speed information of each area. Strategy: Given an agent R i Observation Agent R i The policy network π i Output probability distribution This probability distribution represents the choice of each action The probability of the intelligent experience will select cross-domain actions for the vehicle according to this probability distribution, that is, A cut edge in Reward: Agent R i About route requests The local reward is defined as Among them, t' is the time when the vehicle enters the action section time step; the reward design is consistent with minimizing the total travel time of the vehicle to complete the trip; Third applied part: A regional server is deployed in each area to plan routes for vehicles in the area and communicate with the vehicles in the area; and a central server is deployed to train algorithms and communicate with the regional servers.

3. The cross-domain vehicle path optimization method based on multi-agent reinforcement learning as claimed in claim 2, characterized in that: Agent R i The policy network π i , including road network feature extraction, vehicle path request feature extraction and vehicle action output, among which; Road network feature extraction: The road network feature extraction process aims to extract the characteristics of the traffic road network. According to the results of the road network division, the regional intelligent agent R is connected i The set of cut edges is expressed as The characteristics of each cutting edge are It contains the location information of the road section corresponding to the cut edge, traffic congestion information, and traffic condition information of the area connected by the road section; the input features of each cut edge are encoded through a multi-layer perceptron network to obtain a latent vector representation By The hidden vector of the cut edge represents the connection, and the regional agent R is obtained i Feature extraction for road networks Vehicle route request feature extraction: At time t, agent R i The received vehicle route request set is For agent R at time t i The route request received for the kth vehicle Features Contains the vehicle's current position and target position information; it is encoded into a vector through a multi-layer perceptron network Will The vehicle path request features are input into the gated recurrent unit (GRU) for feature extraction, namely The last latent vector of the GRU unit is used as the vector representation of multiple vehicle query requests, that is, Vehicle action output: Through the road network encoding process and the vehicle path request encoding process, three vector representations are obtained: road network vector representation The route request code of the current vehicle and multiple vehicles route request encoding Taking these three vector representations as input, the agent generates a selection cut edge The probability distribution of , and select the cutting edge according to the probability; First, a multi-layer perceptron is used to map the vector representation into The score of the cut edge is adjusted to -∞ using the action mask operation, and then the edge score is converted to a choice through the softmax function. The probability of cutting edges as cross-domain actions is According to the vector representation and The agent identifies routing requests for vehicles with similar locations.

4. The cross-domain vehicle path optimization method based on multi-agent reinforcement learning as claimed in claim 2, characterized in that: Agent R i A state transition of Represents the agent R i At time t, according to the observed information Make a move The global status at this time is Then at time t′, the vehicle completes the action And get rewards The global state transition is 5. The cross-domain vehicle path optimization method based on multi-agent reinforcement learning as claimed in claim 2, characterized in that: For each state transition Calculate the advantage function of an action and discount value Among them, γ is the discount parameter, V is the state value function; Then, by minimizing the loss function L(θ i ) Optimize the policy network π of the actor i The parameter θ i , the loss function L(θ i ) is defined as follows: in, represents the expected value, θ i are the parameters of the policy network, For the old strategy of collecting trajectories, the clip function will Clip to the interval [1-∈, 1+∈] to avoid excessive updates to the strategy. The value range of the clip parameter ∈ is (0, 1); The critic network parameter φ is optimized by minimizing the squared error. The loss function L(φ) is defined as: in, represents the computational expectation, and φ is the parameter of the critic network.

6. The cross-domain vehicle path optimization method based on multi-agent reinforcement learning as claimed in claim 2, characterized in that: The application part is specifically: The regional server obtains the traffic condition information of all road sections in the region, and obtains the traffic profile information of the neighboring area by accessing the cloud server, including the number of vehicles and average speed in the neighboring area; When a vehicle departs from or arrives at an area, it will send the vehicle's current location information and target location to the server in the area. The strategy network of the intelligent agent deployed in the regional server will take the vehicle's path request and road network status information as input, and output the vehicle's cross-domain path planning, that is, the vehicle's next area and the cut edge passed to reach the next area. According to the cross-domain path planning results, the regional server will plan the intra-domain path to the target cut edge according to the starting and ending points of the vehicle in the area and the current traffic conditions, and finally send the calculated path to the vehicle, that is, the road segment sequence to the next area; when the vehicle arrives at the next area along the path, it will send a path request to the server in the new area, and the new regional intelligent agent will plan the path for the next stage for it; During the training process, when a vehicle completes the action of the regional agent and is about to leave the area, it will send the reward information of the action to the corresponding regional server, that is, the negative value of the time required to complete this action. Each regional server will upload the road condition observation information within the area, all vehicle path request information, vehicle path planning results and action reward information to the central server. The central server will use the centralized training method to train the cross-domain path planning strategy of the agent based on the information of all agents and environment interactions. The central server will estimate the state value based on the global state information and the observation information of each vehicle path request of the regional agent, and use it to update the policy network of the agent, and return the updated regional agent policy network parameters to each regional server. The centralized critic network used to estimate the state value is updated by the reward information obtained. This network structure is only used in the training phase; During the execution process, the regional agent only needs to obtain observation information of the internal traffic conditions in the region and the path request information of the vehicle, and exchange data through the central server, that is, upload the traffic profile information of the region and obtain the traffic profile information of the neighboring area.

Images