A navigation method for traffic uncertainty topological network

Abstract

The application discloses a kind of navigation methods for traffic uncertainty topological network, intelligent traffic path planning technical field, comprising: definition uncertainty parameter, the traffic network to be navigated is modeled as uncertainty topological network, and is converted into variational Markov process;Based on variational Markov process process, construct reinforcement learning network, including the policy network based on variational policy gradient module and the value network based on mask time difference module;Strategy network and value network are updated using off-policy, and the interaction of policy-value network is combined, and the reinforcement learning network is iteratively trained to convergence, and the optimal routing strategy is output;The optimal routing strategy is deployed to agent, and the corresponding navigation path is output according to real-time observation value.The application solves the reliable navigation problem of agent when "topological uncertainty" and "travel time randomness" exist simultaneously in traffic network, and realizes the maximum random punctual arrival probability of agent.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent traffic path planning, and particularly relates to a navigation method for a traffic uncertainty topological network. BACKGROUND

[0002] In the field of robot navigation and intelligent traffic technology, reliable navigation in an uncertain topological network is a key direction with important research value and practical application significance. The core technical requirement is to enable intelligent agents such as autonomous vehicles and mobile robots to reach the destination with the maximum random punctual arrival probability in a scenario where the network topology is uncertain (i.e., some paths may suddenly become impassable during navigation) and the path travel time has random fluctuations. This technology can be widely applied to emergency medical rescue (such as ambulance path planning), urban traffic scheduling, and other scenarios with high time sensitivity requirements, and has important significance for improving the reliability and practicality of intelligent agent navigation. Currently, solutions in the field to meet the above reliable navigation requirements mainly fall into two categories: one is the punctual arrival routing algorithm for travel time randomness, which includes dynamic programming, mathematical programming, and traditional reinforcement learning methods. Almost all of these algorithms assume that the network topology remains unchanged during navigation. The other is the Canadian traveler problem algorithm for network topology uncertainty, which includes exact methods (such as dynamic programming methods based on Markov decision processes) and heuristic methods (such as path selection strategies based on threshold judgment of path passability). These algorithms mainly focus on solving topology uncertainty problems, but rarely consider the random fluctuation characteristics of path travel time.

[0003] However, the existing technical solutions have obvious deficiencies in practical application and cannot meet the reliable navigation requirements in complex uncertain environments. Specifically, traditional punctual arrival routing algorithms for travel time randomness assume that the network topology remains unchanged, so when encountering sudden path impassability (such as road congestion during peak hours or road blockage due to sudden traffic accidents), they need to frequently re-execute path planning, which not only significantly reduces the punctual arrival probability of intelligent agents, but also generates high computational overhead. The Canadian traveler problem algorithm for topology uncertainty can handle topology changes, but its exact methods have high computational complexity and are difficult to adapt to large-scale network scenarios. Heuristic methods have no explicit performance guarantee, and most of these algorithms do not consider travel time randomness, which cannot meet the core requirement of intelligent agent punctual arrival. There is currently no efficient solution that can simultaneously consider network topology uncertainty and travel time randomness, which significantly limits the reliable navigation performance and practical application value of intelligent agents in complex uncertain environments. SUMMARY

[0004] In view of the above problems in the prior art, the navigation method for a traffic uncertainty topological network provided by the present application solves the technical problems in the prior art that the uncertainty of network topology (part of the path may suddenly become impassable) and the randomness of path travel time cannot be considered simultaneously, resulting in low probability of random punctual arrival of an intelligent agent (such as an autonomous vehicle or a mobile robot), poor sample efficiency, and large real-time calculation overhead,

[0005] In order to achieve the above-mentioned object, the technical scheme adopted by the present application is as follows: a navigation method for a traffic uncertainty topological network, comprising the following steps:

[0006] S100, defining an uncertainty parameter and modeling a traffic network to be navigated as an uncertainty topological network;

[0007] S200, converting a navigation problem in the uncertainty topological network into a variational Markov process based on a random action set composed of the uncertainty parameter;

[0008] S300, constructing a reinforcement learning network based on the variational Markov process, including a policy network based on a variational policy gradient module and a value network based on a masked time-difference module;

[0009] S400, updating the policy network and the value network using an off-policy, and iteratively training the reinforcement learning network until convergence in combination with policy-value network interaction, and outputting an optimal routing strategy;

[0010] S500, deploying the optimal routing strategy to an intelligent agent, and outputting a corresponding navigation path according to real-time observation values.

[0011] Further, in the step S100, the uncertainty parameter includes a topological uncertainty and a travel time randomness.

[0012] The topological uncertainty is represented by defining a passable probability of an edge in the uncertainty topological network , and is dynamically updated according to changes in the traffic network topology; wherein only when the intelligent agent arrives at a node , it is observed whether the edge is passable, and a set of passable edges at the node is denoted as .

[0013] The travel time randomness is represented by defining a travel time of an edge in the uncertainty topological network ; wherein the travel time is a random variable, and follows a normal distribution with a mean of and a standard deviation of .

[0014] Further, in the step S200, the elements of the variational Markov process include an agent, a state, a set of random actions, a reward, and an objective function;

[0015] The agent is an entity in the traffic network to be navigated;

[0016] The state is the current node of the agent in combination with the remaining time budget ; ;

[0017] The set of random actions is a set formed by randomly selecting an executable edge from the set of actually passable edges when the agent is in the state ; ;

[0018] The reward is a negative value of the travel time of the edge when the agent performs the action ;

[0019] The objective function is to maximize the probability of the remaining time when the agent reaches the terminal point.

[0020] Further, in the step S300, in the reinforcement learning network, the variational policy gradient module is used to parameterize the routing strategy of the agent, and the policy network parameters are updated by gradient ascent to maximize the on-time arrival rate;

[0021] In the variational policy gradient module, the routing strategy of the agent is parameterized by a variational linear SoftMax policy, which is represented as:

[0022]

[0023] In the formula, represents the action corresponding to the routing strategy in the state , represents the feature vector corresponding to the action in the state , represents the remaining time budget, represents the one-hot encoding vector of the edge , represents the set of random actions, represents the policy network parameters, represents the next action, and the superscript​​​ Indicates the transpose operation;

[0024] By collecting Routing policy The generated trajectory is used to calculate the ascending gradient of the routing policy and then update the policy network parameters. It is represented as:

[0025]

[0026] In the formula, Indicates policy network parameters gradient operator, Indicates routing policy The expected probability of on-time arrival. Indicates an indicator function, Indicates the first A trajectory, Indicates the first The total time taken for each trajectory Indicates the first The length of the trajectory, Indicates length is The The state corresponding to each trajectory Indicates length is The The action corresponding to each trajectory Representing state Next action The corresponding feature vector, Representing state A set of random actions Representing state Next action The corresponding routing strategy, Indicates the total number of generated trajectories. Indicates the maximum trajectory length. Indicates the trajectory length index;

[0027] The formula for updating network parameters using the gradient ascent strategy is:

[0028]

[0029] In the formula, This represents the updated policy network parameters. This represents the learning rate of the policy network.

[0030] Further, in the reinforcement learning network, the masked temporal difference module is configured to estimate the value of the action and the state through the value function, calculate a target value through a state transition observed in real time, and then update the value function through a stochastic gradient descent to minimize a mean square error.

[0031] The value function includes a generalized state-action value function and a generalized state value function, which are respectively represented as:

[0032]

[0033]

[0034] wherein, represents a generalized state-action value, represents a generalized state value, represents a state, represents an action, represents a value network parameter, represents a state -action corresponding to a feature vector, represents an exponential function, and the superscript represents a transposition operation, represents a state -action routing strategy, represents a set of random actions;

[0035] The target value is:

[0036]

[0037] wherein, represents a state after transition, represents a node and a remaining time budget , represents an end point in an uncertainty network, represents a state corresponding to a generalized state value after transition;

[0038] The update formula of the value function is:

[0039]

[0040] wherein, represents an updated value network parameter, represents a learning rate of the masked temporal difference module.

[0041] Further, the method for updating the value network with off-policy is:

[0042] Correcting the behavior policy based policy with importance trajectory ratio The sampled trajectory, update the routing policy gradient, and then update the policy network parameters

[0043] The method for updating the value network with off-policy is:

[0044] On the basis of updating the policy network, using Correct the action weight, and then update the value network parameters ; wherein denotes the state action corresponding to the routing policy, denotes the state action corresponding to the behavior policy.

[0045] Further, the formula for updating the routing policy gradient is:

[0046]

[0047]

[0048] In the formula, denotes the gradient operator of the policy network parameters , denotes the expected on-time arrival probability of the routing policy , denotes the indicator function , denotes the th trajectory, denotes the total time consumption of the th trajectory, denotes the length of the th trajectory, denotes the state corresponding to the th trajectory with length , denotes the action corresponding to the th trajectory with length , denotes the routing policy corresponding to the state action , denotes the total number of generated trajectories , denotes the maximum trajectory length , denotes the trajectory length index

[0049] denotes the th​​​​​​ The importance ratio of each trajectory, Representing state Next action Corresponding behavioral strategies.

[0050] Further, in step S400, the reinforcement learning network is iteratively trained until convergence, and the optimal routing strategy is output, including:

[0051] S401. Initialize the policy network parameters and value network parameters as random vectors;

[0052] S402. Set the training hyperparameters, including the maximum number of iterations, the number of trajectories collected per iteration, the learning rate of the policy network, and the learning rate of the value network;

[0053] S403. Execute the current behavior strategy and collect the trajectory to store in the experience replay pool;

[0054] S404. Sample trajectories from the experience replay pool and update the value network parameters according to the departure strategy;

[0055] S405. Use the generalized state-action value output by the value network with updated parameters as the evaluation signal for updating the policy network, and then update the policy network parameters according to the policy.

[0056] S406. Synchronize the updated policy network parameters to the behavior policy;

[0057] S407. Repeat steps S403 to S406 until the on-time arrival probability fluctuation of the routing policy output by the policy network is less than the preset threshold, complete the reinforcement learning network training, and output the optimal routing policy.

[0058] Further, step S500 includes the following sub-steps:

[0059] S501. Deploy the optimal routing strategy to the agent;

[0060] S502. Substitute the current node, remaining time, and set of traversable edges observed by the agent in real time into the routing strategy of the agent parameterized by the variational linear SoftMax strategy, and calculate the selection probability of each edge in the set of traversable edges.

[0061] S503. Use the edge corresponding to the maximum selection probability as the action to be executed by the agent at the current node;

[0062] S504. Repeat steps S502 to S503 until the agent reaches the destination under the executed action or the remaining time is less than zero, and output the navigation path.

[0063] The beneficial effects of this invention are as follows:

[0064] (1) The present application is directed to a to-be-navigated traffic network, breaks through the limitation of traditional Markov process (MDP) "deterministic action set", proposes a "variational MDP" framework, integrates "random action set caused by topological uncertainty" and "travel time randomness" into a unified mathematical model, clearly defines the state, action and transition rule in the reinforcement learning network, and realizes the modeling process of the first same double uncertainty; The variational MDP framework proposed by the present application solves the defects of traditional algorithms "trade-offs" (such as SOTA algorithm only considers travel time randomness, CTP algorithm only considers topological uncertainty); In SiouxFalls, Anaheim and other four types of traffic networks, the on-time arrival probability (SOTA probability) is improved by 10%-20% compared with traditional algorithms (OS-MIP, PQL, RAO*).

[0065] (2) In the construction of the reinforcement learning network, the present application adopts a reinforcement learning architecture combining a variational policy gradient module and a masked time difference module; wherein, the value network based on the masked time difference module provides an online dense update signal, solving the problem of "sparse update signal and slow convergence" of pure policy algorithms; The variational policy gradient module ensures that the value estimation only considers the passable edges, and adapts to the topological dynamic change. The reinforcement learning architecture provided by the present application has a convergence speed improved by more than 40% compared with pure policy algorithms (VPG); In the scene of frequent topological changes (such as road congestion during peak hours), the value estimation error is reduced by 30%, and the policy adjustment is more timely.

[0066] (3) On the basis of constructing the variational policy gradient module and the masked time difference module, the present application performs off-policy update, corrects the distribution deviation of the behavior policy and the target policy through "importance sampling ratio", and realizes the reuse of historical trajectory data. The off-policy update method proposed by the present application reduces the sample demand by 60% and improves the convergence speed by more than 30% compared with the same type of on-policy algorithm (VAC); In a large-scale network (such as the Winnipeg network), the training time is shortened from 12 hours to 5 hours.

[0067] (4) In the reinforcement learning training stage, the present application adopts double network collaborative optimization, and only the policy network is needed in the deployment stage, and the calculation complexity is (wherein, represents the number of edges in the traffic network graph, is the average number of passable edges, H is the length of the trajectory, represents the maximum number of training rounds), the inference time of the above training method is much lower than that of traditional planning algorithms, and in a network of 1000 edges, the time consumption of a single action selection is less than 10ms; It has been verified in indoor environment and real building that it can still maintain an on-time arrival probability of more than 90% when facing obstacle blockage. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 A flow chart of the navigation method for the traffic uncertainty topological network is provided. DETAILED DESCRIPTION

[0069] The specific embodiments of the present application are described below to facilitate the understanding of the present application for those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments, and for those skilled in the art, it is obvious that various changes are within the spirit and scope of the present application defined and determined by the appended claims, and all the inventions utilizing the concept of the present application are within the scope of protection.

[0070] The present application provides a navigation method for a traffic uncertainty topological network, which is used to solve the reliable navigation problem of an intelligent agent (such as an autonomous vehicle or a mobile robot) in a traffic network with both "topological uncertainty" (such as sudden road blockage) and "travel time randomness" (such as traffic fluctuation), and the core goal is to maximize the random punctual arrival probability of the intelligent agent.

[0071] Reference Figure 1 The navigation method comprises the following steps:

[0072] S100, defining an uncertainty parameter, and modeling the traffic network to be navigated as an uncertainty topological network;

[0073] S200, converting the navigation problem in the uncertainty topological network into a variational Markov process based on a random action set composed of the uncertainty parameter;

[0074] S300, constructing a reinforcement learning network based on the variational Markov process, including a policy network based on a variational policy gradient module and a value network based on a masked time difference module;

[0075] S400, updating the policy network and the value network using off-policy, and combining policy-value network interaction to iteratively train the reinforcement learning network to convergence, and output an optimal routing strategy;

[0076] S500, deploying the optimal routing strategy to the intelligent agent, and outputting a corresponding navigation path according to real-time observation values.

[0077] In step S100 of the present embodiment, the traffic network to be navigated is modeled as a directed connected graph wherein, is a node set (such as an intersection or a room), represents the total number of nodes, and the starting point is , and the ending point is ; is an edge set (such as a road or a room passage), Represents the total number of edges, for any one edge. Indicates from node To the node The path;

[0078] Furthermore, based on the directed connected graph, an uncertain topology network is formed by defining uncertainty parameters. Specifically, the uncertainty parameters include topological uncertainty and travel time randomness.

[0079] Topological uncertainty is defined by the walkable probability of edges in an uncertain topological network. This indicates that it is dynamically updated as the transportation network topology changes; among which, Only when the agent reaches the node At that time, the edge was observed Whether it is passable, will be determined by the node. The set of walkable edges at a point is denoted as . ;

[0080] Travel time randomness is defined by the travel time of edges in an uncertain topological network. Indicates; among which, travel time Let be a random variable, and follow the mean of . Standard deviation is It follows a normal distribution.

[0081] For the aforementioned uncertain topology network, the arrival node of the agent is defined. The remaining time budget is ,like Or reach the destination The navigation cycle ends.

[0082] In step S200 of this embodiment, the elements of the variational Markov process include an agent, a state, a set of random actions, a reward, and an objective function.

[0083] Here, the intelligent agent is an entity in the traffic network to be navigated; for example, the entity includes autonomous vehicles, mobile robots, etc., which are responsible for perceiving the state, selecting actions and updating strategies.

[0084] The current node of the agent. With remaining time budget combination ;

[0085] Random action set When the agent is in a state At that time, perform the action For nodes Set of actual walkable edges A randomly selected executable edge The set formed;

[0086] The reward is when the agent performs an action. For executable edges At that time, on the side Negative travel time; specifically, rewards. During navigation, enable cumulative rewards. The smaller the value, the less time is required, and the easier it is to arrive on time.

[0087] The objective function is to find the optimal routing strategy to maximize the remaining time when the agent reaches the destination. The probability of.

[0088] In step S300 of this embodiment, in the reinforcement learning network, the variational policy gradient module is used to parameterize the routing policy of the agent and update the policy network parameters through gradient ascent to maximize the on-time arrival rate.

[0089] In the variational policy gradient module, the routing policy of the agent is parameterized using a variational linear SoftMax policy, which is expressed as:

[0090]

[0091] In the formula, Representing state Next action The corresponding routing strategy, Representing state Next action The corresponding feature vector, Indicates the remaining time budget. Representing an edge One-hot encoded vector, Represents a set of random actions. Indicates the policy network parameters, Indicates the next action, superscript This represents the transpose operation; where, Only corresponding edges The position is 1, and the rest are 0, therefore .

[0092] By collecting Routing policy The generated trajectory is used to calculate the ascending gradient of the routing policy and then update the policy network parameters. It is represented as:

[0093]

[0094] In the formula, Indicates policy network parameters a gradient operator, a routing policy a desired on-time arrival probability, an indicator function, a first trajectory, a total time consumption of a first trajectory, a length of a first trajectory, a state corresponding to a first trajectory with a length of , an action corresponding to a first trajectory with a length of , a feature vector corresponding to an action under a state , a set of random actions of a state , a routing policy corresponding to a next action under a state , a total number of generated trajectories, a maximum trajectory length, a trajectory length index; wherein for each trajectory , is a terminal state; for the indicator function, a trajectory total time consumption is , otherwise , a reward of a trajectory with a length of .

[0095] The formula for updating the policy network parameters through gradient ascent is:

[0096]

[0097] wherein, denotes the updated policy network parameters, denotes a learning rate of the policy network, .

[0098] In step S300 of the present embodiment, in the reinforcement learning network, the masked time-difference module is used to estimate the value of the action and the state through the value function, and to calculate the target value through the state transition observed in real time, and further to update the value function through the stochastic gradient descent to minimize the mean square error, so as to solve the problem of “update signal sparsity” (feedback only after the end of the trajectory) of the masked time-difference module.

[0099] In this embodiment, the value function includes a generalized state-action value function and a generalized state value function.

[0100] where the generalized state-action value function represents the expected value of a state-action pair selecting an action follows a routing policy of on-time arrival probability, adopts function constraint value in , which is expressed as:

[0101]

[0102] The generalized state value function represents the expected value of a state follows a routing policy of on-time arrival probability, which is weighted sum of action value functions, which is expressed as:

[0103]

[0104] In the formula, represent the generalized state-action value, represent the generalized state value, represent the state, represent the action, represent the value network parameters, represent the state -action corresponding feature vector, represent the exponential function, the superscript represent the transpose operation, represent the state -action routing policy, represent a random action set;

[0105] where the target value is:

[0106]

[0107] In the formula, represent the state after transition, represent the node and the remaining time budget , represent the end point in the uncertainty network, represent the generalized state value corresponding to the state after transition.

[0108] The update formula of the value function is:

[0109]

[0110] In the formula, This represents the updated value network parameters. This represents the learning rate of the mask timing difference module.

[0111] In step S400 of this embodiment, the off-policy method is an enhancement to the policy network and the value network. It addresses the problem of low sample efficiency in the on-policy algorithm (which can only use real-time data generated by the target policy) by reusing historical trajectory data to accelerate training.

[0112] Specifically, the method for updating the policy network using off-policy is as follows:

[0113] Using importance trajectory ratio correction based on behavioral strategies The sampled trajectory is used to update the routing policy gradient, which in turn updates the policy network parameters. Among them, behavioral strategies Used to collect trajectory data, it can be combined with routing strategies. Consistent variational linearity Strategy.

[0114] The formula for updating the gradient of the routing policy is:

[0115]

[0116]

[0117] In the formula, Indicates policy network parameters gradient operator, Indicates routing policy The expected probability of on-time arrival. Indicates an indicator function, Indicates the first A trajectory, Indicates the first The total time taken for each trajectory Indicates the first The length of the trajectory, Indicates length is The The state corresponding to each trajectory Indicates length is The The action corresponding to each trajectory Representing state Next action The corresponding routing strategy, Indicates the total number of generated trajectories. Indicates the maximum trajectory length. Indicates the trajectory length index;

[0118] represent the importance of the i-th trajectory trajectory importance ratio, represent the state next action corresponding routing policy, represent the state next action

[0119] corresponding behavior policy.

[0120] The method for updating the value network off-policy is: correct the action weight, and then update the value network parameters ; wherein, represent the state next action corresponding routing policy, represent the state next action corresponding behavior policy.

[0121] In step S400 of the embodiment, based on the "policy-value network interaction" and "off-policy data multiplexing", the reinforcement learning network is iteratively trained to convergence, and the optimal routing policy is output, including:

[0122] S401, initialize the policy network parameters and the value network parameters to random vectors;

[0123] S402, set the training hyperparameters, including the maximum number of iterations, the number of trajectories collected each iteration, the policy network learning rate, and the value network learning rate;

[0124] S403, execute the current behavior policy and collect trajectories to store in the experience replay pool;

[0125] S404, sample trajectories from the experience replay pool, and update the value network parameters off-policy;

[0126] S405, output the generalized state-action value of the value network with updated parameters as the evaluation signal for the policy network update, and then update the policy network parameters off-policy;

[0127] S406, synchronize the updated policy network parameters to the behavior policy;

[0128] S407, repeat steps S403-S406 until the on-time arrival probability of the routing policy output by the policy network fluctuates less than a preset threshold, complete the reinforcement learning network training, and output the optimal routing policy.

[0129] Exemplarily, in the training process, the updated policy network parameters are synchronized to the behavior policy to balance exploration and utilization every 10 iterations; when the convergence judgment is performed, if the probability fluctuations of the on-time of 5 consecutive iterations are all less than 0.01, the training is terminated.

[0130] The step S500 of the embodiment comprises the following sub-steps:

[0131] S501, deploying the optimal routing strategy to the agent;

[0132] S502, substituting the current node, the remaining time and the set of passable edges observed by the agent in real time into the routing strategy of the agent parameterized by the variational linear SoftMax strategy, and calculating the selection probability of each edge in the set of passable edges;

[0133] S503, taking the edge corresponding to the maximum selection probability as the action performed by the agent at the current node;

[0134] S504, repeating the steps S502-S503 until the agent reaches the end point or the remaining time is less than zero under the performed action, and outputting the navigation path.

[0135] The principles and implementation manners of the present application are described by the specific embodiments in the present application, and the above embodiment descriptions are only used to help understand the method of the present application and its core idea; meanwhile, for the ordinary skilled in the art, according to the idea of the present application, the specific implementation manners and application ranges can be changed, and the above descriptions should not be understood as the limitation of the present application.

[0136] The person skilled in the art will realize that the embodiments described herein are used to help the reader understand the principles of the present application, and should be understood as the protection scope of the present application not being limited to such specific descriptions and embodiments. The person skilled in the art can make various other specific modifications and combinations according to the technical inspirations disclosed in the present application without departing from the essence of the present application, and these modifications and combinations still fall within the protection scope of the present application.

Claims

1. A navigation method for traffic-uncertain topology networks, characterized in that, Includes the following steps: S100. Define uncertainty parameters and model the traffic network to be navigated as an uncertain topology network. The uncertainty parameters include topological uncertainty and travel time randomness; S200, based on a set of random actions composed of uncertain parameters, transforms the navigation problem in uncertain topological networks into a variational Markov process; S300. Based on variational Markov processes, reinforcement learning networks are constructed, including a policy network based on variational policy gradient modules and a value network based on masked temporal difference modules. S400: The policy network and value network are updated using a policy-value network interaction method, and the reinforcement learning network is iteratively trained until convergence, outputting the optimal routing policy. S500 deploys the optimal routing strategy to the agent and outputs the corresponding navigation path based on real-time observations; In step S300, in the reinforcement learning network, the variational policy gradient module is used to parameterize the agent's routing policy and update the policy network parameters through gradient ascent to maximize the on-time arrival rate; the mask temporal difference module is used to estimate the value of actions and states through the value function, calculate the target value through real-time observed state transitions, and then update the value function by minimizing the mean square error through stochastic gradient descent.

2. The navigation method for traffic uncertainty topology networks according to claim 1, characterized in that, In step S100, the topological uncertainty is defined by specifying the walkability probability of edges in an uncertain topological network. This indicates that it is dynamically updated as the transportation network topology changes; among which, Only when the agent reaches the node At that time, the edge was observed Whether it is passable, will be determined by the node. The set of walkable edges at a point is denoted as . ; The randomness of travel time is defined by specifying the travel time of edges in an uncertain topological network. Indicates; among which, travel time Let be a random variable, and follow the mean of . Standard deviation is It follows a normal distribution.

3. The navigation method for traffic uncertainty topology networks according to claim 2, characterized in that, In step S200, the elements of the variational Markov process include an agent, a state, a set of random actions, a reward, and an objective function. The intelligent agent is an entity in the traffic network to be navigated; The state is the current node of the agent. With remaining time budget combination ; The set of random actions When the agent is in a state At that time, perform the action For nodes Set of actual walkable edges A randomly selected executable edge The set formed; The reward is given when the agent performs an action. For executable edges At that time, on the side Negative travel time; The objective function is to find the optimal routing strategy and maximize the remaining time when the agent reaches the destination. The probability of.

4. The navigation method for traffic uncertainty topology networks according to claim 1, characterized in that, In step S300, the variational policy gradient module employs a variational linear SoftMax policy to parameterize the agent's routing strategy, which is expressed as follows: In the formula, Representing state Next action The corresponding routing strategy, Representing state Next action The corresponding feature vector, Indicates the remaining time budget. Representing an edge One-hot encoded vector, Represents a set of random actions. Indicates the policy network parameters, Indicates the next action, superscript Indicates the transpose operation; By collecting Routing policy The generated trajectory is used to calculate the ascending gradient of the routing policy and then update the policy network parameters. It is represented as: In the formula, Indicates policy network parameters gradient operator, Indicates routing policy The expected probability of on-time arrival. Indicates an indicator function, Indicates the first A trajectory, Indicates the first The total time taken for each trajectory Indicates the first The length of the trajectory, Indicates length is The The state corresponding to each trajectory Indicates length is The The action corresponding to each trajectory Representing state Next action The corresponding feature vector, Representing state A set of random actions Representing state Next action The corresponding routing strategy, Indicates the total number of generated trajectories. Indicates the maximum trajectory length. Indicates the trajectory length index; The formula for updating network parameters using the gradient ascent strategy is: In the formula, This represents the updated policy network parameters. This represents the learning rate of the policy network.

5. The navigation method for traffic uncertainty topology networks according to claim 1, characterized in that, In step S300, the value function includes a generalized state-action value function and a generalized state value function, which are expressed as follows: In the formula, Represents the generalized state-action value. Represents the value of a generalized state. Indicates state, Indicates an action, Represents the parameters of the value network. Representing state -action The corresponding feature vector, To represent an exponential function, superscript This indicates the transpose operation. Representing state -action Routing strategy, Represents a set of random actions; The target value for: In the formula, Indicates the state after the transition. Represents a node With remaining time budget , Indicates the endpoint in an uncertain network. Indicates the state after the transition. The corresponding generalized state value; The update formula for the value function is: In the formula, This represents the updated value network parameters. This represents the learning rate of the mask timing difference module.

6. The navigation method for traffic uncertainty topology networks according to claim 1, characterized in that, In step S400, the method for updating the policy network using the policy off-policy approach is as follows: Using importance trajectory ratio correction based on behavioral strategies The sampled trajectory is used to update the routing policy gradient, which in turn updates the policy network parameters. ; The method for updating the value network using an off-policy approach is as follows: Based on the updated policy network, utilize Adjust action weights to update value network parameters ;in, Representing state Next action The corresponding routing strategy, Representing state Next action Corresponding behavioral strategies.

7. The navigation method for traffic uncertainty topology networks according to claim 6, characterized in that, The formula for updating the routing policy gradient is: In the formula, Indicates policy network parameters gradient operator, Indicates routing policy The expected probability of on-time arrival. Indicates an indicator function, Indicates the first A trajectory, Indicates the first The total time taken for each trajectory Indicates the first The length of the trajectory, Indicates length is The The state corresponding to each trajectory Indicates length is The The action corresponding to each trajectory Representing state Next action The corresponding routing strategy, Indicates the total number of generated trajectories. Indicates the maximum trajectory length. Indicates the trajectory length index; Indicates the first The importance ratio of each trajectory, Representing state Next action Corresponding behavioral strategies.

8. The navigation method for traffic uncertainty topology networks according to claim 6, characterized in that, In step S400, the reinforcement learning network is iteratively trained until convergence, and the optimal routing policy is output, including: S401. Initialize the policy network parameters and value network parameters as random vectors; S402. Set the training hyperparameters, including the maximum number of iterations, the number of trajectories collected per iteration, the learning rate of the policy network, and the learning rate of the value network; S403. Execute the current behavior strategy and collect the trajectory to store in the experience replay pool; S404. Sample trajectories from the experience replay pool and update the value network parameters according to the departure strategy; S405. Use the generalized state-action value output by the value network with updated parameters as the evaluation signal for updating the policy network, and then update the policy network parameters according to the policy. S406. Synchronize the updated policy network parameters to the behavior policy; S407. Repeat steps S403 to S406 until the on-time arrival probability fluctuation of the routing policy output by the policy network is less than the preset threshold, complete the reinforcement learning network training, and output the optimal routing policy.

9. The navigation method for traffic uncertainty topology networks according to claim 1, characterized in that, Step S500 includes the following sub-steps: S501. Deploy the optimal routing strategy to the agent; S502. Substitute the current node, remaining time, and set of traversable edges observed by the agent in real time into the routing strategy of the agent parameterized by the variational linear SoftMax strategy, and calculate the selection probability of each edge in the set of traversable edges. S503. Use the edge corresponding to the maximum selection probability as the action to be executed by the agent at the current node; S504. Repeat steps S502 to S503 until the agent reaches the destination under the executed action or the remaining time is less than zero, and output the navigation path.

Images