Dual-objective path planning method based on deep reinforcement learning

Abstract

The present invention relates to the field of path planning technology, and in particular to a dual-objective path planning method based on deep reinforcement learning, comprising the following steps: S1, representing the state of a road network, and constructing a reward function r, to obtain a dual-objective deep reinforcement learning path planning model to be trained; the state includes a coordinate state, a distance state, and a cu state; S2, training the dual-objective path planning model to be trained, so that it learns an optimal strategy π that can obtain a maximum cumulative reward * , obtain the trained dual-objective path planning model; the cumulative reward is the sum of the immediate reward given by the reward function and the additional future reward estimated by the neural network; S3, use the trained dual-objective path planning model to perform dual-objective path planning. The present invention has the ability to minimize path distance and the versatility to optimize losses and utilities in different scenarios.

Description

Technical Field

[0001] The present invention relates to the field of path planning technology, and in particular to a dual-objective path planning method based on deep reinforcement learning. Background Art

[0002] Path planning on roads is one of the most basic and important planning activities in smart cities. Usually, urban residents rely on navigation platforms such as GPS navigators and Google Maps to find the ideal path to their destinations. The core path planning algorithms in these navigation services mainly focus on minimizing travel distance or travel time. In recent years, with the increasing demand for diversified path planning services, new path planning strategies that consider more optimization objectives have received widespread attention in recent years. In these new path planning strategies, in addition to optimizing path distance, other indicators are also considered as secondary optimization objectives, such as the scenic beauty of the path, the crime risk near the path, and whether the path is quiet and comfortable. In specific scenarios, the optional second optimization objective can be considered as a loss or utility. For example, the crime risk on a "safe" route and the noise level on a "quiet" route are losses that need to be minimized, while the scenery score along a "scenic" route is a utility that needs to be maximized.

[0003] Existing path planning algorithms focus on either minimizing losses or maximizing utility, and lack the ability to optimize both losses and utility at the same time. Moreover, even the same indicator can be used as both loss and utility in different scenarios. Taking solar radiation as an example, in the hot summer, pedestrians usually tend to choose paths with less radiation, while solar vehicles with charging needs need to receive more solar radiation when planning paths. For another example, in the indicator of crime risk, residents hope to find safe paths with the lowest crime risk, while patrolling police hope to patrol on unsafe paths. Existing graph search-based methods are usually scenario-oriented and difficult to apply to other path planning tasks. For example, the traditional shortest distance algorithm (i.e., Dijkstra) cannot be used to search for paths that maximize utility. For this type of indicator that can be used as both loss and utility, two different algorithms can be used to optimize loss and utility respectively, but such a solution will introduce additional algorithm deployment costs and complexity.

[0004] Therefore, how to have the ability to minimize path distance while also having the versatility to optimize losses and utilities in different scenarios has become a problem that needs to be urgently solved in current dual-objective path planning algorithms. Summary of the invention

[0005] In view of the above-mentioned deficiencies of the prior art, the present invention provides a dual-objective path planning method based on deep reinforcement learning, which not only has the ability to minimize path distance, but also has the versatility to optimize losses and utilities in different scenarios.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] The dual-objective path planning method based on deep reinforcement learning includes the following steps:

[0008] S1. Represent the state of the road network and construct a reward function r to obtain a dual-objective deep reinforcement learning path planning model to be trained; the state includes a coordinate state, a distance state and a cu state;

[0009] The coordinate state includes the latitude and longitude coordinates of the starting point and the latitude and longitude coordinates of the end point; the distance state is the shortest distance from all adjacent nodes of the current node to the end point; the cu is the edge attribute except the path distance and travel time; the cu state is represented by the weight of the cu of all edges connected to the current node, which is used to represent the state of the second optimization goal and help the agent learn the spatial characteristics of cu; the second optimization goal is to minimize the loss or maximize the utility; the reward function r is used to calculate the comprehensive reward of the main goal and the second goal in the current step; the main goal is to minimize the path distance;

[0010] S2: Train the dual-objective path planning model to learn the optimal strategy π that can obtain the maximum cumulative reward * , obtaining a trained dual-objective path planning model; the cumulative reward is the sum of the immediate reward given by the reward function and the additional future reward estimated by the neural network in the dual-objective path planning model;

[0011] S3. Use the trained dual-objective path planning model to perform dual-objective path planning.

[0012] Preferably, in S1, the reward function Among them, r dest represents the main target reward, r cu represents the second target reward, the main target reward is the reward for minimizing the path distance, and the second target reward is the reward for optimizing the path cu; α is a preset hyperparameter for weighing the main target and the second target, when cu represents utility, α is positive, and when cu represents loss, α is negative; is 0 or 1, A value of 0 indicates that the agent has not reached the end point. 1 means the agent has reached the end point; r T For additional future rewards.

[0013] Preferably, in S1, the main target reward r dest =norm -1,1 (dis eff ), where dis eff Indicates the valid distance of the path.

[0014] Preferably, dis eff =(|N o N d |-|A1N d |)-|A1N a |, where |N o N d |-|A1N d |For the intelligent agent Distance moved in the direction; |A1N a | for deviation Distance in direction; N o As the starting point; N d A1 is the end point N d As the coordinate origin, The direction is the x-axis, the normal direction After establishing the coordinate system for the y-axis, from N a The foot point of a perpendicular line drawn from the point to the x-axis; N a Point is the starting point N o A neighboring node of

[0015] Preferably, in S1, r cu =norm 0,1 (cu a ), where cu a is the cu weight on the edge corresponding to the action of the agent.

[0016] Preferably, in S1, the accumulated reward Among them, t is the current step; T is the termination step; r t represents the immediate reward after the agent executes the current step according to the corresponding strategy; γ is the discount factor and γ∈[0,1], which is used to measure the importance of the reward generated by the future steps.

[0017] Preferably, in S2, when training the dual-target path planning model to be trained, a dual-depth Q network is used for training, and the dual-depth network includes a Q network and a target network, and action selection and evaluation are decoupled through the Q network and the target network; the Q network and the target network are both neural networks.

[0018] Preferably, in S2, when training the dual-objective path planning model to be trained, it also includes transfer preprocessing, which is the transfer input into the neural network of the dual-objective path planning model; the transfer is the experience corresponding to the action after the action is completed.

[0019] Preferably, the preprocessing includes arrival detection and reward correction; arrival detection is used to identify actions that can ultimately lead the agent to the end point; reward correction is used to punish actions that produce a cyclic path to prevent the agent from being trapped in a loop.

[0020] Preferably, in S2, when training the dual-objective path planning model to be trained, the Q value Q π (s,a), represents the quality of taking action a given state s and strategy π;

[0021] Among them, E π [·] Compute the expected value of strategy π;

[0022] And will find the optimal strategy π * The problem is transformed into finding the optimal Q value Q * (s,s) problem;

[0023] a * refers to the optimal action in state s; based on the Bellman equation,

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention designs a state representation method to capture the state of the intelligent agent in the environment, which not only includes the spatial state of the current position, but also provides key information of the indicators to be optimized; in this way, the path distance and other optimization indicators (loss or utility) are represented in a unified manner. This comprehensive state can promote the effective training of deep reinforcement learning path planning strategies. In addition, the present invention also designs an effective reward function, in which loss and utility can be represented in a unified manner. Unlike the existing path planning methods for specific scenarios, this method is a general method that can be applied to various dual-objective path planning problems by utilizing deep reinforcement learning, minimizing losses and maximizing utility while minimizing path distance. The trained dual-objective path planning model in the present invention not only has the ability to minimize path distance, but also has the versatility to optimize losses and utilities in different scenarios.

[0026] 2. The reward function designed by the present invention takes into account both the primary target reward and the secondary target reward, and can achieve a balance between the shortest distance and other optimization targets.

[0027] 3. In addition to the rewards for loss and utility, a new concept of effective distance is introduced to establish a trade-off mechanism between the two objectives, thereby ensuring the effectiveness and applicability of the reward function.

[0028] 4. During the training phase, the present invention also includes transition preprocessing, which preprocesses the transitions input into the network. The transition preprocessing includes arrival detection and reward correction. The purpose of arrival detection is to identify actions that can ultimately lead the agent to the end point. The reward correction function penalizes actions that produce a cyclic path, thereby preventing the agent from being trapped in a cycle. Through these two functions, training can be made more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to make the purpose, technical solution and advantages of the invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings, in which:

[0030] Figure 1 It is a flow chart of the dual-objective path planning method based on deep reinforcement learning of the present invention;

[0031] Figure 2 This is the overall framework diagram of the dual-objective path planning model;

[0032] Figure 3 This is an example diagram of the optimal path for dual-objective path planning with solar radiation as cu;

[0033] Figure 4 Schematic diagram of an example of calculating the effective distance;

[0034] Figure 5 Schematic diagram of the distribution of cu in the road network in the verification experiment;

[0035] Figure 6 Schematic diagram of the evaluation results of different application scenarios in the verification experiment;

[0036] Figure 7 Schematic diagram of the evaluation results when different α is set in the reward function in the verification experiment. DETAILED DESCRIPTION

[0037] The following is a further detailed description through specific implementation methods:

[0038] Deep reinforcement learning (DRL) has a flexible reward mechanism and powerful learning ability. The reward can integrate multiple goals to meet the scenarios of loss and utility optimization. Therefore, path planning based on DRL is a potential feasible solution. In recent years, DRL has been widely used in many sequential decision-making research fields such as robot control and autonomous driving. Similarly, the path planning problem is also a sequential decision-making process, and the current road choice has a great impact on future decisions. Recent studies have also shown the possibility of applying DRL to path planning. The agent in DRL uses deep neural networks (DNN) to gain experience in repeated experiments and gradually learns path planning strategies. However, the existing DRL-based path planning methods are not universal and cannot meet the needs of loss and utility optimization at the same time. The applicant expects to find a general dual-objective path planning method to optimize path distance and various diversified indicators. However, to achieve the above goals, the following challenges need to be addressed:

[0039] First, path distance and other optimization metrics (loss or utility) need to be represented in a unified manner;

[0040] Secondly, the weights of the diverse optimization indicators lack spatial proximity on adjacent road sections, and are even sparsely distributed in the city, making them difficult to represent and learn in the DRL framework. For example, the crime risk level on the road is affected by multiple factors such as road conditions, public facilities, and economic level, and the distribution varies greatly on different road sections. In addition, some roads without any crime reports are considered absolutely safe, making the crime risk sparsely distributed throughout the city;

[0041] Third, path planning strategies that optimize loss or utility may lead to more detours, which contradicts the goal of minimizing path distance. Therefore, a balance should be struck between distance and other optimization objectives.

[0042] Based on the above reasons, although there have been attempts to apply deep reinforcement learning to path planning, there is no technical content for universal dual-objective planning.

[0043] Example:

[0044] To facilitate understanding, some terms in this embodiment are first explained.

[0045] The cuRL in this embodiment is a dual-objective path planning model. For the convenience of description and labeling in the drawings, the dual-objective path planning model is named cuRL in this embodiment.

[0046] Road network: A road network is a graph consisting of nodes and edges G = (N, E), where N represents the set of nodes, including intersections and road endpoints; Represents a set of directed edges (roads).

[0047] Edge attributes: Edges in a road network (i.e., a graph) may have certain attributes depending on the application. For example, the most common edge attributes are path distance or travel time. For the diversified optimization criteria described in this paper, edges can carry more types of attributes, including scenery, quietness, comfort, crime risk, etc., thereby achieving a wider range of urban path planning services.

[0048] cu, loss and utility: For edge attributes, it can be loss or utility according to the actual application scenario. Taking the crime risk attribute as an example, for ordinary road pedestrians or vehicles, it should be a loss on the edge; on the contrary, for patrolling police, it becomes a utility on the edge. Therefore, even for the same attribute, whether it is a loss or a utility should be determined by the actual application scenario. For loss, people hope to avoid it as much as possible; while for utility, it is pleasant and helpful, and people will expect to get more. For the sake of ease of expression, only cu is needed to uniformly represent edge attributes. In addition, cu in this embodiment specifically refers to edge attributes other than path distance and travel time.

[0049] Path: A path is a sequence of connected edges or nodes in a road network. It should be noted that unlike edge attributes such as distance where each edge has a certain weight, cu can be discontinuous or even very sparse on a road network. In other words, cu lacks spatial proximity and continuity and may be zero on some edges, depending on the physical meaning of the specific attribute. Taking the crime risk attribute as an example again, in a city, only a portion of the roads near where crimes have occurred or are considered dangerous, and the more crimes there are, the greater the crime risk.

[0050] Dual-objective optimal path: For a given OD (origin-destination) pair, the dual-objective optimal path is defined as the path that achieves the best trade-off between path distance and path cu, i.e., the path with minimum distance and minimized loss c (or maximized utility u).

[0051] The path planning problem in a road network is usually expressed as a search problem on a graph. In fact, it is also a natural sequential decision problem that matches the RL framework (i.e., reinforcement learning framework). The goal of path planning is to learn a decision strategy π * In order to find the optimal path in time according to the current environment. RL is usually expressed as a sequential Markov decision process (MDP), which can be expressed in the form of a five-tuple<S,A,P,R,γ> s∈S represents the state of the agent in the environment, A is the action space containing all actions that the agent can take, and p∈P is the agent in a given state s. t Next, take action a t Then go to the next state s t+1The probability of t ∈R represents the immediate reward after the agent performs an action according to the strategy π. Once an action is completed, the experience corresponding to the action is called transition, which is expressed as Tr = (s t ,a t ,r t ,s t+1 ).

[0052] like Figure 1 As shown, this embodiment discloses a dual-objective path planning method based on deep reinforcement learning, including the following steps:

[0053] S1. Represent the state of the road network and construct the reward function r to obtain the dual-objective deep reinforcement learning path planning model to be trained. The overall framework of the dual-objective path planning model is as follows: Figure 2 shown.

[0054] The state includes coordinate state, distance state and cu state. The coordinate state includes the latitude and longitude coordinates of the starting point and the latitude and longitude coordinates of the end point; the distance state is the shortest distance from all adjacent nodes of the current node to the end point; cu is the edge attribute except the path distance and travel time; the cu state is represented by the weight of cu of all edges connected to the current node, which is used to represent the state of the second optimization goal and help the agent learn the spatial characteristics of cu; the second optimization goal is to minimize loss or maximize utility; the reward function r is used to calculate the comprehensive reward of the main goal and the second goal in the current step.

[0055] For better understanding, the setting principle and function of the status are explained in detail.

[0056] The state represents the current state of the agent in the road network. First, it should contain the current point N c and end point N d The location information tells the agent where it is currently and where to go. Intuitively, there are two ways to represent the location of nodes in the state in the RL framework, namely longitude and latitude coordinates and one-hot encoding.

[0057] When a node is represented by longitude and latitude coordinates, its nearby nodes in the road network will also have similar representations, which may eventually lead to agents choosing the same action for situations with similar starting points. Obviously, such a result may not be applicable to the dual-objective path planning problem. For example, taking minimizing solar radiation, Agent 1 and Agent 2 have two close starting points (N1 and N2) and a common destination (N d ),like Figure 3 shown. and They represent the optimal dual-objective paths of the two OD pairs. Although N1 and N2 are very close in the road network, it is obvious that the optimal paths of Agent 1 and Agent 2 are completely different because the solar radiation distribution on the edges connecting them is also completely different. If the state representation is simply completed using longitude and latitude coordinates, the difference in solar radiation distribution cannot be well distinguished, and Agent 1 and Agent 2 may take the same action due to the close state representation.

[0058] When one-hot encoding is used, no matter how close two nodes are in geographic space, their distance in the representation space is far enough that the two points can be distinguished. However, it also has the following two major disadvantages. First, the dimension of one-hot encoding increases with the size of the road network, making the neural network more complex and leading to the dimensionality disaster. Second, all geographical features in the road network will be completely lost, which will cause new training samples to be unable to utilize the experience of other training samples, which greatly reduces the speed of network convergence, because when using one-hot encoding as a state representation, DDQN (Double Deep Q-network) works in an inefficient memory manner and cannot learn useful knowledge.

[0059] Based on the above analysis, the present invention designs a new state representation, which not only provides sufficient feature information in the road for the agent to learn, but also sufficiently distinguishes the adjacent nodes. Specifically, the state designed by the present invention includes the following three parts:

[0060] Coordinate status. The latitude and longitude coordinates of the start and end points are still retained because coordinate data provides the most intuitive spatial information.

[0061] Distance state. This method also integrates the prior knowledge of the shortest distance into the representation. The distance state is defined as the shortest distance from all adjacent nodes of the current point to the end point.

[0062] cu state. It is used to represent the state of the second optimization objective and help the agent learn the spatial features of cu. The cu state is represented by the weights of cu of all edges connected to the current point.

[0063] In general, a state s consists of a two-tuple (X c ,X d ) definition, where two elements X c , X d Represents the current point N c and end point N d state. Note that X c contains all three states, and X d Distance status omitted.

[0064] Based on this state, the central agent performs an action selection operation each time. In the model, the action space a is set in the form of an octet (i.e., north, northwest, west, southwest, south, southeast, east, northeast), with each element covering 45 degrees in the coordinate system. If there is no connected road in the direction of the selected action, the environment will select an action again until there is a road. When the agent performs an action from one node to the next node in the road network, a reward r is immediately returned for this action. This reward r should be carefully designed to ensure efficient learning of the dual-objective path planning model.

[0065] The reward function Among them, r dest represents the main target reward, r cu represents the second target reward, the main target reward is the reward for minimizing the path distance, and the second target reward is the reward for optimizing the path cu; α is a preset hyperparameter for weighing the main target and the second target, when cu represents utility, α is positive, and when cu represents loss, α is negative; is 0 or 1, A value of 0 indicates that the agent has not reached the end point. 1 means the agent has reached the end point; r T For additional future rewards.

[0066] Specifically, the main goal reward r dest =norm -1,1 (dis eff ), where dis ef Indicates the effective distance of the path;

[0067] dis eff =(|N o N d |-|A1N d |)-|N1N a |, where |N o N d |-|A1N d |For the intelligent agent Distance moved in the direction; |A1N a | for deviation Distance in direction; N o As the starting point; N d A1 is the end point N d As the coordinate origin, The direction is the x-axis, the normal direction After establishing the coordinate system for the y-axis, from N a The foot point of a perpendicular line drawn from the point to the x-axis; N a Point is the starting point N o A neighboring node of like Figure 4 shown.

[0068] The accumulated rewards Among them, t is the current step; T is the termination step; r t represents the immediate reward after the agent executes the current step according to the corresponding strategy; γ is the discount factor and γ∈[0,1], which is used to measure the importance of the reward generated by the future steps.

[0069] S2: Train the dual-objective path planning model to learn the optimal strategy π that can obtain the maximum cumulative reward * , obtaining a trained dual-objective path planning model; the dual-objective path planning model includes a neural network; the cumulative reward is the sum of the immediate reward given by the reward function and the additional future reward estimated by the neural network.

[0070] Among them, when training the dual-objective path planning model to be trained, a double deep Q-network (DDQN) is used to decouple action selection and evaluation through two networks (i.e., Q network and target network), thereby solving the problem of over-estimation of Q value in traditional DQN. The central agent with DDQN gradually learns the optimal strategy to obtain the maximum cumulative reward when interacting with the road network environment (i.e., map environment).

[0071] In specific implementation, the Q value Q π (s,a), represents the quality of taking action a given state s and strategy π;

[0072] Among them, E π [·] Calculate the expected value of the strategy π; and find the optimal strategy π * The problem is transformed into finding the optimal Q value Q * (s,a) problem; a * refers to the optimal action in state s; based on the Bellman equation,

[0073] It also includes transfer preprocessing, which is the transfer input into the neural network of the dual-objective path planning model; the transfer is the experience corresponding to the action after the action is completed. The preprocessing includes arrival detection and reward correction; arrival detection is used to identify the action that can eventually lead the agent to the end point; reward correction is used to punish the action that produces a cyclic path to prevent the agent from being trapped in a loop. For ease of understanding, the transfer preprocessing is explained as follows:

[0074] In the field of deep reinforcement learning, the ∈-greedy method is often used as a strategy for action selection, which tends to select the behavior with the largest neural network output value and considers the exploration of other behaviors with a probability of ∈. The transition set {T r =(s t ,a t ,r t ,s t+1 )} will be stored in the experience recycling pool of DDQN. However, if only the original ∈-greedy method is used, the experimental results are unsatisfactory. Specifically, in the inference phase of the model, ∈ is set to 0, and the agent often wanders in the road network or circles in the loop and cannot reach the end. Therefore, the present invention designs a transfer preprocessing including arrival detection function and reward correction function in the training phase to alleviate these problems.

[0075] Arrival detection uses a buffer to store the transfer set {T r =(s t ,a t ,r t ,s t+1 )}, and then choose whether to send them to the experience recycling pool of DDQN. Only the transitions on the path that can successfully reach the end point will be moved from the buffer to the experience recycling pool to train the model. Reward correction is responsible for cycle path detection. If all actions within the cycle path are not generated in an exploratory manner in the ∈-greedy method, their corresponding rewards will be modified to negative values. Negative rewards penalize behaviors in the cycle, thereby preventing the agent from being trapped in the cycle in subsequent stages. The new transfer is then stored in the experience recycling pool. Due to the existence of the learning rate, even if the reward for the new transfer is negative, the parameters in DDQN will be updated smoothly without drastic fluctuations.

[0076] Figure 2 The algorithm framework of the dual-objective path planning model and the data flow of the training and reasoning stages are shown in FIG. In order to make the training process clearer, the corresponding pseudo code is further provided in this embodiment as follows:

[0077] Algorithm 1: Training of cuRL based on DDQN

[0078]

[0079]

[0080] The loss calculation formula is:

[0081]

[0082] Among them, Q targrepresents the target Q value, and θ is the Q network parameter. In specific implementation, the target network parameter is θ ′ Indicates. And

[0083] In RL, the training process consists of multiple episodes, each of which corresponds to an instance of path planning. In the algorithm, an episode refers to the process of the agent moving from a random starting point to a random end point within a preset maximum step size T. If the agent still does not reach the end point after the given maximum step size, it terminates at this step and starts the next episode. At each step of each episode, the state vector s = (X c ,X d ) are input into the network, and the agent will adopt the ∈-greedy strategy (lines 6 to 10 of Algorithm 1) to balance the exploration process and the exploitation process. During the exploration process, the agent will randomly select an action in the action space; while in the exploitation process, the agent will select the behavior with the highest Q value generated by the DDQN network. The exploitation process uses the experience learned by the agent from historical data, while the exploration process randomly selects actions to avoid falling into local optimality and generate potentially better path planning strategies. In the initial cycle, there is no experience to exploit, so ∈ is set to 1. As experience accumulates in the training phase, as shown in line 11 of the algorithm, ∈ will gradually decrease as the iteration proceeds, where δ is a hyperparameter that adjusts the decay rate of ∈. In the final cycle, ∈ approaches 0, and only the exploitation process is adopted by the algorithm.

[0084] For ease of understanding, in this embodiment, t represents the sequence identifier of the current step in a cycle. t After that, it gets an immediate reward r from the environment t , the environmental state is given by s t Transfer to s t+1 . Transfer T r It is temporarily stored in the buffer B of the transfer preprocessing, and then arrives at the detection and reward correction function to start working. According to the judgment of the transfer preprocessing, the transfer in the buffer B may be moved to the experience recycling pool M in DDQN. M is a queue with a finite length (|M|), which is initially empty and gradually filled with transfers. Once the number of transfers stored in M ​​reaches |M|, the earliest stored transfer will be popped out and replaced by a new transfer. The training frequency of the neural network is adjustable. In this embodiment, it is set to the same as the number of cycles of the agent, that is, the neural network is updated once for each OD. The target network and Q network in DDQN have the same network structure, and the weight parameters are θ′ and θ respectively. The target Q value Q targetUsed to estimate the optimal Q value. In each training, as shown in lines 24 to 30 of Algorithm 1, the Q network parameters θ and the target network parameters θ′ are updated.

[0085] It should be noted that the pre-training stage is necessary because it enables the model to have the basic ability to reach the end point. In the pre-training stage, the parameter α in the reward function is set to 0, indicating that only the primary goal is optimized and the secondary goal is ignored. In addition, the arrival detection function is also disabled in the transfer preprocessing to explore possible paths to the end point. The other settings are exactly the same as the training shown in Algorithm 1. After the pre-training stage, α is gradually increased to optimize the secondary goal. Figure 2 As shown in Figure 2. The agent simply executes the action with the highest Q value step by step until it reaches the end. Although the transfer preprocessing is indeed effective in the training phase, there are still a small number of OD pairs that will cause the agent to enter a cycle path. In this case, the ∈-greedy algorithm is performed on these OD pairs again to help the agent jump out of the cycle path.

[0086] S3. Use the trained dual-objective path planning model to perform dual-objective path planning.

[0087] The present invention designs a state representation method to capture the state of the agent in the environment, which not only includes the spatial state of the current position, but also provides key information of the indicator to be optimized; in this way, the path distance and other optimization indicators (loss or utility) are represented in a unified manner. This comprehensive state can promote the effective training of deep reinforcement learning path planning strategies. In addition, the present invention also designs an effective reward function, in which loss and utility can be represented in a unified manner. Unlike the existing path planning methods for specific scenarios, this method is a general method that can be applied to various dual-objective path planning problems by utilizing deep reinforcement learning, minimizing loss and maximizing utility while minimizing path distance. The dual-objective path planning model after training in the present invention has the versatility to optimize loss and utility in different scenarios while having the ability to minimize path distance.

[0088] The reward function designed by the present invention takes into account both the main goal reward and the secondary goal reward, and can achieve a balance between the shortest distance and other optimization goals. In addition to the rewards for loss and utility, a new concept of effective distance is introduced to establish a trade-off mechanism between the two goals. Thereby ensuring the effectiveness and applicability of the reward function. During the training phase, the present invention also includes transition preprocessing, which preprocesses the transitions (Transition) input into the network, and the transition preprocessing includes arrival detection and reward correction. The purpose of arrival detection is to identify actions that can ultimately lead the agent to the end point. The reward correction function penalizes actions that produce cyclic paths, thereby preventing the agent from being trapped in a loop. Through these two functions, training can be made more efficient.

[0089] In order to verify the performance of the trained dual-objective path planning model in the present invention, the applicant conducted the following experiment.

[0090] Experimental setup

[0091] Data description: The experiment is conducted with two representative cu (solar radiation and crime risk) in the road network of a city N. A road network with 998 nodes and 3326 edges was crawled from the crowdsourcing platform OpenStreetMap. The specific calculation method of the cu value is as follows:

[0092] Solar radiation. The concept of solar radiation here refers to the horizontal radiation intensity. The calculation of solar radiation intensity adopts the method proposed in, which takes into account the obstruction of human facilities on both sides of the street. At the same time, since human facilities such as trees and buildings on both sides of different roads in the city will block solar radiation to varying degrees, the solar radiation on different roads will also be different. In addition, the height, density and distribution direction of human facilities are also different. Therefore, the solar radiation between roads lacks spatial proximity. In the experiment, the weight of cu used the average solar radiation intensity within one hour from 11 o'clock to 12 o'clock. The spatial distribution of this average solar radiation in the road network is as follows Figure 5 As shown in the schematic diagram on the left, it can be seen that for the solar radiation attribute, each edge in the road network has a certain weight value.

[0093] Crime risk. Crime risk is a statistical method for analyzing historical crime data near a road section to indicate the degree of road insecurity. The crime risk of a certain edge is weighted and summed using the measurement method used in the previous article, as shown in the following formula:

[0094]

[0095] In the formula, n is the number of historical crime reports within a circle with the length of the road section as its diameter; c j is the jth crime report, x mis the midpoint of the road section; distance(c j ,x m ) Calculate c j and x m The Euclidean distance between them; l represents the length of the road segment. The crime data comes from the aforementioned city N open data, including crime time, coordinates, type, etc. Since this example focuses more on street-level safety, crimes such as assault, robbery, and use of dangerous weapons are used. Some edges have no history of crime, and the crime risk of these edges is 0. Therefore, the distribution of crime risk weights in the road network is relatively sparse, such as Figure 5 The diagram on the right shows that only a portion of the edges have different levels of crime risk compared to the solar radiation attribute.

[0096] Baseline algorithms and evaluation metrics: Since both distance and CU are considered in the path planning objective, they are used as evaluation metrics. An algorithm is considered to have better performance if it generates a shorter path distance and generates a larger CU when CU needs to be maximized and a smaller CU when CU needs to be minimized. For intuitive comparison, the applicant also tested some baseline algorithms, as shown below.

[0097] Shortest Path. It only considers distance when planning a path, and uses the classic Dijkstra algorithm to generate the shortest path. It is the upper bound of distance optimization.

[0098] Least cu. It only considers minimizing cu, which is only applicable when cu is used as loss. It is the upper bound of minimizing cu.

[0099] SPTH. This is the path planning algorithm proposed above, which takes into account both the scenery value and quietness along the way. The core ideas of this algorithm include: 1) generating the top-k shortest paths; 2) selecting the path with the highest second target score among the k paths. In the experiment of this embodiment, SPTH is used to find the path with the smallest cu.

[0100] 2TD-AOP. Its goal is to find the most scenic path along the road network. The attributes of the edges (scenic value) are sparse and change over time. It is only used as a comparison when maximizing the crime risk. In addition, when running 2TD-AOP, the cu weights can be simply set to the same at different times. In other words, the time-dependent characteristics of edge weights are ignored in the calculation.

[0101] Reversed cu. First, according to Reverse the value of cu and then use the Least cu algorithm to find cu revMinimum path. In this embodiment, the algorithm is set to verify whether the edge attribute minimization algorithm can be transformed into the edge attribute maximization algorithm by reversing the weight of the edge attribute.

[0102] For different application scenarios, the dimensions of the evaluation results are different. In order to compare the experimental results more comprehensively, the ratio of the path distance d(τ) and path cu cu(τ) of each algorithm to the path distance and path cu of the Shortest algorithm is used to measure its performance. In other words, the path distance and path cu obtained by the Shortest algorithm are used as references. The average ratio of all test OD pairs is used as the evaluation index, which is called the distance ratio and cu ratio, respectively, and is defined as follows:

[0103]

[0104]

[0105] Where P is the set of all tested OD pairs; τ i is the path obtained by the test algorithm for the i-th OD pair; It refers to the path of the i-th OD pair obtained by the Shortest algorithm. Obviously, the path distance and cu ratio of the Shortest algorithm are both equal to 1. In addition, the closer the distance ratio is to 1 and the farther the cu ratio is from 1, the better the algorithm performance is.

[0106] According to the distribution of cu and the characteristics of the baseline algorithm, when minimizing cu (solar radiation and crime risk), this embodiment selects Shortest, Least cu and SPTH as baselines; when maximizing solar radiation, Shortest and Reversedcu are selected as baselines; when maximizing crime risk, Shortest, Reversed cu and 2TD-AOP are selected as baselines.

[0107] Evaluation Environment and Parameter Settings: All evaluation experiments are programmed in Python 3.7 with TensorFlow-1.4 and Keras-2.3 and run on a PC with 4 NVIDIA GeForce RTX 2080Ti GPUs and 192GB RAM.

[0108] A total of 5000 pairs of ODs with random start and end points were generated in the road network (i.e. |P|=5000). In the training phase, the number of cycles N=50000, the maximum number of steps in a cycle T=100, the range of ∈ during training is 0.5~0, and the discount factor γ=0.96. In the DDQN network, the experience recycling pool size |M|=200000, the mini-batch size M batc=64, the neural networks of the Q network and the target network are implemented with fully connected layers, the hidden layer sizes are 1024, 512, and 128 respectively, and the synchronization frequency of the target network is Δ = 300. During the learning process of the neural network, the learning rate lr = 0.00001.

[0109] Overall Results

[0110] The evaluation results of different application scenarios are summarized, and the overall results are as follows: Figure 6 As shown. While minimizing solar radiation and crime risk, Figure 6 (a) and Figure 6 As shown in (b), cuRL is generally better able to balance path cu and path distance compared to the baseline algorithms. Not surprisingly, Shortest performs best on distance ratio, while Least cu obtains the best cu ratio, since their goal is to optimize only distance or cu. When minimizing solar radiation, cuRL is able to generate paths with a distance ratio of 1.04 and a cu ratio of 0.95, while Least cu generates paths with a distance ratio of 1.11 and a cu ratio of 0.86. A more unexpected result is that SPTH slightly outperforms cuRL in both distance ratio and cu ratio, with values ​​of 1.01 and 0.94, respectively. A possible explanation for this phenomenon is that the top-k shortest paths obtained by SPTH happen to contain the path with the smallest cu. To verify this conclusion, we further give Figure 6 (b) shows the result of minimizing crime risk, while the distribution of crime risk attributes in the road network is very uneven and irregular. It can be seen that the path distance ratio and cu ratio obtained by SPTH are 1.03 and 0.67 respectively, while the cu ratio performance of cuRL is much better, with distance ratio and cu ratio of 1.08 and 0.34 respectively.

[0111] like Figure 6 (c) and Figure 6 As shown in the results shown in (d), when maximizing solar radiation and crime risk, cuRL still outperforms other baseline algorithms in most cases. As mentioned above, the results of 2TD-AOP are only compared when maximizing crime risk. More specifically, when maximizing solar radiation, cuRL has a distance ratio of 1.09 and a cu ratio of 1.13, which is much better than Reversed cu in terms of distance ratio and slightly worse in terms of cu ratio. This is because Reversed cu is proposed to maximize cu, while ignoring the path distance optimization. In this example, we further investigate whether Reversed cu can still obtain a better cu ratio than cuRL when maximizing crime risk. The results are shown in Figure 2. Figure 6(d). It can be seen that the cu ratio of the path generated by Reversed cu is 1.16, which is the worst performance compared with 2TD-AOP and cuRL. It can be concluded that it is not always feasible to transform the minimization problem into a maximization problem by simply reversing the weights during path planning. In terms of cu ratio, 2TD-AOP has the best performance of 1.27, but it does not achieve satisfactory results in terms of distance ratio. What's worse, it is only applicable to cases where edge attributes are sparse. In contrast, cuRL can obtain the best path for both objectives because it optimizes both distance and cu at the same time.

[0112] In summary, although cuRL is sometimes inferior to some comparative algorithms, it has greater versatility because it can minimize loss or maximize utility while keeping path distances short, and can work stably in road networks with dense or sparse edge attributes.

[0113] Verification of the effectiveness of reward design

[0114] In order to verify the effectiveness of the reward design in cuRL, this example takes the optimization of crime risk as an example and examines the performance of the model under different α during the training process. Figure 7 It can be seen that when α changes, the distance ratio fluctuates within a certain range and reaches a minimum value of 1.04 when α = 0, indicating that the pre-trained model is effective and the path distance obtained is very similar to the shortest distance algorithm. In this case, the model can be regarded as a model for finding the shortest path because only the main goal is included in the reward function. In addition, the model with α < 0 corresponds to cu minimization, and the model with α > 0 corresponds to cu maximization. The specific analysis is as follows.

[0115] In the maximum crime risk scenario, Figure 7 As shown in the right half, both the distance and the cu ratio gradually increase with the increase of α. Compared with the cu ratio, the fluctuation of the distance ratio is smaller, indicating that the realization of cu maximization does not require too much additional loss in path distance. When α is greater than 0.5, the cu ratio stops growing. Even worse, the distance ratio increases instead. One possible explanation is that the agent tries to get more r cu , but does not move to the end point, resulting in unnecessary detours during path planning. When the α value is 0.5, the maximization ability of cu reaches its limit. Therefore, it is meaningless to continue to increase α, and the training can be terminated in this case. Similar conclusions can be drawn for the crime risk minimization scenario. The results are as follows Figure 7The left half is shown. In this case, the cu ratio is always less than 1. Similarly, when α is a different negative value, the distance ratio fluctuates within a smaller range, indicating that the minimization of the path cu is also completed at the cost of increasing the path distance less. When the α value is -2, the model reaches the limit of minimizing cu.

[0116] Validity of state representation

[0117] In the cuRL algorithm framework, the state representation is carefully designed to effectively learn and optimize the primary and secondary objectives. As described in Chapter 4, the state consists of three parts: coordinate state, distance state, and cu state. In order to evaluate the effectiveness of each state type, an ablation study was conducted. In addition, the performance of the algorithm using latitude and longitude coordinates and one-hot encoding as state representation was further compared.

[0118] In summary, the status representation methods for all comparisons are as follows.

[0119] Coord. The status indicates that only the longitude and latitude coordinates are included.

[0120] Onehot. The state is represented by only one-hot encoding.

[0121] cuRL-coord. The state representation includes the distance state and the cu state.

[0122] cuRL-dis. The state representation includes the coordinate state and the cu state.

[0123] cuRL-cu. The state representation includes coordinate state and distance state.

[0124] The scenario of minimizing solar radiation is used to evaluate different state representation methods. The overall results on distance ratio and cu ratio are shown in Table 1. It should be noted that all the results are obtained after the parameter α in the reward function is carefully adjusted. Based on the results shown in Table 1, the following conclusions can be drawn:

[0125] Coord can neither optimize the path distance nor the path cu. As shown in Table 1. The distance ratio and cu ratio of Coord are both greater than 1. This is because Coord only considers the longitude and latitude coordinates and does not optimize the path cu. In addition, in terms of distance optimization, Onehot and cuRL both perform better than Coord, which verifies the applicant's previous analysis that the longitude and latitude coordinates are not enough for the agent to optimize the path distance.

[0126] Compared with Coord, Onehot achieves better results. Although Onehot does not explicitly encode cu and coordinate information in the state representation, it can still distinguish nodes well. Specifically, it does not need to learn any deep knowledge of path planning to remember the best action at each point. It performs better than Coord and can optimize cu. However, due to the influence of the input dimension, Onehot has a very large amount of calculation and converges slowly, and cannot learn efficiently. In more detail, the dimension of the state vector represented by Onehot is very high, reaching 1996, while the dimension of Coord is only 4.

[0127] Both the coordinate state and the distance state are essential. On the one hand, without considering the distance state, the distance ratio of cuRL-dis is greater than 1; on the other hand, the cu ratio of cuRL-coord is also greater than 1, indicating that the path cu is not actually optimized.

[0128] The cu state can effectively improve the optimization of cu. Comparing the results of cuRL-cu and cuRL, it can be easily found that the distance ratios they obtained are similar (i.e. 1.033), but the cu ratios of the two are quite different. The cu ratio of cuRL-cu is greater than 1, and the solar radiation is not optimized. In contrast, the cu ratio of cuRL is much smaller, achieving a better optimization effect.

[0129] Table 1 Evaluation results of different state representation methods

[0130]

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit the technical solution. Those skilled in the art should understand that those modifications or equivalent substitutions of the technical solution of the present invention that do not depart from the purpose and scope of the technical solution should be included in the scope of the claims of the present invention.

Claims

1. A dual-objective path planning method based on deep reinforcement learning, characterized in that: The following steps are involved: S1. Represent the state of the road network and construct a reward function r to obtain a dual-objective deep reinforcement learning path planning model to be trained; the state includes a coordinate state, a distance state and a cu state; The coordinate state includes the latitude and longitude coordinates of the starting point and the latitude and longitude coordinates of the end point; the distance state is the shortest distance from all adjacent nodes of the current node to the end point; the cu is the edge attribute except the path distance and travel time; the cu state is represented by the weight of the cu of all edges connected to the current node, which is used to represent the state of the second optimization goal and help the agent learn the spatial characteristics of cu; The second optimization goal is to minimize loss or maximize utility; the reward function r is used to calculate the comprehensive reward of the main goal and the second goal in the current step; the main goal is to minimize the path distance; S2: Train the dual-objective path planning model to learn the optimal strategy π that can obtain the maximum cumulative reward * , obtaining a trained dual-objective path planning model; the cumulative reward is the sum of the immediate reward given by the reward function and the additional future reward estimated by the neural network in the dual-objective path planning model; S3, using the trained dual-objective path planning model to perform dual-objective path planning; Among them, in S1, the reward function Among them, r dest represents the main target reward, r cu represents the second target reward, the main target reward is the reward for minimizing the path distance, and the second target reward is the reward for optimizing the path cu; α is a preset hyperparameter for weighing the main target and the second target, when cu represents utility, α is positive, and when cu represents loss, α is negative; is 0 or 1, A value of 0 indicates that the agent has not reached the end point. 1 means the agent has reached the end point; r T For additional future rewards.

2. The dual-objective path planning method based on deep reinforcement learning according to claim 1, characterized in that: In S1, the main target reward r dest =norm -1,1 (dis eff ), where dis eff Indicates the valid distance of the path.

3. The dual-objective path planning method based on deep reinforcement learning as claimed in claim 2, characterized in that: dis eff =(|N o N d |-|A1N d |)-|A1N a |, where |N o N d |-|A1N d |For the intelligent agent Distance moved in the direction; |A1N a | for deviation Distance in direction; N o As the starting point; N d A1 is the end point N d As the coordinate origin, The direction is the x-axis, the normal direction After establishing the coordinate system for the y-axis, from N a The foot point of a perpendicular line drawn from the point to the x-axis; N a Point is the starting point N o A neighboring node of 4. The dual-objective path planning method based on deep reinforcement learning as claimed in claim 3, characterized in that: In S1, r cu =norm 0,1 (cu a ), where cu a is the cu weight on the edge corresponding to the action of the agent.

5. The dual-objective path planning method based on deep reinforcement learning according to claim 4, characterized in that: In S2, the cumulative reward Among them, t is the current step; T is the termination step; r t represents the immediate reward after the agent executes the current step according to the corresponding strategy; γ is the discount factor and γ∈[0,1], which is used to measure the importance of the reward generated by the future steps.

6. The dual-objective path planning method based on deep reinforcement learning as claimed in claim 5, characterized in that: In S2, when training the dual-target path planning model to be trained, a dual-depth Q network is used for training, and the dual-depth Q network includes a Q network and a target network, and action selection and evaluation are decoupled through the Q network and the target network; the Q network and the target network are both neural networks.

7. The dual-objective path planning method based on deep reinforcement learning according to claim 6, characterized in that: In S2, when the dual-objective path planning model to be trained is trained, it also includes transfer preprocessing, which is the transfer input into the neural network of the dual-objective path planning model; the transfer is the experience corresponding to the action after the action is completed.

8. The dual-objective path planning method based on deep reinforcement learning according to claim 7, characterized in that: The preprocessing includes arrival detection and reward correction; arrival detection is used to identify actions that can ultimately lead the agent to the end point; reward correction is used to punish actions that produce a cyclic path to prevent the agent from being trapped in a cycle.

9. The dual-objective path planning method based on deep reinforcement learning as claimed in claim 8, characterized in that: In S2, when training the dual-objective path planning model to be trained, the Q value Q π (s,a), represents the quality of taking action a given state s and strategy π; Among them, E π [·] Compute the expected value of strategy π; And will find the optimal strategy π * The problem is transformed into finding the optimal Q value Q * (s,a) problem; a * refers to the optimal action in state s; based on the Bellman equation,

Images