Reinforcement learning strategy iterative path planning method based on dynamic planning
By combining dynamic programming and reinforcement learning in path planning, we realize the method of efficiently finding global optimal paths in large-scale maps and dynamic environments, and solve the problem of low computational efficiency and easy to fall into local optimality.
Patent Information
- Application Number
- CN202411549306.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-01
- Publication Date
- 2025-02-21
AI Technical Summary
When existing path planning algorithms deal with large-scale maps or dynamic environments, they have low computational efficiency, difficulty finding global optimal paths, and are prone to falling into local optimal.
The iterative path planning method of reinforcement learning strategy based on dynamic programming is adopted, and the path evaluation is performed through dynamic programming algorithms to improve the computing efficiency, and the strategy iteration is used to find the global optimal path.
It improves the computational efficiency of path planning, can quickly find shorter paths on complex maps, avoid local optimal solutions, and adapt to dynamic environment changes.
Smart Images
Figure CN119493422A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of artificial intelligence and relates to a reinforcement learning strategy iterative path planning method based on dynamic programming. Background Art
[0002] Path planning is one of the key technologies in the fields of autonomous driving, drones, and robots, and aims to find the optimal path from the starting point to the end point. Existing path planning algorithms are mainly divided into two categories:
[0003] Algorithms based on graph search: such as Dijkstra algorithm, A* algorithm, etc. This type of algorithm builds a graph model of the map and uses heuristic information to search to find the shortest path from the starting point to the end point. Its advantages are simple algorithm principles and easy implementation, but when dealing with large-scale maps or dynamic environments, the computational efficiency is low and it is difficult to find the global optimal path.
[0004] Algorithms based on reinforcement learning: such as Q-learning, Sarsa, DQN, etc. These algorithms learn the optimal path planning strategy by interacting with the environment. Their advantages are that they can adapt to dynamic environments and find the global optimal path, but they require a large amount of training data, and the training process is complex and difficult to converge.
[0005] Existing path planning algorithms have the following limitations:
[0006] As the size of the map increases, the calculation time of graph search-based algorithms increases exponentially or even exponentially, making it difficult to meet real-time requirements.
[0007] Graph search-based algorithms are prone to fall into local optimality and find it difficult to find the global optimal path.
[0008] Algorithms based on reinforcement learning require a large amount of training data, and the training process is complex and difficult to converge.
[0009] Existing algorithms have low computational efficiency for dynamic environments, such as the dynamic appearance of obstacles and changes in traffic flow. Summary of the invention
[0010] In view of this, the object of the present invention is to provide a reinforcement learning strategy iterative path planning method based on dynamic programming. The dynamic programming algorithm is used to perform path evaluation to improve the computational efficiency of path planning. The reinforcement learning algorithm is used to perform strategy iteration to find the global optimal path. The dynamic programming algorithm can adapt to dynamic environments, such as the dynamic appearance of obstacles, changes in traffic flow, etc.
[0011] In order to achieve the above object, the present invention provides the following technical solutions:
[0012] A reinforcement learning strategy iterative path planning method based on dynamic programming, the method comprising the following steps:
[0013] S1: Input map data into the map construction module of the DRLP algorithm. The input map data includes the path Node points and the relationship between Node points.
[0014] S2: Reconstruct map elements, build state transfer matrix according to map Node information, design cost function, and initialize state transfer probability;
[0015] S3: Input the target point location information into the DRLP algorithm path planning module, and obtain the Node point selection strategy π after strategy iteration and strategy evaluation;
[0016] S4: Insert path replanning switch T into the state transfer matrix re-compute , T re-compute The default value is True;
[0017] S5: Input the starting point location information into the DRLP algorithm path planning module, and obtain the initial global planning path according to the strategy π;
[0018] S6: When path planning is required, T re-compute Set to False to update the current location information to the DRLP algorithm path planning module and use the π strategy for replanning.
[0019] Furthermore, in S1, the specific method for obtaining the location relationship information of the Node points on the map path is:
[0020] S1.1: Connect to map data, parse and obtain map data format;
[0021] S1.2: Obtain the data format of the Node path point, determine and record the connection method of adjacent path points and the data expression format corresponding to the connection form;
[0022] Further, in S2, constructing a state transfer matrix and initializing the state transfer probability comprises the following steps:
[0023] S2.1: According to the data expression format corresponding to the connection mode of the adjacent Node path points and the connection form, the initial probability transfer matrix is established according to the data format: if the current Node point has adjacent points, the current point and its adjacent points are connected, and the initial probability value of the current point n transferring to the adjacent point n' is π(n, n') = 1 / the number of adjacent Node points; if the current Node point has no adjacent points or the adjacent points are impassable, then π(n, n') = 0;
[0024] S2.2: Establish the cost function (reward function) r(n, n') from Node n to the next point n'. The cost function is defined by the following formula:
[0025]
[0026] Among them, n target represents the end point, H(·) represents the distance evaluation function, which is calculated by Euclidean distance, C1 and C2 represent constant values, n′=target represents reaching the target point, and n′=obstarct or collision represents encountering an obstacle or a collision.
[0027] S2.3: Construct the initial state transfer matrix. The state value of a single node is defined as follows:
[0028] s n =[n′,r(n,n'),done]
[0029] Among them, s n is the state value of the current Node point n, done indicates whether the current state is the final stop state. If the next Node point n' is the end point or is inaccessible, done is True, otherwise done is False.
[0030] Further, in S3, the DRLP algorithm path planning module comprises the following steps:
[0031] S3.1: Initialize the strategy function π(n) and the value function V(n);
[0032] S3.2: Strategy evaluation;
[0033] S3.3: If Δ>θ, return to S3.2; otherwise, proceed to the next step, where θ is the policy iteration threshold;
[0034] That is, after obtaining the state value function of the current strategy, we can improve the strategy based on it, stating that in any state s, there exists:
[0035] V π′ (s)>V π (s)
[0036] S3.4: Strategy improvement.
[0037] Further, in S3.2, the strategy evaluation comprises the following steps:
[0038] Assign Δ to 0;
[0039] For each state s∈S, assign V(s) to v in turn, and assign r(s,π(s))+γ∑ s'Assign P(s'|s,π(s))V(s') to V(s) and assign max(Δ,|vV(s)|) to Δ.
[0040] Among them, the calculation principle of V(s) is:
[0041] According to the Bellman expectation equation, the value function of the state can be expressed as follows:
[0042]
[0043] Where π(a|s) represents the probability of selecting action a in state s according to strategy π, and r(a|s) represents the reward that the agent can obtain by selecting action a in state s. γ is the discount factor. P(·) is the state transition matrix. Combining the idea of dynamic programming, we can regard the calculation of the value of the next possible state as a sub-problem, and the calculation of the value of the current state as the current problem. After knowing the solution of the sub-problem, we can solve the current problem.
[0044] Further, in S3.4, the strategy enhancement comprises the following steps:
[0045] Assign the current strategy π to the old strategy π old ;
[0046] For each state s∈S, selecting action a according to strategy π′ will get a higher expected return than the original strategy π. At each state, the action with the largest action value is selected, that is:
[0047]
[0048] Will Assign value to the current strategy π(s);
[0049] If π old =π, the strategies π and π old It is called the optimal strategy, then the value function V and strategy π are returned and the calculation ends, otherwise it returns to S3.2.
[0050] Further, in S4, the path replanning switch T re-compute is defined as:
[0051] T re-compute It is Boolean data. When its value is True, it means that policy evaluation and policy improvement operations are needed to obtain a new policy π for global path planning. When its value is False, it means that the previous policy π is directly used for global path planning.
[0052] Further, in S5, the specific steps of obtaining the initial global planning path according to strategy π are:
[0053] S5.1: Input the starting point n and create an empty path set path = [];
[0054] S5.2: Select an adjacent path Node n' by argmax{π(n, n'), n'∈{n's adjacent Node set}};
[0055] S5.3: Let n = n', add the path point n to the path set;
[0056] S5.4: If n = n target , then return to the planned path path, otherwise return to S5.2.
[0057] The beneficial effects of the present invention are:
[0058] (1) By performing path evaluation through the dynamic programming algorithm, the DRLP algorithm can quickly find a relatively short path on a complex map, effectively improving the computational efficiency of path planning.
[0059] (2) By using the reinforcement learning algorithm to iterate the strategy, the DRLP algorithm can continuously optimize the path selection strategy and eventually find the global optimal path to avoid falling into the local optimal solution.
[0060] (3) The dynamic programming algorithm can update the state transfer matrix and cost function in real time, adapt to dynamic environments such as the dynamic appearance of obstacles and changes in traffic flow, and ensure the effectiveness of path planning.
[0061] (4) The DRLP algorithm has a simple principle and is easy to implement. It can be applied to different scenarios such as autonomous driving, drones, and robots, and has wide application value.
[0062] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0064] Figure 1 Comparison of the number of nodes included in each planned path for Node 39 of the CarlaTown04 map;
[0065] Figure 2 Comparison of the number of nodes included in each planned path for Node 102 in the CarlaTown04 map;
[0066] Figure 3 Comparison of the number of nodes included in each planned path for Node 161 of the CarlaTown04 map;
[0067] Figure 4 Comparison of the number of nodes included in each planned path for Node 171 in the CarlaTown04 map;
[0068] Figure 5 Comparison of the time spent on each planning of Node 39 in CarlaTown04 map;
[0069] Figure 6 Comparison of the time spent on each planning of Node 102 in CarlaTown04 map;
[0070] Figure 7 Comparison of the time spent on each planning of Node 161 in CarlaTown04 map;
[0071] Figure 8 Comparison of the time spent on planning each time for Node 171 in the CarlaTown04 map. DETAILED DESCRIPTION
[0072] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.
[0073] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.
[0074] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0075] The path planning process of the dynamic-programming-based reinforcement learning based on policy iteration (DRLP) is as follows:
[0076] S1: Input map data into the map construction module of the DRLP algorithm. The input map data includes the path Node points and the relationship between Node points.
[0077] S2: Reconstruct map elements, build state transfer matrix according to map Node information, design cost function, and initialize state transfer probability;
[0078] S3: Input the target point location information into the DRLP algorithm path planning module, and obtain the Node point selection strategy π after strategy iteration and strategy evaluation;
[0079] S4: Insert path replanning switch T into the state transfer matrix re-compute , T re-compute The default value is True;
[0080] S5: Input the starting point location information into the DRLP algorithm path planning module, and obtain the initial global planning path according to the strategy π;
[0081] S6: When path planning is required, T re-compute Set to False to update the current location information to the DRLP algorithm path planning module and use the π strategy for replanning.
[0082] Furthermore, in S1, the specific method for obtaining the location relationship information of the Node points on the map path is:
[0083] S1.1: Connect to map data, parse and obtain map data format;
[0084] S1.2: Obtain the data format of the Node path point, determine and record the connection method of adjacent path points and the data expression format corresponding to the connection form;
[0085] Further, in S2, constructing a state transfer matrix and initializing the state transfer probability comprises the following steps:
[0086] S2.1: According to the data expression format corresponding to the connection mode of the adjacent Node path points and the connection form, the initial probability transfer matrix is established according to the data format: if the current Node point has adjacent points, the current point and its adjacent points are connected, and the initial probability value of the current point n transferring to the adjacent point n' is π(n, n') = 1 / the number of adjacent Node points; if the current Node point has no adjacent points or the adjacent points are impassable, then π(n, n') = 0;
[0087] S2.2: Establish the cost function (reward function) r(n, n') from Node n to the next point n'. The cost function is defined by the following formula:
[0088]
[0089] Among them, n target represents the end point, H(·) represents the distance evaluation function, which is calculated by Euclidean distance, C1 and C2 represent constant values, n′=target represents reaching the target point, and n′=obstarct or collision represents encountering an obstacle or a collision.
[0090] S2.3: Construct the initial state transfer matrix. The state value of a single node is defined as follows:
[0091] s n =[n′,r(n,n'),done]
[0092] Among them, s n is the state value of the current Node point n, done indicates whether the current state is the final stop state. If the next Node point n' is the end point or is inaccessible, done is True, otherwise done is False.
[0093] Further, in S3, the DRLP algorithm path planning module comprises the following steps:
[0094] S3.1: Initialize the strategy function π(n) and the value function V(n);
[0095] S3.2: Strategy evaluation;
[0096] S3.3: If Δ>θ, return to S3.2; otherwise, proceed to the next step, where θ is the policy iteration threshold;
[0097] That is, after obtaining the state value function of the current strategy, we can improve the strategy based on it, stating that in any state s, there exists:
[0098] V π′ (s)>V π (s)
[0099] S3.4: Strategy improvement.
[0100] Further, in S3.2, the strategy evaluation comprises the following steps:
[0101] Assign Δ to 0;
[0102] For each state s∈S, assign V(s) to v in turn, and assign r(s,π(s))+γ∑ s' Assign P(s'|s,π(s))V(s') to V(s), and assign max(Δ,|vV(s)|) to Δ.
[0103] Among them, the calculation principle of V(s) is:
[0104] According to the Bellman expectation equation, the value function of the state can be expressed as follows:
[0105]
[0106] Where π(a|s) represents the probability of selecting action a in state s according to strategy π, and r(a|s) represents the reward that the agent can obtain by selecting action a in state s. γ is the discount factor. P(·) is the state transition matrix. Combining the idea of dynamic programming, we can regard the calculation of the value of the next possible state as a sub-problem, and the calculation of the value of the current state as the current problem. After knowing the solution of the sub-problem, we can solve the current problem.
[0107] Further, in S3.4, the strategy enhancement comprises the following steps:
[0108] Assign the current strategy π to the old strategy π old ;
[0109] For each state s∈S, selecting action a according to strategy π′ will get a higher expected return than the original strategy π. At each state, the action with the largest action value is selected, that is:
[0110]
[0111] Will Assign value to the current strategy π(s);
[0112] If πold =π, the strategies π and π old It is called the optimal strategy, then the value function V and strategy π are returned and the calculation ends, otherwise it returns to S3.2.
[0113] Further, in S4, the path replanning switch T re-compute is defined as:
[0114] T re-compute It is Boolean data. When its value is True, it means that policy evaluation and policy improvement operations are needed to obtain a new policy π for global path planning. When its value is False, it means that the previous policy π is directly used for global path planning.
[0115] Further, in S5, the specific steps of obtaining the initial global planning path according to strategy π are:
[0116] S5.1: Input the starting point n and create an empty path set path = [];
[0117] S5.2: Select an adjacent path Node n' by argmax{π(n, n'), n'∈{n's adjacent Node set}};
[0118] S5.3: Let n = n', add the path point n to the path set;
[0119] S5.4: If n = n target , then return to the planned path path, otherwise return to S5.2.
[0120] Experimental content:
[0121] Conduct a path planning experiment in the Carla map, select 4 Node path points as the end points, select any starting point, and require that the Euclidean distance between the starting point and the end point is greater than 100m.
[0122] Experimental content: Comparison with other algorithms
[0123] 1. Compare the number of nodes included in the paths planned in each round. Taking the DRLP algorithm as the benchmark, for example, the value displayed in the figure for the A* algorithm = the number of nodes calculated by the A* algorithm / the number of nodes calculated by DRLP. DRLP is used as the benchmark, and its displayed value = 1.
[0124] The experimental results show that the path planned by DRLP contains fewer nodes and has a shorter theoretical path length, such as Figure 1 to Figure 4 shown.
[0125] 2. Compare the time spent in each round of planning, using the DRLP algorithm as the benchmark.
[0126] Experimental results: Except for the first round of planning, the planning time of DRLP is shorter in the remaining experimental results, such as Figures 5 to 8 shown.
[0127] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. A reinforcement learning strategy iterative path planning method based on dynamic programming, characterized by: The method comprises the following steps: S1: Input map data into the map construction module of the DRLP algorithm. The input map data includes the path Node points and the relationship between Node points. S2: Reconstruct map elements, build state transfer matrix according to map Node information, design cost function, and initialize state transfer probability; S3: Input the target point location information into the DRLP algorithm path planning module, and obtain the Node point selection strategy π after strategy iteration and strategy evaluation; S4: Insert path replanning switch T into the state transfer matrix re-compute , T re-compute The default value is True; S5: Input the starting point location information into the DRLP algorithm path planning module, and obtain the initial global planning path according to the strategy π; S6: When path planning is required, T re-compute Set to False to update the current location information to the DRLP algorithm path planning module and use the π strategy for replanning.
2. The method for iterative path planning based on dynamic programming of reinforcement learning strategy according to claim 1, characterized in that: In S1, the specific method for obtaining the location relationship information of the Node points on the map path is: S1.1: Connect to map data, parse and obtain map data format; S1.2: Obtain the data format of the Node path point, determine and record the connection method of adjacent path points and the data expression format corresponding to their connection form.
3. The method for iterative path planning based on dynamic programming of reinforcement learning strategy according to claim 1, characterized in that: In S2, constructing a state transfer matrix and initializing the state transfer probability comprises the following steps: S2.1: According to the data expression format corresponding to the connection mode of the adjacent Node path points and the connection form, the initial probability transfer matrix is established according to the data format: if the current Node point has adjacent points, the current point and its adjacent points are connected, and the initial probability value of the current point n transferring to the adjacent point n' is π(n, n') = 1 / the number of adjacent Node points; if the current Node point has no adjacent points or the adjacent points are impassable, then π(n, n') = 0; S2.2: Establish the cost function from Node n to the next point n', that is, the reward function r(n, n'). The cost function is defined by the following formula: Among them, n target represents the end point, H(·) represents the distance evaluation function, which is calculated by Euclidean distance, C1 and C2 represent constant values, n′=target represents reaching the target point, and n′=obstarct or collision represents encountering an obstacle or collision; S2.3: Construct the initial state transfer matrix. The state value of a single node is defined as follows: s n =[n′,r(n,n'),done] Among them, s n is the state value of the current Node point n, done indicates whether the current state is the final stop state. If the next Node point n' is the end point or is inaccessible, done is True, otherwise done is False.
4. The method for iterative path planning based on dynamic programming of reinforcement learning strategy according to claim 1, characterized in that: In S3, the DRLP algorithm path planning module includes the following steps: S3.1: Initialize the strategy function π(n) and the value function V(n); S3.2: Strategy evaluation; S3.3: If Δ>θ, return to S3.2; otherwise, proceed to the next step, where θ is the policy iteration threshold; That is, after obtaining the state value function of the current strategy, the strategy is improved based on it, and the statement exists in any state s: V π′ (s)>V π (s) S3.4: Strategy improvement.
5. The DRLP algorithm path planning module according to claim 4, characterized in that: In S3.2, the strategy evaluation comprises the following steps: Assign Δ to 0; For each state s∈S, assign V(s) to v in turn, and assign r(s,π(s))+γΣ s' Assign P(s's,π(s))V(s') to V(s), and assign max(Δ,|vV(s)) to Δ; Among them, the calculation principle of V(s) is: According to the Bellman expectation equation, the value function of the state is expressed as follows: Where π(a|s) represents the probability of selecting action a in state s according to strategy π, r(a|s) represents the reward that the agent can obtain by selecting action a in state s; γ is the discount factor; P(·) is the state transition matrix; combined with the idea of dynamic programming, the calculation of the value of the next possible state is regarded as a subproblem, and the calculation of the value of the current state is regarded as the current problem. After knowing the solution of the subproblem, the current problem is solved.
6. The DRLP algorithm path planning module according to claim 4, characterized in that: In S3.4, the strategy enhancement includes the following steps: Assign the current strategy π to the old strategy π old ; For each state s∈S, selecting action a according to strategy π′ will get a higher expected return than the original strategy π. At each state, the action with the largest action value is selected, that is: Will Assign value to the current strategy π(s); If π old =π, the strategies π and π old It is called the optimal strategy, then the value function V and strategy π are returned and the calculation ends, otherwise it returns to S3.
2.
7. The method for iterative path planning based on dynamic programming of reinforcement learning strategy according to claim 1, characterized in that: In S4, the path replanning switch T re-compute is defined as: T re-compute It is Boolean data. When its value is True, it means that policy evaluation and policy improvement operations are needed to obtain a new policy π for global path planning. When its value is False, it means that the previous policy π is directly used for global path planning.
8. The method for iterative path planning based on dynamic programming of reinforcement learning strategy according to claim 1, characterized in that: In S5, the specific steps of obtaining the initial global planning path according to strategy π are: S5.1: Input the starting point n and create an empty path set path = []; S5.2: Select an adjacent path Node n' by argmax{π(n, n'), n'∈{n's adjacent Node set}}; S5.3: Let n = n', add the path point n to the path set; S5.4: If n = n target , then return to the planned path path, otherwise return to S5.2.