A path planning method based on dynamic exploration and heuristic initialization
Patent Information
- Application Number
- CN202610685611.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-19
- Publication Date
- 2026-10-23
AI Technical Summary
[0003]然而，传统Q学习算法在应用于路径规划时存在明显缺陷：（1）收敛速度慢
[0044]（1）、本发明的一种基于动态探索与启发式初始化的路径规划方法，通过基于曼哈顿距离的启发式Q表初始化，智能体在训练初期即获得目标方向引导，大幅减少了盲目的随机探索，使学习过程快速聚焦于有效区域，从而显著缩短了收敛所需的训练轮次。
Smart Images
Figure CN122961420A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robot navigation and autonomous control technology, and in particular to a path planning method based on dynamic exploration and heuristic initialization. Background Technology
[0002] Path planning is a core function of autonomous systems such as mobile robots and self-driving vehicles, aiming to find efficient, collision-free paths from a starting point to a target point. Reinforcement learning, especially Q-learning algorithms, has attracted widespread attention in this field due to its ability to learn optimal policies through trial and error without requiring prior environmental models. Traditional Q-learning algorithms store state-action values (Q-values) in a tabular format and update them iteratively using the Bellman equation.
[0003] However, traditional Q-learning algorithms have significant drawbacks when applied to path planning: (1) Slow convergence speed. The Q-table is usually initialized to zero or random values, and the agent lacks directional guidance in the early stages of training, resulting in a large number of ineffective explorations. (2) Low exploration efficiency. A fixed greedy exploration strategy is generally adopted, which cannot adaptively adjust the balance between exploration and utilization according to the learning stage, and is prone to getting stuck in local optima or prolonging training time. (3) Sparse reward signals. Rewards are usually given only when the target is reached, resulting in insufficient learning gradients and low algorithm training efficiency.
[0004] Existing improvements often focus on optimizing a single dimension, such as introducing simulated annealing to adjust the exploration rate, or introducing radial basis function neural networks or Boltzmann strategies to improve the reward function. However, these solutions fail to systematically address the coupling issues between initialization, exploration, and reward, leaving considerable room for improvement in planning efficiency and path quality in complex or large-scale environments. Summary of the Invention
[0005] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a path planning method based on dynamic exploration and heuristic initialization. This invention significantly improves the convergence speed, training efficiency and final path quality of the Q-learning algorithm in path planning tasks by integrating heuristic initialization, dynamic hybrid exploration strategy and continuous reward mechanism.
[0006] To achieve the above and other related objectives, this invention provides a path planning method based on dynamic exploration and heuristic initialization, comprising:
[0007] S1. Environment Modeling: Discretize the robot's motion environment into a grid map and define the current state of the corresponding agent. Define the action space of the agent based on its grid coordinates, and mark the obstacle grid, the starting grid, and the target grid.
[0008] S2. Heuristic Q-value initialization: Based on the current state of each agent... Calculate the current state With the target state Manhattan distance According to each action in the action space of the intelligent agent For all actions Initialize the Q value;
[0009] S3, Dynamic Hybrid Exploration Strategy Execution: In the current state Below, with a dynamically decaying probability Execute a greedy strategy and select the current state. The action with the highest Q-value is selected as the candidate optimal action; based on probability... Implement the Boltzmann strategy and select actions based on the probability distribution;
[0010] S4. Instant Reward Calculation: Based on the action selected through a dynamic hybrid exploration strategy, after the agent executes the action, the current state is calculated... Transition to the next state According to the next state Receive instant rewards ;
[0011] S5, Q-value iterative update: based on the aforementioned instant reward Update the current state-action equation based on the Q-value. The corresponding Q value updates the current state-action. The corresponding Q value;
[0012] S6. Repeat steps S3 to S5 until the preset number of training rounds or policy convergence is reached. By selecting the action with the maximum Q value in each current state, the optimal path policy from the starting point to the target is generated.
[0013] S7. Convert the optimal path strategy obtained from training into a specific grid path sequence, and draw the starting point, obstacles, target point and planned path in the graphical interface.
[0014] In one embodiment of the present invention, the action space of the intelligent agent is {up, down, left, right}.
[0015] In one embodiment of the present invention, the step S2, which involves determining each action in the action space of the agent, is... For all actions Initialize the Q value, including:
[0016] Based on each action in the action space of the intelligent agent According to formula (1), all actions Initialize Q value:
[0017] (1)
[0018] Based on the initial Q value, the agent is provided with initial target direction guidance.
[0019] In one embodiment of the present invention, the step S3 described in terms of probability Implementing a Boltzmann policy involves selecting actions based on a probability distribution, including:
[0020] With probability When implementing a Boltzmann strategy, actions are selected according to the probability distribution in formula (2):
[0021] (2)
[0022] in, For temperature parameters, This is the action space of the intelligent agent. The dynamically decaying probability is the action selected after executing the exploration strategy. Update according to the rules of formula (3):
[0023] (3)
[0024] in, To minimize the exploration rate, The attenuation coefficient (0 < <1), The exploration rate before selecting the optimal action.
[0025] In one embodiment of the present invention, the instant reward in step S4 Determined by the following rules:
[0026] If the next state If it is an obstacle state, then Negative value penalty;
[0027] If the next state If the target state is... It's the perfect time for rewards;
[0028] otherwise, ,
[0029] in, The base reward for a single step of movement. This is the distance reward coefficient. The maximum Manhattan distance in the environment, For the next state With the target state The Manhattan distance between them is used to quantify how close the agent is to the target after performing an action.
[0030] In one embodiment of the present invention, the update current state action in step S5 The corresponding Q values include:
[0031] ,
[0032] in, For learning rate, As a discount factor, For the next state The maximum Q-value corresponding to the optimal action among all available actions.
[0033] This invention also provides a path planning system based on dynamic exploration and heuristic initialization, comprising:
[0034] The environment modeling module is used to discretize the robot's motion environment into a grid map and define the current state of the agent corresponding to the robot. Define the action space of the agent based on its grid coordinates, and mark the obstacle grid, the starting grid, and the target grid.
[0035] An initialization module is used to initialize each agent based on its current state. Calculate the current state With the target state Manhattan distance According to each action in the action space of the intelligent agent For all actions Initialize the Q value;
[0036] The strategy decision module is used in the current state Below, with a dynamically decaying probability Execute a greedy strategy and select the current state. The action with the highest Q-value is selected as the candidate optimal action; based on probability... Implement the Boltzmann strategy and select actions based on the probability distribution;
[0037] The reward calculation module is used to calculate the reward based on the action selected through a dynamic hybrid exploration strategy, after which the agent executes the action and determines the reward from the current state. Transition to the next state According to the next state Receive instant rewards ;
[0038] The learning update module is used to update the instant reward. Update the current state-action equation based on the Q-value. The corresponding Q value updates the current state-action. The corresponding Q value;
[0039] The training control module is used to repeat steps S3 to S5 until the preset training rounds or policy convergence are reached. By selecting the action with the maximum Q value in each current state, the optimal path policy from the starting point to the target is generated.
[0040] The path output module is used to convert the optimal path strategy obtained from training into a specific grid path sequence, and to draw the starting point, obstacles, target point and planned path in the graphical interface.
[0041] The present invention also provides an electronic device, characterized in that it includes a processor and a memory, the memory storing program instructions, and the processor executing the program instructions to implement the above-described path planning method based on dynamic exploration and heuristic initialization.
[0042] The present invention also provides a computer-readable storage medium, characterized in that it stores a computer program thereon, which, when executed by a processor, implements the above-described path planning method based on dynamic exploration and heuristic initialization.
[0043] As described above, the path planning method based on dynamic exploration and heuristic initialization of the present invention has the following beneficial effects:
[0044] (1) The path planning method based on dynamic exploration and heuristic initialization of the present invention uses heuristic Q-table initialization based on Manhattan distance to guide the target direction in the early stage of training, which greatly reduces blind random exploration and enables the learning process to quickly focus on the effective area, thereby significantly shortening the training rounds required for convergence.
[0045] (2) The path planning method based on dynamic exploration and heuristic initialization proposed in this invention combines the stability of the greedy strategy with the flexibility of the Boltzmann strategy, and achieves an adaptive transition from "extensive exploration" to "refined utilization" through an exploration rate that decays over time. This effectively balances the contradiction between exploration and utilization, avoids prematurely falling into local optima, and improves the global optimization capability of the strategy.
[0046] (3) The path planning method based on dynamic exploration and heuristic initialization of the present invention designs a multi-level continuous reward function, which adds distance-based dense reward guidance on the basis of sparse final reward. This enables the agent to obtain gradient signals related to the environment state at each step, which greatly alleviates the sparse reward problem and improves the utilization efficiency of training data and the stability of the algorithm. Attached Figure Description
[0048] Figure 1 This is a flowchart of a path planning method based on dynamic exploration and heuristic initialization according to the present invention.
[0049] Figure 2 This is a schematic diagram of a grid environment used in the path planning method based on dynamic exploration and heuristic initialization according to the present invention.
[0050] Figure 3 This is a block diagram of a path planning system based on dynamic exploration and heuristic initialization according to the present invention.
[0051] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0053] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.
[0054] This section will describe in detail specific embodiments of the present invention. Preferred embodiments of the present invention are shown in the accompanying drawings. The purpose of the drawings is to supplement the textual description with graphics, so that people can intuitively and vividly understand each technical feature and overall technical solution of the present invention, but they should not be construed as limiting the scope of protection of the present invention.
[0055] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," and "horizontal," etc., indicating orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. The terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two elements. For those skilled in the art, the specific meaning of the above terms in this invention can be understood according to the specific circumstances.
[0056] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0057] Please see Figure 1 , Figure 2 , Figure 1 This is a flowchart of a path planning method based on dynamic exploration and heuristic initialization according to the present invention. Figure 2 This is a schematic diagram of a grid environment used in the path planning method based on dynamic exploration and heuristic initialization according to the present invention.
[0058] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.
[0059] This invention includes a path planning method based on dynamic exploration and heuristic initialization, comprising:
[0060] Step S1, Environment Modeling: Discretize the robot's motion environment into a grid map and define the current state of the agent corresponding to the robot. Given the grid coordinates of the agent, define the agent's action space as {up, down, left, right}, and mark the obstacle grid, starting grid, and target grid.
[0061] Step S2, Heuristic Q-value initialization: Based on the current state of each agent... Calculate the current state With the target state Manhattan distance According to each action in the action space of the intelligent agent All actions are based on the following formula. Initialize Q value:
[0062]
[0063] Target grid localization based on the grid map in S1 provides initial target direction guidance for the agent.
[0064] Step S3, Execution of dynamic hybrid exploration strategy: In the current state Below, with a dynamically decaying probability Execute a greedy strategy and select the current state. The action with the highest Q-value is selected as the candidate optimal action; based on probability... When implementing a Boltzmann strategy, actions are selected based on probability distributions:
[0065] ,
[0066] in, For temperature parameters, The agent's action space is defined in S1. The action selected after executing the exploration strategy; the dynamically decaying probability. Update according to the following rules:
[0067]
[0068] in, To minimize the exploration rate, The attenuation coefficient (0 < <1), The exploration rate before selecting the optimal action.
[0069] Step S4, Instant Reward Calculation: Based on the action selected through a dynamic hybrid exploration strategy, after the agent executes the action, the current state is determined... Transition to the next state According to the next state Receive instant rewards The reward Determined by the following rules:
[0070] like If it is an obstacle state, then (Negative value penalty);
[0071] like If the target state is... (Positive reward);
[0072] otherwise,
[0073] in, The base reward for a single step of movement. This is the distance reward coefficient. The maximum Manhattan distance in the environment, For the next state With the target state The Manhattan distance between them is used to quantify how close the agent is to the target after performing an action; this step provides dense gradient feedback to the agent based on the execution results of the selected actions in S3, guiding it to optimize in the direction of the target.
[0074] Step S5, Q-value iterative update: based on the instant reward Update the current state-action equation based on the Q-value. The corresponding Q value updates the current state-action. The corresponding Q value.
[0075] The action of updating the current state in step S5 The corresponding Q values include:
[0076] ,
[0077] in, For learning rate, As a discount factor, For the next state The maximum Q-value corresponding to the optimal action among all available actions.
[0078] Step S6: Repeat steps S3 to S5 until the preset number of training rounds or policy convergence is reached. By selecting the action with the maximum Q value in each current state, the optimal path policy from the starting point to the target is generated.
[0079] Step S7: Convert the optimal path strategy obtained from training into a specific grid path sequence, and draw the starting point, obstacles, target point and planned path in the graphical interface.
[0080] In one embodiment of the present invention, the present invention includes a path planning method based on dynamic exploration and heuristic initialization, comprising:
[0081] Step S1, as follows Figure 2 As shown, a 10×10 grid environment is constructed. The starting point is set to (0,0), and the ending point is set to (9,9). Several obstacle grids are randomly or fixedly placed. The state is the agent's coordinates, and the action is to move one grid in each of the four directions: up, down, left, and right.
[0082] Step S2: Traverse the current state of all non-obstacle grid cells. Calculate its Manhattan distance from the destination. For all possible actions in the current state. Initialize Q value: Set the initial Q value Set as .
[0083] Step S3: Set the learning rate =0.1, discount factor =0.9, initial exploration rate =0.9, minimum exploration rate =0.1, attenuation coefficient =0.995, temperature parameter =1.0. Set the total number of training rounds. =1000.
[0084] Step S4: For each training round:
[0085] (1) Smart weight is placed at the starting point;
[0086] (2) In the current state Select actions based on hybrid exploration strategies Generate a random number between [0,1]. ,like Then a greedy strategy is adopted, selecting the option that makes... The largest action is selected; otherwise, the Boltzmann strategy is used to calculate the probability of each action according to the formula and sample according to the probability.
[0087] (3) Perform the action , transition to a new state And based on the multi-level reward function (see logic below) Figure 4 ) Calculate rewards If it hits an obstacle, =-1, if the destination is reached, =+10;
[0088] (4) Update the formula based on Q, and utilize , , , Update the Q table;
[0089] (5) Update the current exploration rate;
[0090] (6) If If the time limit for the end point or the number of rounds exceeds the limit, the round ends.
[0091] Step S5: After training is complete, starting from the starting point, in each state... Choose below Biggest movement The optimal path is determined by recording the sequence of grid cells traversed until the destination is reached.
[0092] Step S6: Draw a grid map, obstacles, start point, and end point in the graphical interface, and connect the path generated in step S5 with lines.
[0093] Please see Figure 3 , Figure 3 This is a block diagram of a path planning system based on dynamic exploration and heuristic initialization according to the present invention. The path planning system based on dynamic exploration and heuristic initialization of the present invention includes:
[0094] The environment modeling module 10 is used to discretize the robot's motion environment into a grid map and define the current state of the agent corresponding to the robot. Define the action space of the agent based on its grid coordinates, and mark the obstacle grid, the starting grid, and the target grid.
[0095] Initialization module 20 is used to initialize each agent based on its current state. Calculate the current state With the target state Manhattan distance According to each action in the action space of the intelligent agent For all actions Initialize the Q value;
[0096] Strategy decision module 30, used in the current state Below, with a dynamically decaying probability Execute a greedy strategy and select the current state. The action with the highest Q-value is selected as the candidate optimal action; based on probability... Implement the Boltzmann strategy and select actions based on the probability distribution;
[0097] The reward calculation module 40 is used to calculate the reward based on the action selected through a dynamic hybrid exploration strategy, after which the agent executes the action and determines the reward from the current state. Transition to the next state According to the next state Receive instant rewards ;
[0098] Learning update module 50, used for updating based on the instant reward Update the current state-action equation based on the Q-value. The corresponding Q value updates the current state-action. The corresponding Q value;
[0099] The training control module 60 is used to repeat steps S3 to S5 until a preset number of training rounds or policy convergence is reached. By selecting the action with the maximum Q value in each current state, the optimal path policy from the starting point to the target is generated.
[0100] The path output module 70 is used to convert the trained optimal path strategy into a specific grid path sequence and draw the starting point, obstacles, target point and planned path in the graphical interface.
[0101] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 4An electronic device provided in this application includes a processor and a memory. The memory stores a computer program that, when executed by the processor, enables the implementation of the path planning method based on dynamic exploration and heuristic initialization. This electronic device can be the main control computer of an autonomous vehicle, an embedded system of a mobile robot, or a server / workstation for algorithm development.
[0102] On the other hand, this embodiment provides a computer-readable storage medium, such as a USB flash drive, hard disk, optical disk, or solid-state drive, on which a computer program is stored. When the program is accessed by a computing device (such as...) Figure 4 When the processor of an electronic device (in the device) executes, it is able to achieve Figure 1 The path planning method based on dynamic exploration and heuristic initialization is described above.
[0103] In summary, the path planning method based on dynamic exploration and heuristic initialization of the present invention, through heuristic Q-table initialization based on Manhattan distance, enables the agent to obtain target direction guidance in the early stage of training, greatly reducing blind random exploration, enabling the learning process to quickly focus on the effective region, thereby significantly shortening the training rounds required for convergence.
[0104] The technical principles of the present invention have been described above with reference to specific embodiments. These descriptions are merely for explaining the principles of the invention and should not be construed as limiting the scope of protection of the invention in any way. Based on this explanation, those skilled in the art can readily conceive of other specific embodiments of the invention without inventive effort, and these equivalent variations or substitutions are all included within the scope defined by the claims of this application.
[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A path planning method based on dynamic exploration and heuristic initialization, characterized in that, include: S1. Environment Modeling: Discretize the robot's motion environment into a grid map and define the current state of the corresponding agent. Define the action space of the agent based on its grid coordinates, and mark the obstacle grid, the starting grid, and the target grid. S2. Heuristic Q-value initialization: Based on the current state of each agent... Calculate the current state With the target state Manhattan distance According to each action in the action space of the intelligent agent For all actions Initialize the Q value; S3, Dynamic Hybrid Exploration Strategy Execution: In the current state Below, with a dynamically decaying probability Execute a greedy strategy and select the current state. The action with the highest Q-value is selected as the candidate optimal action; based on probability... Implement the Boltzmann strategy and select actions based on the probability distribution; S4. Instant Reward Calculation: Based on the action selected through a dynamic hybrid exploration strategy, after the agent executes the action, the current state is calculated... Transition to the next state According to the next state Receive instant rewards ; S5, Q-value iterative update: based on the aforementioned instant reward The current state-action sequence is updated by learning the Bellman equation based on the Q-value. The corresponding Q value updates the current state-action. The corresponding Q value; S6. Repeat steps S3 to S5 until the preset number of training rounds or policy convergence is reached. By selecting the action with the maximum Q value in each current state, the optimal path policy from the starting point to the target is generated. S7. Convert the optimal path strategy obtained from training into a specific grid path sequence, and draw the starting point, obstacles, target point and planned path in the graphical interface.
2. The path planning method based on dynamic exploration and heuristic initialization according to claim 1, characterized in that: The action space of the intelligent agent is {up, down, left, right}.
3. The path planning method based on dynamic exploration and heuristic initialization according to claim 1, characterized in that: Step S2, based on each action in the agent's action space For all actions Initialize the Q value, including: Based on each action in the action space of the intelligent agent According to formula (1), all actions Initialize Q value: （1） Based on the initial Q value, the agent is provided with initial target direction guidance.
4. The path planning method based on dynamic exploration and heuristic initialization according to claim 3, characterized in that: The probability mentioned in step S3 Implementing a Boltzmann policy involves selecting actions based on a probability distribution, including: With probability When implementing a Boltzmann strategy, actions are selected according to the probability distribution in formula (2): （2） in, For temperature parameters, This is the action space of the intelligent agent. The dynamically decaying probability is the action selected after executing the exploration strategy. Update according to the rules of formula (3): （3） in, To minimize the exploration rate, The attenuation coefficient (0 < <1), The exploration rate before selecting the optimal action.
5. The path planning method based on dynamic exploration and heuristic initialization according to claim 4, characterized in that: The instant reward in step S4 Determined by the following rules: If the next state If it is an obstacle state, then Negative value penalty; If the next state If the target state is... It's the perfect time for rewards; otherwise, , in, The base reward for a single step of movement. This is the distance reward coefficient. The maximum Manhattan distance in the environment, For the next state With the target state The Manhattan distance between them is used to quantify how close the agent is to the target after performing an action.
6. The path planning method based on dynamic exploration and heuristic initialization according to claim 5, characterized in that: The action of updating the current state in step S5 The corresponding Q values include: ， in, For learning rate, As a discount factor, For the next state The maximum Q value corresponding to the optimal action among all available actions.
7. A path planning system based on dynamic exploration and heuristic initialization, characterized in that, include: The environment modeling module (10) is used to discretize the robot's motion environment into a grid map and define the current state of the agent corresponding to the robot. Define the action space of the agent based on its grid coordinates, and mark the obstacle grid, the starting grid, and the target grid. Initialization module (20) is used to initialize each agent based on its current state. Calculate the current state With the target state Manhattan distance According to each action in the action space of the intelligent agent For all actions Initialize the Q value; The strategy decision module (30) is used in the current state Below, with a dynamically decaying probability Execute a greedy strategy and select the current state. The action with the highest Q-value is selected as the candidate optimal action; based on probability... Implement the Boltzmann strategy and select actions based on the probability distribution; The reward calculation module (40) is used to calculate the reward based on the action selected by the agent through a dynamic hybrid exploration strategy, after the agent performs the action, based on the current state. Transition to the next state According to the next state Receive instant rewards ; The learning update module (50) is used to update the instant reward. The current state-action sequence is updated by learning the Bellman equation based on the Q-value. The corresponding Q value updates the current state-action. The corresponding Q value; The training control module (60) is used to repeat steps S3 to S5 until the preset training rounds or policy convergence are reached. By selecting the action with the maximum Q value in each current state, the optimal path policy from the starting point to the target is generated. The path output module (70) is used to convert the optimal path strategy obtained from training into a specific grid path sequence, and to draw the starting point, obstacles, target point and planned path in the graphical interface.
8. An electronic device, characterized in that, It includes a processor and a memory, the memory storing program instructions, and the processor executing the program instructions to implement the path planning method based on dynamic exploration and heuristic initialization as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the path planning method based on dynamic exploration and heuristic initialization as described in any one of claims 1 to 6.