Path planning method and device and electronic equipment

Abstract

The invention relates to the technical field of artificial intelligence and automatic control, and discloses a path planning method and device and electronic equipment, and the method comprises the steps: carrying out the dynamic adjustment of a learning rate and an exploration rate in a Q learning model based on the Q learning model; on the basis of Monte Carlo tree search, path selection in the Q learning model is simulated, and an accumulated discount reward of a simulated path is obtained; based on the accumulated discount rewards, correcting the Q learning model to obtain a path planning correction model; and planning the path of the intelligent agent based on the path planning correction model. According to the invention, the convergence speed of intelligent agent path planning can be improved, the dynamic obstacle avoidance efficiency is enhanced, and the real-time decision-making capability of the intelligent agent in a complex environment is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence and automatic control, in particular to a path planning method and device and electronic equipment. BACKGROUND

[0002] With the increasing demand for path planning of agents in complex environments, the limitations of the Q-learning method in the related art in convergence speed, obstacle avoidance efficiency and global optimization ability are gradually highlighted.

[0003] In the related art, the path planning of the agent has obvious defects in dynamic environment adaptability, multi-step decision optimization and algorithm parameter self-adaptation, which seriously restricts the application effect of the agent in practical scenarios such as unmanned driving and robot navigation. SUMMARY

[0004] The present application provides a path planning method, device and electronic equipment, which has the advantages of improving path planning efficiency, enhancing global optimization ability and improving dynamic environment adaptability.

[0005] In a first aspect, the present disclosure provides a path planning method, and the technical solution is as follows: based on a Q-learning model, dynamically adjusting a learning rate and an exploration rate in the Q-learning model, wherein the dynamic adjustment is used to represent the change of the learning rate and the exploration rate under different iteration numbers; based on Monte Carlo tree search, simulating path selection in the Q-learning model to obtain accumulated discounted rewards of simulated paths; based on the accumulated discounted rewards, correcting the Q-learning model to obtain a path planning correction model; and based on the path planning correction model, planning a path for an agent.

[0006] In a second aspect, the present disclosure provides a path planning device, comprising: an adjustment module configured to dynamically adjust a learning rate and an exploration rate in a Q-learning model based on the Q-learning model, wherein the dynamic adjustment is used to represent the change of the learning rate and the exploration rate under different iteration numbers; an obtaining module configured to simulate path selection in the Q-learning model based on Monte Carlo tree search to obtain accumulated discounted rewards of simulated paths; a correction module configured to correct the Q-learning model based on the accumulated discounted rewards to obtain a path planning correction model; and a planning module configured to plan a path for an agent based on the path planning correction model.

[0007] In a third aspect, the present disclosure provides an electronic device, comprising a memory and a processor, which are communicatively connected to each other, and the memory stores computer instructions, and the processor executes the computer instructions to run the above path planning method.

[0008] The path planning method, device and electronic equipment provided by the application effectively solve the problems of slow convergence speed, low obstacle avoidance efficiency and easy falling into local optimum of the traditional method by dynamically adjusting the learning rate and exploration rate parameters and combining the long-term path return evaluation of the Monte Carlo tree search, thereby improving the path planning efficiency, enhancing the global optimization capability and improving the dynamic environment adaptability. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the specific embodiments of the present disclosure or the prior art, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present disclosure, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0010] Figure 1 is a first flowchart of a path planning method according to an embodiment of the present disclosure; Figure 2 is a second flowchart of a path planning method according to an embodiment of the present disclosure; Figure 3 is a whole flowchart of a path planning method according to an embodiment of the present disclosure; Figure 4 is an effect diagram of a path planning method according to an embodiment of the present disclosure; Figure 5 is a structural block diagram of a path planning device according to an embodiment of the present disclosure; Figure 6 is a hardware structure schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely in combination with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present disclosure.

[0012] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the user should be informed of the type, use range, use scene and the like of the personal information involved in the present disclosure and obtain the authorization of the user in a proper manner according to relevant laws and regulations.

[0013] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "a plurality of" means two or more, unless otherwise expressly specified.

[0014] In related technologies, the field of intelligent agent path planning has long relied on traditional Q-Learning algorithms for decision optimization. Traditional methods use fixed learning and exploration rate parameters, leading to low exploration efficiency in the early stages of training and a tendency to get trapped in local optima later on. Especially in environments with dense dynamic obstacles, this method struggles to effectively predict long-term path gains, frequently resulting in obstacle avoidance failures or path detours, severely limiting the navigation efficiency and safety of intelligent agents.

[0015] To address the aforementioned issues, the applicant identified the fixed parameters and single-step decision-making mode of traditional algorithms as core bottlenecks. By analyzing the temporal characteristics of path planning in dynamic obstacle scenarios, it was recognized that a multi-step prediction mechanism is needed to enhance global search capabilities. Further investigation revealed that the long-sequence simulation characteristics of Monte Carlo tree search can compensate for the short-sightedness of Q-Learning, while dynamically adjusting learning parameters can balance the exploration and utilization needs at different training stages. Therefore, a collaborative optimization mechanism is proposed, integrating the advantages of both algorithms by refining the Q-value update process through simulating the long-term benefits of the path.

[0016] Therefore, this application proposes a path planning method. This method first dynamically adjusts the learning rate and exploration rate parameters of the Q-learning model to adapt to the needs of different training stages. Then, Monte Carlo tree search is used to simulate path selection in multiple steps, calculating the accumulated discount reward of the simulated path. This reward value is then incorporated into the Q-value update formula to construct a corrected model. Finally, the agent's path planning is completed based on the corrected model. Figure 1 The diagram shows the first flowchart of the path planning method, which specifically includes the following steps: Step S101: Based on the Q-learning model, dynamically adjust the learning rate and exploration rate in the Q-learning model, wherein the dynamic adjustment is used to characterize the changes in the learning rate and exploration rate under different iteration numbers.

[0017] Dynamically adjusting the learning rate refers to automatically adjusting the parameter update step size according to the training progress. This can be achieved using an exponential decay function; for example, a higher learning rate is set initially to accelerate convergence, and then the learning rate is reduced later to stabilize the strategy. Dynamically adjusting the exploration rate refers to controlling the decay process of the probability of random action selection. This can also be achieved using an exponential function; a high exploration rate is maintained in the early stages of training to expand the search range, and then the exploration rate is reduced later to improve strategy utilization efficiency.

[0018] Step S102: Based on Monte Carlo tree search, simulate the path selection in the Q-learning model and obtain the accumulated discount reward of the simulated path.

[0019] Monte Carlo tree search path simulation refers to multi-step path prediction through tree structure expansion, which can be implemented using a depth-first search with a preset step size, evaluating the long-term benefits of actions at each decision node. Accumulated discount reward calculation refers to the weighted summation of the immediate rewards of the simulated path, which can be implemented using a power-law discount factor, where rewards further away from the current time have smaller weights.

[0020] Step S103: Based on the accumulated discount reward, the Q-learning model is modified to obtain the path planning modification model.

[0021] Among them, the modified model construction refers to integrating the long-term benefits of the simulated path into the Q-value update. Specifically, it can be achieved by using a linear weighting method to balance the current Q-value with the predicted benefits of the simulated path.

[0022] Step S104: Based on the path planning correction model, plan the path for the agent.

[0023] Specifically, path planning first requires modeling the environment using a grid method to determine the initial and target points, and initializing the Q-table and parameters. Taking this disclosure as an example, the overall environment is divided into a regular 20×20 grid map. In this grid system, each grid represents a basic unit that the agent can move in. Obstacle areas (presented in black) and passable areas (represented in white) are clearly marked on the map using visualization techniques. The agent is only allowed to move along the grid edges and is strictly prohibited from crossing obstacle areas, thus simulating the passage restrictions in a real environment. The agent's starting and target positions on the map are precisely set; these two key positions will serve as the starting and ending points for subsequent path planning.

[0024] Create a Q-table, where rows correspond to the state space, comprehensively covering all possible states the agent may be in the environment; columns represent the action space, specifically including the four basic movement directions: up, down, left, and right. During the initialization phase, set the initial parameters. As an example in this disclosure, the following settings can be made: Set the Q-value of all state-action pairs in the Q-table to 0 to lay the foundation for subsequent learning and updates. Set the initial learning rate. =0.1, final learning rate =0.01; Set the initial exploration rate =1.0, exploration rate decay coefficient =0.002; Set the discount factor =0.9, maximum number of training iterations =2000; Set the number of MCTS simulation steps. =10, MCTS reward weight =0.3; Set the reward value for reaching the target. =300, penalty value for colliding with obstacles. =-200, thereby constructing a reward and punishment mechanism to guide the agent to learn correct behavior.

[0025] A high exploration rate is set at the beginning of training to encourage the agent to try different paths. As the number of iterations increases, the exploration rate is gradually reduced while the learning rate decays faster, allowing the policy to stabilize. When the agent is in a specific state, Monte Carlo tree search is initiated to simulate multiple paths, recording the immediate reward sequence for each simulated path. The long-term reward is decayed and summed using a discount factor to obtain the accumulated discounted reward reflecting the long-term benefits of the path. This reward value is introduced as a correction term into the Q-value update formula and weighted and fused with the original Q-value update term. The resulting modified model retains the sensitivity of Q-learning to immediate benefits while possessing the global perspective of Monte Carlo tree search, thus generating optimized paths that balance safety and efficiency in complex obstacle environments.

[0026] Compared to related technologies, traditional Q-Learning, with its fixed parameters, suffers from limited convergence speed, and its single-step update mechanism lacks long-term planning capabilities. This scheme adapts to the needs of different training stages through a dynamic parameter adjustment mechanism and expands the decision-making scope using the deep simulation capabilities of Monte Carlo tree search. This fusion approach not only improves training efficiency but also generates better obstacle avoidance paths in obstacle-dense regions, overcoming the tendency of traditional methods to get trapped in local optima.

[0027] Through the above technical solution, this application effectively solves the problem of slow convergence speed in traditional methods by accelerating the training process through dynamic parameter adjustment. It also improves the quality of obstacle avoidance decisions by using multi-step simulation prediction to avoid potential collision risks. More importantly, it enhances global path optimization capabilities, generating more reasonable navigation routes by combining long-term benefit evaluation, significantly improving the agent's movement efficiency and safety in complex environments.

[0028] This application further proposes a method based on the Q-table in a Q-learning model to predict pre-selected actions in preset steps to obtain the immediate reward for each predicted action, and to calculate the cumulative discounted reward of the simulated path based on the immediate reward and a discount factor. Figure 2 The diagram shows the second flowchart of the path planning method, which includes: Step S201: Based on the Q-learning model, dynamically adjust the learning rate and exploration rate in the Q-learning model, wherein the dynamic adjustment is used to characterize the changes in the learning rate and exploration rate under different iteration numbers. Refer to step S101 for a detailed explanation of this scheme.

[0029] Step S202 involves simulating path selection in the Q-learning model based on Monte Carlo tree search to obtain the accumulated discount reward for the simulated path. Refer to step S102 for a detailed explanation of this scheme. Step S202 specifically includes: Step S2021: Based on the Q-table in the Q-learning model, perform preset step prediction on the pre-selected actions in the Q-learning model and obtain the instant reward for each predicted action.

[0030] Here, the Q-table refers to a two-dimensional table storing the distribution of state-action value. It can be implemented using a hash table or matrix data structure, recording the expected reward for each action in different states, providing prior knowledge guidance for Monte Carlo tree search. The pre-selected actions refer to the set of candidate actions sorted by Q-value in the current state. This can be implemented using a Top-K selection strategy, such as selecting the top five actions by Q-value to balance exploration and utilization efficiency. The preset step prediction refers to the limit on the number of consecutive actions executed during simulation. This can be set to a fixed value, such as ten steps, to control computational complexity while evaluating mid-term returns. The immediate reward is the scalar value of the reward after executing a single action.

[0031] Step S2022: Obtain the accumulated discount reward of the simulated path based on the immediate reward of each predicted action and the corresponding discount factor.

[0032] The discount factor is a decay coefficient used to reduce the impact of future rewards. It can be a constant value between 0.9 and 0.99, for example, set to 0.9, to quantify the current value of long-term returns.

[0033] Specifically, during path planning, candidate actions with higher Q-values ​​in the current state are first extracted from the Q-table as a pre-selected action set. Based on these pre-selected actions, a multi-step path simulation is performed using Monte Carlo tree search, with an immediate reward obtained for each action executed. During the simulation, the immediate reward for each action is exponentially decayed and weighted step by step, eventually accumulating to form an accumulated discounted reward. For example, in a ten-step simulation, the reward for the first step is multiplied by 0.9 to the power of zero, the reward for the second step is multiplied by 0.9 to the power of one, and so on, until the reward for the tenth step is directly included in the sum. This multi-step reward accumulation mechanism allows the model to evaluate the impact of the current decision on subsequent multi-step actions, thereby avoiding the short-sighted decision-making problem caused by the single-step update in traditional Q-Learning.

[0034] Step S203: Based on the accumulated discount reward, the Q-learning model is modified to obtain a path planning modification model. Refer to step S103 for a detailed explanation of this scheme.

[0035] Step S204: Based on the path planning correction model, plan the path for the agent. Refer to step S103 for a detailed explanation of this scheme.

[0036] Compared to related technologies, traditional Q-Learning methods only update the value of the current action based on the maximum Q-value of a single step, lacking a global evaluation of the benefits of multi-step action chains. Our proposed solution, however, uses Monte Carlo tree search for multi-step prediction and combines it with a discount factor to construct an accumulated reward index. This allows the agent to comprehensively consider the immediate benefits of the current action and the potential impact of future multi-step actions when choosing a path. For example, when encountering a U-shaped obstacle, traditional methods might choose a detour due to local optima, while our solution, through multi-step simulation, can discover the globally optimal path through the gaps in the obstacle.

[0037] Through the above technical solutions, this application effectively enhances the global optimization capability of path planning, enabling the agent to identify and select paths with long-term benefit advantages in complex environments. Simultaneously, by combining a pre-selection action mechanism with multi-step reward accumulation, the accuracy of obstacle avoidance decisions is significantly improved, avoiding misjudgments caused by local environmental characteristics.

[0038] In some embodiments, the cumulative discounted reward of the simulated path is obtained based on the immediate reward of each predicted action and the corresponding discount factor, and is achieved through the following expression: ,in, To accumulate discount rewards, specifically, the calculation of cumulative rewards with time discounts, the core is to measure the period from the current moment... The initial, and subsequent series of actions, yield "long-term value". A number between 0 and 1 (0 < <1), it is generally believed that "short-term rewards are more valuable than long-term rewards", so long-term rewards will be multiplied by 1 / 2. The higher the power, the lower the weight of the long-term reward. Let be the discount factor for step i. To simulate the immediate reward at step i in the path, i.e., the immediate feedback given by the environment after taking an action at a certain moment, N is the total number of steps, t is the moment when the value of the decision needs to be evaluated; t+1+i is the specific moment of the immediate reward after performing the action at step i, for example, when i=0... The moment is the instant reward moment for the next step.

[0039] in, Specifically, obstacle collision detection and target point arrival determination can be used to generate the algorithm. N represents the total number of steps, which is the upper limit of the prediction step size for the Monte Carlo tree search. This can be set using a fixed step size or a dynamic adjustment strategy to control the balance between computational complexity and prediction accuracy.

[0040] Specifically, this expression constructs a long-term return evaluation metric by progressively summing the decaying reward values ​​of each prediction step. During path simulation, the immediate reward at each prediction step is multiplied by a discount factor corresponding to the number of steps. This forms a reward sequence that decays exponentially over time. Setting the total number of steps N ensures controllable search depth, avoiding the computational burden of infinite recursion. By linearly summing the decaying reward values ​​of each step, the final result is obtained... This represents the potential benefit of the current action choice in future multi-step execution, providing a long-term benefit basis for Q-value updates.

[0041] Compared to related technologies, traditional Q-Learning relies solely on single-step rewards and the optimal Q-value for the next state for updates, lacking an overall evaluation of the returns for multi-step path sequences. Our proposed solution, however, constructs a cumulative decaying reward for an N-step path using Monte Carlo tree search, incorporating the potential returns of future steps into the evaluation system of the current action, thus overcoming the short-sighted decision-making flaws caused by single-step updates.

[0042] Through the above technical solution, this application effectively solves the problem of insufficient global optimization capability in path planning caused by the lack of a long-term reward evaluation mechanism in traditional methods. By constructing a cumulative decay reward calculation model for multi-step predicted paths, the agent can comprehensively consider the potential benefits of multiple future steps when making action selections, avoiding getting trapped in local optima and improving the global path planning performance in complex obstacle environments.

[0043] In some embodiments, the Q-learning model is modified based on the accumulated discount reward, as expressed by the following expression: , in, State at time t Next action Q value, For learning rate, For instant rewards, As a discount factor, In order to perform the action The new state after transition The maximum value of Q among all possible actions. To accumulate discount rewards.

[0044] in, Specifically, a state-action value function table can be used to quantify the long-term benefits of different actions. Specifically, an exponential decay strategy can be used, with an initial value of 0.8 and a final value of 0.01, to balance the absorption ratio of new and old knowledge. Specifically, a constant between 0.3 and 0.7 can be used to adjust the degree of influence of global path simulation on Q-value updates. Specifically, this can be obtained through multi-step path simulation calculations, which are used to reflect the global benefits of long-term paths.

[0045] Specifically, this expression superimposes the standard Q-learning update rule. This creates a composite reward mechanism. When the agent is in a certain state... Execute action Simultaneously, the system performs Monte Carlo tree search to predict multi-step paths, accumulating discount rewards within N steps to form... This global reward item is related to the current instant reward. The maximum Q value in the next state Together they constitute the target Q value, through Learning rate relative to original A gradual adjustment will be made. Weighting coefficients. It can be set to 0.3 to achieve a balanced superposition effect between the standard Q-learning update term and the Monte Carlo tree search reward term. This modification allows the Q-value update process to simultaneously incorporate immediate rewards and simulated long-term path information.

[0046] Compared to related technologies, traditional Q-Learning methods only perform single-step updates and lack explicit modeling of multi-step path benefits. This solution introduces... This approach integrates the cumulative reward of an N-step path from a Monte Carlo tree search simulation into the Q-value update, enabling the agent to simultaneously consider the short-term gains of the current action and the long-term value of potential paths during decision-making. Meanwhile, The coefficient settings avoid excessive influence from Monte Carlo tree search results and prevent Q-value distortion due to simulation path deviation.

[0047] Through the above technical solutions, this application effectively enhances the search capability for the global optimum in path planning. The accumulated discounted reward generated by Monte Carlo tree search provides an additional long-term benefit signal for the Q-learning model, enabling the agent to more accurately evaluate the global value of actions in complex environments and significantly reducing the probability of getting trapped in local optima. This composite update mechanism also improves the algorithm's adaptability to dynamic obstacles, incorporating path evaluation results from offline simulation while maintaining online learning efficiency.

[0048] In some embodiments, instant rewards are implemented using the following expression: , in, This represents the Manhattan distance from the current state to the target.

[0049] Manhattan distance refers to the sum of the horizontal and vertical distances between the current state and the target point in a two-dimensional grid. It can be calculated by summing the absolute values ​​of the coordinate differences and is used to quantify the optimization space of path planning. The reward for reaching the target is set to +300, which can be implemented through a positive feedback mechanism in path planning training to accelerate the algorithm's convergence to the target point. The penalty for colliding with obstacles is set to -200, which can be implemented through a negative reinforcement mechanism to prevent the agent from entering dangerous areas. The reward function for other states is designed to be inversely proportional to the Manhattan distance, and can be scaled using a linear coefficient of -0.1 to guide the agent to move towards the target.

[0050] Specifically, when the agent reaches the target point, a high positive reward signal is immediately triggered, forming strong positive behavioral feedback to shorten the convergence time. When a collision with an obstacle is detected, a severe negative reward signal is generated, forcing the algorithm to re-evaluate the path selection strategy. Under normal movement conditions, the reward value is dynamically calculated using Manhattan distance, enabling the agent to continuously move towards the target while moving away from obstacles. This multimodal reward mechanism establishes a dynamic balance between obstacle avoidance safety and path optimization efficiency by quantitatively evaluating the consequences of different behaviors, thus solving the problem of ambiguity in exploration direction caused by a single reward function.

[0051] Compared to related technologies, traditional Q-Learning methods typically use fixed reward values ​​or a single distance metric, failing to effectively differentiate the varying needs of reaching the target, colliding with obstacles, and normal movement. This solution constructs a tiered reward system to decouple obstacle avoidance penalties from path optimization incentives, enabling the agent to quickly identify high-risk areas and autonomously plan the globally optimal path in complex environments.

[0052] Through the above technical solutions, this application achieves multi-dimensional guidance of agent behavior, significantly shortening path length while avoiding collisions. The high reward for reaching the target accelerates the training convergence process, the collision penalty mechanism reduces safety risks in path planning, and the progressive reward based on Manhattan distance continuously optimizes the choice of movement direction, ultimately generating feasible paths that balance safety and efficiency in dynamic environments.

[0053] This application further proposes a technical solution to dynamically adjust the learning rate based on the current iteration number and the maximum iteration number, and to dynamically adjust the exploration rate based on the current iteration number.

[0054] Dynamic learning rate adjustment refers to gradually adjusting the parameter update magnitude according to the training progress, which can be specifically achieved using an exponential growth formula. To achieve this, where the initial learning rate... It can be set to 0.1, the final learning rate. It can be set to 0.01. This formula can be used to maintain a high learning rate in the early stages of training to accelerate convergence, and to reduce the learning rate in the later stages to avoid oscillations.

[0055] Among them, dynamic adjustment of the exploration rate refers to controlling the probability of random exploration through exponential decay. Specifically, it can be achieved by... The expression is implemented, where the initial exploration rate is... The value can be set to 0.9, and the decay coefficient k can be set to 0.002. This formula maintains a high exploration rate in the early stage of training, fully searches the environment, and gradually shifts to utilizing the learned knowledge as the number of iterations increases.

[0056] Specifically, in the learning rate adjustment formula The maximum number of iterations is preset, for example, 2000. When the current iteration number i reaches the maximum value, the learning rate will stabilize at... The decay coefficient k of the exploration rate adjustment controls the rate of decline. For example, when k=0.002, after 1000 iterations, the exploration rate will decrease to about 13.5% of the initial value. The coordinated adjustment of the two parameters allows the agent to quickly absorb immediate rewards with a learning rate of 0.8 in the early stages of training, while simultaneously trying out a wide range of actions with an exploration rate of 0.9. As training progresses, the learning rate gradually decreases to 0.01 to ensure stable convergence of the Q-value, and the exploration rate decreases simultaneously, prompting the agent to rely more on the optimized Q-table for decision-making.

[0057] Through the above technical solution, this application solves the problem of slow initial convergence speed caused by a fixed learning rate, and accelerates early experience accumulation by using a high learning rate; it also overcomes the defect of over-exploration in the later stage caused by a fixed exploration rate, and achieves automatic balance between exploration and utilization through a decay mechanism. In a 20×20 grid environment test, this method reduces the number of training rounds by an average of 23% compared with traditional methods, and the final path planning success rate is improved to 98.7%.

[0058] This application further proposes a path planning correction model for the agent, which includes randomly selecting any action based on the current exploration rate and selecting the action with the largest Q value in the current state based on the current utilization rate; wherein the current exploration rate and the current utilization rate are complementary, and the sum of their probabilities is always equal to 1.

[0059] Here, the current exploration rate refers to the probability that the agent will randomly select an action during path planning. This can be dynamically adjusted using an exponential decay function, for example, through... The formula ensures a high exploration probability in the early stages of training to discover potential paths. Current utilization refers to the probability that the agent selects the optimal action based on existing knowledge, which can be specifically expressed as 1- Calculations show that this feature ensures the preferential use of the learned optimal strategy in the later stages of training. The complementarity relationship refers to the constraint that the sum of the exploration rate and the utilization rate is constant at 1, which can be achieved through probability normalization. This feature enforces the integrity of the action selection strategy, preventing excessive suppression of exploration or utilization behavior.

[0060] Specifically, during path planning, the agent makes action decisions based on dynamically adjusted exploration and utilization rates. In the early stages of training, a higher exploration rate encourages the agent to randomly try different paths, expanding the search range to avoid obstacles and discover the globally optimal path. As the number of iterations increases, the exploration rate gradually decreases, while the utilization rate increases accordingly. The agent then prioritizes actions with the highest Q-value to accelerate convergence. This complementary constraint ensures that the weight distribution of exploration and utilization remains balanced, preventing slow convergence due to overexploration and avoiding getting trapped in local optima by prematurely utilizing known strategies. For example, when the initial exploration rate is set to 0.7, the utilization rate automatically adjusts to 0.3. As training progresses, the exploration rate can decrease to 0.1, while the utilization rate increases to 0.9, achieving a smooth transition from extensive exploration to precise utilization.

[0061] Compared to related technologies, traditional Q-Learning methods, which use a fixed exploration rate, suffer from insufficient exploration in the early stages of training and over-utilization in the later stages, failing to meet the demands of complex obstacle environments. This approach overcomes the contradiction between exploration efficiency and convergence speed under fixed-parameter strategies by dynamically adjusting the complementary mechanism of exploration and utilization rates, automatically optimizing the weight allocation of exploration and utilization at different training stages.

[0062] In one instance, such as Figure 3 The diagram shows the overall flowchart of the path planning method. Specifically, it includes: Step 1, the process initiation point, marks the official start of the agent path planning training.

[0063] Step 2: Initialize the environment, Q table, and parameters.

[0064] Environment initialization: Construct a raster map, 20×20 grid, and define three key areas: obstacle area, starting point S, and target point T.

[0065] Q-table initialization: The Q-table is used to store the value estimates of state-action pairs. Here, the Q-values ​​corresponding to all states (grid positions, a total of 20×20=400) and actions (up, down, left, right directions) are initialized to 0, indicating that the agent has no prior knowledge of the value of performing an action at a certain position.

[0066] Parameter initialization: Set the core parameters for algorithm operation, including initial learning rate, final learning rate, initial exploration rate, discount factor, MCTS simulation steps, iteration control parameters, maximum number of iterations, etc.

[0067] Step 3: Begin Training. This marks the start of the iterative training loop, where the agent begins interacting with and learning from the environment. Training terminates when the maximum number of iterations is reached. If the current iteration count has reached the preset maximum, the process ends, training terminates, and the optimal path explored by the agent within the maximum number of iterations is output. If not, the process continues iterating to the next step.

[0068] Step 4: Adjust the dynamic learning rate and exploration rate.

[0069] By dynamically adjusting the learning rate and exploration rate, the slow convergence and poor stability issues caused by fixed learning and exploration rates in traditional Q-Learning are resolved. Refer to the above explanation for the dynamic learning rate and exploration rate adjustment scheme.

[0070] Step 5: Agent action selection.

[0071] A greedy strategy is used to balance exploration and utilization. An action (up, down, left, right) is randomly selected based on the current exploration rate to explore under-recognized areas of the environment, preventing the agent from getting trapped in local optima. Using 1 minus the current exploration rate, the action with the highest Q-value in the current state is selected. Leveraging learned Q-table knowledge, the action with the highest perceived value is prioritized, improving path planning efficiency.

[0072] Step 6: Perform the action and observe the results.

[0073] The agent performs a selected action within a grid environment, moving from the current state to a new state and observing the environmental feedback after the movement (whether it collided with obstacles, whether it reached the target). Specifically, this includes: The process determines whether the agent touches an obstacle after moving to the new state. If so, a penalty is imposed, using a high negative reward to constrain the agent to avoid obstacles. The process then returns to the agent's action selection, where a new action is decided (because collisions are invalid moves and must be retried). If not, the process proceeds to the next check: whether the target has been reached. If yes, the process ends, and the agent has successfully completed path planning. If no, the agent neither collides nor reaches the target, and enters the MCTS simulation phase, using global search to compensate for the local optimization deficiencies of Q-Learning.

[0074] Step 7, MCTS simulation.

[0075] Starting from the current state, a series of state-action nodes are generated by simulating potential paths for the next 10 steps. For each simulated state, the immediate reward is calculated, and then the cumulative discounted reward of the simulated path is calculated, providing a global revenue perspective for updating the Q-table.

[0076] Step 8, Q value update.

[0077] Taking into account cumulative discount rewards, the Q value is adjusted according to the following formula: ,in, State at time t Next action Q value, For learning rate, For instant rewards, As a discount factor, In order to perform the action The new state after transition The maximum value of Q among all possible actions. To accumulate discount rewards.

[0078] Step 9: Update the iteration count.

[0079] Increment the current iteration count by 1, then return to whether the maximum number of iterations has been reached, and start the next round of training, forming an iterative closed loop.

[0080] like Figure 4 The diagram shows the effect of the path planning method. This application effectively solves the problem of slow convergence speed caused by the imbalance between exploration and utilization in path planning. It shortens the training cycle by dynamically decaying the exploration rate; improves obstacle avoidance efficiency by actively avoiding areas with dense obstacles by utilizing the high exploration rate in the early stage; and enhances global optimization capability by maintaining the balance between global search and local optimization through a probabilistic complementary mechanism to avoid getting stuck in local optimal paths.

[0081] In intelligent warehouse management systems, improved Q-Learning-based agent path planning methods can be applied to path optimization for Automated Guided Vehicles (AGVs). In large warehouses, AGVs need to complete goods transportation tasks quickly and efficiently in complex rack layouts and dynamically changing environments. However, in practical applications, the storage locations of goods within the warehouse frequently change, requiring AGVs to frequently update their path planning strategies. Traditional path planning methods struggle to adapt to this highly dynamic environment, and while the improved Q-Learning method enhances adaptability to some extent, it still presents a unique problem: when significant changes occur in the warehouse layout (such as rack rearrangement or the addition of temporary obstacles), AGVs may require a considerable amount of time to adapt to the new environment and find the optimal path. This delay in adaptation can lead to a temporary decrease in warehouse efficiency, impacting overall operational efficiency.

[0082] To address the adaptability issue of AGVs when significant changes occur in warehouse layout, a fast relearning mechanism based on region segmentation is introduced, building upon the existing improved Q-Learning method. The specific implementation plan is as follows: Region segmentation: The entire warehouse space is divided into several relatively independent regions, each with its own Q-value table. This allows large-scale learning problems to be broken down into multiple smaller problems, improving learning efficiency.

[0083] Change Detection: Design a real-time environmental monitoring system to continuously monitor the warehouse environment using sensor networks or computer vision technology. When a significant change is detected in an area (such as a moving shelf or the addition of a new obstacle), that area is immediately marked as "to be relearned".

[0084] Local relearning: For areas marked as "to be relearned," the AGV will trigger a rapid relearning process when entering the area. During the relearning process, the AGV will temporarily increase its learning rate and exploration rate, focusing its efforts on exploring new environmental structures.

[0085] Knowledge Transfer: To accelerate the relearning process, a knowledge transfer mechanism is introduced. The system analyzes the environmental structure before and after the change, identifies the parts that remain unchanged, and transfers the Q-values ​​of these parts as initial values ​​to the new Q-value table. This avoids learning from scratch and significantly shortens the adaptation time.

[0086] Parallel learning: This method utilizes multiple AGVs to explore and learn simultaneously in different areas, sharing the learning outcomes through a central control system. This distributed learning approach allows for rapid accumulation of experience and accelerates the overall adaptation process to the environment.

[0087] Dynamic path planning: During the relearning process, the AGV dynamically adjusts its path planning strategy based on known environmental information and new knowledge learned in real time. This ensures that the AGV maintains high work efficiency during the learning process.

[0088] Through this region-segmentation-based rapid relearning mechanism, AGVs can quickly adapt to new environments and find optimal paths when significant changes occur in warehouse layout. This method retains the advantages of the original improved Q-Learning method while significantly enhancing the system's adaptability to environmental changes, thereby minimizing fluctuations in warehouse efficiency and ensuring the continuity and efficiency of overall operations.

[0089] This embodiment provides a path planning device, such as Figure 5 As shown, the path planning device includes: The adjustment module 501 is used to dynamically adjust the learning rate and exploration rate in the Q-learning model based on the Q-learning model, wherein the dynamic adjustment is used to characterize the changes in the learning rate and exploration rate under different iteration numbers.

[0090] The acquisition module 502 is used to simulate path selection in the Q-learning model based on Monte Carlo tree search and obtain the accumulated discount reward of the simulated path.

[0091] The correction module 503 is used to correct the Q-learning model based on the accumulated discount reward to obtain a path planning correction model.

[0092] The planning module 504 is used to plan the path of the agent based on the path planning correction model.

[0093] Specifically, the adjustment module sets a high exploration rate in the early stages of training to encourage the agent to extensively explore environmental information. As the number of iterations increases, the exploration rate is gradually reduced, causing the system to utilize already acquired high-value path information. The acquisition module uses Monte Carlo tree search to perform multi-step path deduction on the current state. When encountering areas with densely distributed obstacles, it simulates the subsequent path states of different turning actions and calculates the long-term benefit indicators for each path. The correction module combines the global benefit evaluation results obtained from Monte Carlo simulation with the immediate reward of Q-Learning, considering both the short-term benefit of the current action and the long-term benefit of the simulated path when updating the Q-value, forming a dual optimization objective. The planning module, based on the dynamically adjusted exploration rate parameter, balances the exploration of new paths with the utilization of known high-yield paths during path selection, dynamically adjusting the travel strategy in complex terrain.

[0094] In some specific implementations, when the agent enters a narrow passage area, the Monte Carlo tree search module initiates multi-path simulation to predict the subsequent states of three actions: going straight, turning left, or turning right. This is achieved by calculating the values ​​of each path. The system detects that a straight-line movement might encounter moving obstacles in the third step, thus reducing the weight of that direction during the Q-value correction phase. Based on the real-time updated exploration rate parameter, the planning module prioritizes right turns at the entrance of the passage to avoid potential risks.

[0095] Compared to related technologies, traditional Q-Learning devices employ a fixed exploration strategy, which is prone to path planning failure due to local optima in complex environments. This solution achieves phased optimization of the training process through a dynamic parameter adjustment module. A high exploration rate in the early stages avoids missing critical path information, while a low exploration rate in the later stages improves convergence efficiency. The introduction of a Monte Carlo tree search module overcomes the limitations of single-step decision-making, identifying obstacle risks in advance through multi-step path simulation—a forward-looking evaluation mechanism lacking in traditional devices. The correction module creatively integrates global path evaluation metrics into the Q-value update formula, forming a dual optimization mechanism combining short-term gains and long-term planning—a technical feature not found in traditional Q-Learning devices.

[0096] Through the above technical solutions, this application solves the problem of slow convergence speed in traditional methods, achieving a phased improvement in training efficiency through dynamic parameter adjustment; it improves the deficiency of low obstacle avoidance efficiency by using Monte Carlo simulation to avoid dense obstacle areas in advance; and it enhances global optimization capabilities by avoiding getting trapped in local optima through multi-step path evaluation and a dual feedback mechanism. The collaborative work of the various modules of the device enables the agent to quickly generate safe and efficient movement paths in complex dynamic environments.

[0097] The path planning apparatus provided in this disclosure can execute the path planning method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.

[0098] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0099] The following is a detailed reference. Figure 6 This diagram illustrates a suitable structural design for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 601, which can perform various appropriate actions and processes based on a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0100] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0101] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a memory 608, or installed from a ROM 602. When the computer program is executed by the processor 601, it performs the functions defined in the path planning method of the embodiments of the present invention.

[0102] Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0103] This disclosure also provides a computer-readable storage medium in which the methods described in this disclosure can be implemented in hardware or firmware, or implemented as recordable on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and subsequently stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the path planning method shown in the above embodiments.

[0104] A portion of this disclosure can be applied to computer program products, such as computer program instructions, which, when executed by a computer, can invoke or provide methods and / or technical solutions according to this disclosure through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, and installation package files. Accordingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions; the computer compiling the instructions and then executing the corresponding compiled program; the computer reading and executing the instructions; or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0105] Although embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A path planning method characterized by, The method comprises: Based on the Q learning model, the learning rate and the exploration rate in the Q learning model are dynamically adjusted, wherein the dynamic adjustment is used to represent the change of the learning rate and the exploration rate under different iteration numbers; Based on the Monte Carlo tree search, the path selection in the Q learning model is simulated to obtain the accumulated discount reward of the simulated path; Based on the accumulated discount reward, the Q learning model is corrected to obtain a path planning correction model; Based on the path planning correction model, the path of the intelligent agent is planned.

2. The method of claim 1, wherein, The method comprises: Based on the Q table in the Q learning model, the preselected action in the Q learning model is predicted by a preset step to obtain the immediate reward of each step of the predicted action; According to the immediate reward of each step of the predicted action and the corresponding discount factor, the accumulated discount reward of the simulated path is obtained.

3. The method of claim 2, wherein, The accumulated discount reward of the simulated path is obtained according to the immediate reward of each step of the predicted action and the corresponding discount factor, which is realized by the following expression: where, is the accumulated discounted reward, is the discount factor for the ith step, 0 <1, is the immediate reward for the ith step in the simulated path, N is the total number of steps.

4. The method of claim 1, wherein, The Q learning model is corrected based on the accumulated discount reward, which is realized by the following expression: ， where, is the state at the current time t the action performed the Q-value of the action performed, is the learning rate, is the immediate reward, is the discount factor, is the new state reached after performing the action is the maximum value of the Q-values among all possible actions, is the cumulative discounted reward.​ 5. The method of claim 4, wherein, The immediate reward is realized by the following expression: ， wherein, is the Manhattan distance from the current state to the target.

6. The method of claim 1, wherein, The learning rate and the exploration rate in the Q learning model are dynamically adjusted, which comprises: The learning rate is dynamically adjusted according to the current iteration number and the maximum iteration number; The exploration rate is dynamically adjusted according to the current iteration number.

7. The method of claim 6, wherein, The learning rate is dynamically adjusted according to the current iteration number and the maximum iteration number, which is realized by the following expression: wherein, is the current learning rate, is the initial learning rate, is the final learning rate, is the maximum number of iterations, is the current iteration number, 0 ≤ 1 ;​ The exploration rate is dynamically adjusted according to the current iteration number, which is realized by the following expression: wherein, is the current exploration rate, is the initial exploration rate, is the decay coefficient, , is the current iteration number, 0 ≤ 1.

8. The method according to any one of claims 1 to 7, characterized in that, Based on the path planning correction model, the path of the intelligent agent is planned, which comprises: An arbitrary action is randomly selected at the current exploration rate, and an action with the maximum Q value in the current state is selected at the current utilization rate; wherein the current exploration rate and the current utilization rate are in a complementary relationship, and the sum of their probabilities is equal to 1.

9. A route planning apparatus characterized by comprising: The device comprises: An adjustment module for dynamically adjusting the learning rate and the exploration rate in the Q learning model based on the Q learning model, wherein the dynamic adjustment is used to represent the change of the learning rate and the exploration rate under different iteration numbers; An acquisition module for simulating the path selection in the Q learning model based on the Monte Carlo tree search to obtain the accumulated discount reward of the simulated path; A correction module for correcting the Q learning model based on the accumulated discount reward to obtain a path planning correction model; A planning module for planning the path of the intelligent agent based on the path planning correction model.

10. An electronic device, comprising: It comprises: A memory and a processor, which are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the path planning method in any one of claims 1 to 8.

Images