Mobile robot path planning method and system based on artificial lemming prior guidance Q learning

Abstract

The method comprises the following steps: performing artificial ant algorithm prior search based on a static grid map model to generate a path population composed of multiple candidate paths, and evaluating the fitness of the candidate paths; constructing a prior value map according to the path population; initializing an action value function of Q learning according to the prior value map and action direction consistency to obtain an initial Q table with prior direction bias; calculating the environmental complexity according to the statistical characteristics of the path population, and adaptively adjusting the epsilon-greedy exploration rate of Q learning based on the environmental complexity; in the Q learning training process, selecting an action based on the current state and performing state transition to obtain an immediate reward, and constructing a heuristic guide item according to the prior value map and the action direction consistency, and fusing the heuristic guide item into the action value function update process; repeatedly performing adaptive adjustment and training until convergence.

Description

Technical Field

[0001] This invention belongs to the field of mobile robot path planning technology, and particularly relates to a mobile robot path planning method and system based on artificial lemming prior-guided Q-learning. It is applicable to service robots, warehousing and logistics robots, inspection robots, cleaning robots, and other mobile robot systems that need to perform global path planning in complex static grid environments. Background Technology

[0002] Path planning for mobile robots is a core task in autonomous navigation systems. Its goal is to plan a feasible path from the starting position to the target position for the mobile robot, while satisfying obstacle avoidance and motion execution constraints, and optimizing path length, search time, training convergence speed, and policy stability as much as possible. As the application scenarios of mobile robots continue to expand, real-world environments often feature complex topologies such as dense obstacles, narrow passages, concave obstacles, and trap structures, placing higher demands on the global search capability, adaptability, and robustness of path planning algorithms.

[0003] Existing path planning methods mainly include traditional graph search algorithms, heuristic search algorithms, sampling planning algorithms, swarm intelligence optimization algorithms, and reinforcement learning algorithms. Traditional graph search algorithms and heuristic search algorithms can generate paths relatively quickly in regular, static environments with complete prior information. However, they typically rely on manually designed heuristic functions or fixed search rules, and are prone to problems such as high search costs, excessive path redundancy, and insufficient generalization ability when obstacles are irregularly distributed, the environment scale increases, or trap structures exist. Swarm intelligence optimization algorithms have strong global search capabilities and can discover high-quality candidate paths in complex search spaces. However, they usually lack the ability to learn long-term value for sequential decision problems, and their stability is insufficient when used alone for path planning.

[0004] Q-learning, a typical model-free reinforcement learning algorithm, learns the state-action value function through agent-environment interaction, thereby obtaining a better path strategy. Therefore, it is widely used in path planning for mobile robots in discrete grid environments. However, classic Q-learning still has significant shortcomings in complex path planning tasks: First, Q-tables are usually initialized with zero or random initialization, lacking structured prior information in the early training stages, making the agent prone to blind exploration in the grid environment; second, environmental rewards in path planning tasks are sparse, and value information propagates slowly in the state space, resulting in numerous convergence rounds; third, in environments with narrow passages, concave obstacles, or trap structures, the agent is prone to local oscillations or getting stuck on suboptimal paths; fourth, most existing improvement methods only optimize a part of the reward function, exploration rate, or local escape mechanism, lacking a unified prior guidance mechanism that can run through the entire process of Q-table initialization, exploration rate adjustment, and Q-value update.

[0005] The Artificial Lemming Algorithm (ALA) is a swarm intelligence optimization algorithm with search characteristics such as swarm migration, random perturbation, and local exploitation, making it suitable for generating a population of candidate paths with reference value in complex environments. However, some swarm intelligence path planning methods mainly rely on a single optimal individual to directly output the planned path, or only use a single candidate path to guide the learning process once, failing to fully utilize the access frequency, path quality, and spatial distribution information reflected by multiple candidate paths in the path population. If the ALA is used only as an independent path optimizer, its search results are also difficult to continuously participate in the exploration rate adjustment and action value update of Q-learning. Therefore, how to transform the multi-path search information obtained by the ALA into a structured prior suitable for Q-learning, and how to make this prior permeate the Q-table initialization, exploration control, and action value update processes, is a technical problem that needs to be solved to improve the efficiency and stability of mobile robot path planning in complex static environments. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a mobile robot path planning method and system based on artificial lemming prior-guided Q-learning. This addresses the problems of blind exploration in the early stages of classic Q-learning training, slow value propagation due to sparse rewards, susceptibility to local optima in complex obstacle environments, and insufficient utilization of prior information in existing hybrid methods. This invention aims to combine the global prior search capability of the artificial lemming algorithm with the sequential decision-making learning capability of Q-learning. This allows the path population generated by the artificial lemming algorithm to not only be used in the path search phase but also transformed into a prior value map, continuously participating in Q-table initialization, exploration rate adjustment, and action value function updates during Q-learning.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] The first aspect is a mobile robot path planning method based on artificial lemming prior-guided Q-learning, which includes the following steps:

[0009] S1. Construct a static grid map model, set the starting position and target position of the mobile robot, and define the obstacle area, passable area and action space.

[0010] S2. Perform prior search using the artificial lemming algorithm based on the static grid map model to generate a path population consisting of multiple candidate paths, and evaluate the fitness of the candidate paths.

[0011] S3. Construct a prior value map based on the path population to characterize the potential access value of each grid state in the global path search;

[0012] S4. Initialize the action value function of Q-learning based on the prior value map and action direction consistency to obtain the initial Q-table with prior direction bias;

[0013] S5. Calculate the environmental complexity based on the statistical characteristics of the path population, and adaptively adjust the ε-greedy exploration rate of Q-learning based on the environmental complexity.

[0014] S6. During Q-learning training, select an action based on the current state and perform a state transition to obtain an immediate reward. At the same time, construct a heuristic guidance item based on the prior value map and the consistency of action direction, and integrate the heuristic guidance item into the action value function update process.

[0015] S7. Repeat S5 and S6 until the mobile robot reaches the target position, the training rounds reach the preset upper limit, or the preset convergence condition is met. Output a collision-free path from the starting position to the target position based on the trained Q table.

[0016] As a preferred option, fitness evaluation in S2 includes:

[0017] Perform a feasibility assessment on the candidate paths; set the fitness of candidate paths that are determined to be invalid to zero.

[0018] For valid candidate paths, calculate the fitness of the candidate paths:

[0019]

[0020] In the formula, Indicates the first The fitness of the candidate paths , and These are preset positive weighting coefficients, and Greater than , Greater than , For the first The arrival indication value for whether each candidate path reaches the target location; when the endpoint of a candidate path is the target location. Select 1 if the value is 1, otherwise select 0. For the first The path length of each candidate path, This represents the distance from the endpoint of the candidate path to the target location. This is a reference value for path length. This is a reference value for the destination distance;

[0021] After each iteration, candidate paths with zero fitness are removed, and the remaining candidate paths are sorted from highest to lowest fitness, retaining the top-performing paths. The candidate paths form a path population;

[0022] Based on path population The sum of the fitness of the candidate paths is related to the first... 10 candidate paths, based on their fitness Calculate fitness weights .

[0023] Preferably, S3 includes:

[0024] Traverse the candidate paths in the path population and count the access frequency of each accessible grid by the candidate path; for each grid, calculate the prior score based on its access frequency, the fitness weight of the candidate path, and the distance from the grid to the target state; normalize the prior scores of all accessible grids; perform neighborhood diffusion or Gaussian smoothing on the normalized prior scores to obtain a spatially continuous and stable prior value map.

[0025] As a preferred option, the prior score in S3 is calculated as follows:

[0026]

[0027] In the formula, Represents a grid Prior scores, Indicates the first Does each candidate path pass through a grid? The path indicator function, when the first A value of 1 is assigned if a candidate path passes through this grid cell, otherwise a value of 0 is assigned. This represents the fitness weight of the k-th candidate path; Represents a grid To the target state The Euclidean distance or Manhattan distance.

[0028] As a preferred option, S3 also includes:

[0029] Centered on the currently passable grid, the normalized prior scores of the surrounding passable grids are weighted and propagated within its neighborhood according to preset weights, while keeping obstacle grids out of the propagation process; after at least one diffusion or smoothing, a spatially continuous and stable prior value map is obtained.

[0030] As a preferred embodiment, in S4, the initial action value function is calculated as follows:

[0031]

[0032] In the formula, Representing state Next action The initial value of the action. Represents the successor state Prior value in a prior value map The score indicates the consistency of movement direction. and These are positive weighting coefficients, and each passable grid corresponds to a state in Q-learning.

[0033] As a preferred embodiment, S5 includes:

[0034] Calculate the average path length of the path population and the average distance from the endpoint of each candidate path to the target state; normalize the average path length and average distance to define the initial complexity; nonlinearly amplify the initial complexity to obtain the environmental complexity; based on the environmental complexity... The ε-greedy exploration rate is updated according to the following formula:

[0035]

[0036] In the formula, Indicates the first The exploration rate at each training moment or training round. and These represent the upper limit and lower limit of the exploration rate, respectively. This represents the exploration rate adjustment coefficient.

[0037] As a preferred embodiment, S6 includes:

[0038] In each training round, the mobile robot starts from the initial state and selects an action according to the adaptive ε-greedy policy in the current state; after executing the action, the environment returns the subsequent state and an immediate reward. ;

[0039] The heuristic guiding term is constructed based on the consistency score of successor states and directions as follows:

[0040]

[0041] In the formula, Indicates the state Execute action Heuristic guidance at the time Represents the successor state Prior value in a prior value map Indicates action From the current state Pointing to target state Directional consistency score and These represent the weight coefficients of the prior value guiding term and the directional consistency guiding term, respectively;

[0042] By incorporating heuristic guidance into the action value function update process of Q-learning, the following update formula is obtained:

[0043]

[0044] In the formula, Indicates the learning rate. Indicates the discount factor. Represents the successor state Optional actions below.

[0045] As a preferred embodiment, S7 includes:

[0046] After training, the mobile robot is reset to the starting position. In each state, the effective action with the largest Q value is selected as the next action. If multiple effective actions have the same Q value, the action with higher prior value and better directional consistency is selected, and finally a collision-free path from the starting position to the target position is generated.

[0047] The preset convergence condition adopts a sliding window determination method: within a sliding window consisting of W consecutive training rounds, the proportion of successful arrivals at the target position, the average length of successful paths, and the coefficient of variation of successful path length are statistically analyzed; when the proportion of successful arrivals at the target position within the sliding window is not lower than the preset success rate threshold, the coefficient of variation of successful path length is not higher than the preset stability threshold, and the average length of successful paths does not exceed a preset multiple of the current optimal path length, the current strategy is determined to have reached a stable convergence state.

[0048] Secondly, a mobile robot path planning system based on artificial lemming prior-guided Q-learning includes:

[0049] The gridding module is used to build a static grid map model, set the starting position and target position of the mobile robot, and define the obstacle area, passable area and action space.

[0050] The artificial lemming prior search module is used to perform prior search of the artificial lemming algorithm based on a static grid map model, generate a path population consisting of multiple candidate paths, and evaluate the fitness of the candidate paths.

[0051] The prior value map construction module is used to construct a prior value map based on the path population to characterize the potential access value of each grid state in global path search.

[0052] The Q-table initialization module is used to initialize the action value function of Q-learning based on the prior value map and action direction consistency, so as to obtain an initial Q-table with prior direction bias.

[0053] The adaptive exploration module is used to calculate the environmental complexity based on the statistical characteristics of the path population, and adaptively adjust the ε-greedy exploration rate of Q-learning based on the environmental complexity.

[0054] The prior guidance update module is used to select an action based on the current state and perform a state transition during Q-learning training to obtain an immediate reward. At the same time, it constructs a heuristic guidance term based on the prior value map and the consistency of action direction, and integrates the heuristic guidance term into the action value function update process.

[0055] The path output module is used to output a collision-free path from the starting position to the target position based on the trained Q-table;

[0056] The mobile robot path planning system based on artificial lemming prior-guided Q-learning is used to implement the mobile robot path planning method and steps based on artificial lemming prior-guided Q-learning as described in the first aspect.

[0057] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0058] This invention uses the artificial lemming algorithm as a prior searcher before Q-learning training, rather than directly using the single optimal path obtained by the artificial lemming algorithm as the final planning result. Multiple candidate paths are generated through artificial lemming population search, and the global spatial information contained in the path population is transformed into a structured prior that Q-learning can utilize. The artificial lemming algorithm is responsible for global path structure search, while Q-learning is responsible for iterating action value and correcting policies based on environmental feedback, thereby reducing ineffective exploration caused by a lack of effective guidance in the early stages of Q-learning training.

[0059] In constructing the prior value map, this invention utilizes the frequency of candidate paths accessing grids to reflect the common selection trend of the path population towards traversable areas. It leverages the fitness weights of candidate paths to enhance the contribution of high-quality candidate paths and reduce the interference of low-quality candidate paths. The distance from a grid to the target state imparts a priori scoring target proximity. Through normalization and neighborhood diffusion or Gaussian smoothing, discrete path information is transformed into spatially continuous value guidance. Simultaneously, obstacle grids are excluded from value propagation to prevent prior value from diffusing across obstacle regions. This results in a prior value map that simultaneously reflects the global search trend of the path population, candidate path quality, target direction, and obstacle constraints.

[0060] This invention utilizes prior value maps and action direction consistency to initialize the Q-table, so that the initial state of the Q-table is no longer zero or random, but has a directional bias toward the target region and high-potential path region, thereby improving the path search efficiency in the early stage of learning.

[0061] This invention calculates environmental complexity based on the statistical characteristics of the path population generated by the artificial lemming algorithm, and dynamically adjusts the ε-greedy exploration rate based on the environmental complexity. It maintains strong exploration capability in complex environments, reducing the risk of prematurely getting trapped in local optima; and accelerates exploration rate decay in simple environments, improving utilization efficiency.

[0062] This invention introduces prior value guidance terms and direction consistency guidance terms into the action value function update at each step of Q-learning, enabling the prior information provided by the artificial lemming algorithm to continuously participate in action value updates. The aforementioned artificial lemming prior search, prior value graph construction, Q-table initialization, adaptive exploration rate adjustment, and prior guidance update are interconnected: the path population generated by the artificial lemming algorithm is used, on the one hand, to construct the prior value graph, providing global spatial guidance for Q-table initialization and action value updates; on the other hand, it is used to calculate environmental complexity, providing a basis for exploration rate adjustment. This forms a continuous guidance link from global prior generation and injection to policy iteration, alleviating the problem of slow value propagation under sparse reward conditions and improving policy convergence speed and stability.

[0063] This invention achieves shorter paths and faster convergence speeds in random obstacle environments, structured obstacle environments, and trap-structured environments. Simulation results show that, compared to classical Q-learning, this invention reduces the average search time by approximately 80.58%, the average number of convergence rounds by approximately 72.93%, and the average path length by approximately 10.06% in a 50×50 environment with an obstacle density of 30%. In a trap-structured environment, the average path length is reduced by approximately 12.97%, and the average search time is reduced by approximately 78.99%, demonstrating good adaptability to complex environments. Attached Figure Description

[0064] Figure 1 This is a flowchart of the method in Embodiment 1 of the present invention;

[0065] Figure 2 This is a schematic diagram of a static grid map model according to Embodiment 1 of the present invention;

[0066] Figure 3 This is a schematic diagram of the eight-directional action space and correct path in Embodiment 1 of the present invention;

[0067] Figure 4 This is a flowchart of the prior search process of the artificial lemming algorithm in Embodiment 1 of the present invention;

[0068] Figure 5 This is a schematic diagram of the prior value map construction and Q-table initialization process in Embodiment 1 of the present invention;

[0069] Figure 6 This is a schematic diagram of merging six test maps according to Embodiment 1 of the present invention;

[0070] Figure 7 This is a schematic diagram comparing the paths generated by six test map algorithms in Embodiment 1 of the present invention;

[0071] Figure 8 This is a schematic diagram illustrating the improved path planning effect under different environments in Embodiment 1 of the present invention;

[0072] Figure 9 This is a schematic diagram of the system structure of Embodiment 2 of the present invention. Detailed Implementation

[0073] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0074] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0075] Example 1:

[0076] like Figure 1 The method for mobile robot path planning based on artificial lemming prior-guided Q-learning, as shown, includes the following steps:

[0077] Step 1: Construct a static grid map model, set the starting position and target position of the mobile robot, and define the obstacle area, passable area and action space.

[0078] Specifically, the steps for constructing a static raster map model include:

[0079] (1) Determine the working area and grid resolution of the mobile robot. The working area represents the operating range of the mobile robot, and the grid resolution is used to determine the map division accuracy. The resolution can be set according to the task requirements to balance map accuracy and computational cost.

[0080] (2) Based on the grid resolution, the working area is divided into an M×N two-dimensional grid matrix. Each grid represents the smallest identifiable unit in the environment and is used to record the status information of obstacles or passage areas.

[0081] (3) Obtain obstacle information within the environment. In the static grid map model, the obstacle positions remain unchanged during training and planning.

[0082] (4) Assign a state value to each grid. Unobstructed grids are marked as passable, and grids containing obstacles are marked as impassable, forming a complete environment map. Each passable grid corresponds to a state in Q-learning, and the grid where the starting position is located corresponds to the initial state. The grid cell where the target location is located corresponds to the target state. .like Figure 2 As shown, the static grid map includes passable grids, obstacle grids, initial state, and target state; the mobile robot performs state transitions between passable grids, and obstacle grids, as impassable areas, participate in effective action determination and path collision constraints.

[0083] like Figure 3 As shown, the mobile robot's motion space uses an eight-directional motion set, including up, down, left, right, upper left, upper right, lower left, and lower right. If the current state is... Select Action Then, the candidate successor state is When a candidate successor state exceeds the map boundary or is an obstacle state, the action is deemed invalid. For diagonal actions, it is further determined whether the two orthogonal grids adjacent to the diagonal direction are both passable; if at least one is impassable, the diagonal action is prohibited to prevent the mobile robot from crossing obstacle corners.

[0084] Step 2: Perform prior search using the artificial lemming algorithm based on the static grid map model to generate a path population consisting of multiple candidate paths, and evaluate and filter the candidate paths based on their fitness and calculate their weights.

[0085] like Figure 4 As shown, before formal training in Q-learning, a path population consisting of multiple lemming individuals is initialized, with each lemming individual representing a candidate path or a set of key nodes for candidate paths. The lemming individuals in the path population are iteratively updated through migration updates, random perturbations, and local refinement operations. After each iteration, fitness evaluation and optimal path retention are performed, allowing the candidate paths to gradually approach the target location. Finally, the candidate path population used to construct the prior value map is output.

[0086] Furthermore, a fitness evaluation is performed on each candidate path. First, the feasibility of the candidate path is determined: if a candidate path contains boundary-crossing grids, obstacle grids, or adjacent path nodes do not satisfy the eight-directional adjacency relationship, then the candidate path is determined to be an invalid path, and its fitness is set to zero. For valid candidate paths, let the... The arrival indication value for whether each candidate path reaches the target location is... When the endpoint of the candidate path is the target location, Select 1 if the value is 1, otherwise select 0; record the first... The path length of the candidate paths is The distance from the candidate path endpoint to the target location is Within the current path population, the path length is used as a reference value. Reference value for distance to the destination Normalize the path length and destination distance, and calculate the fitness of the candidate paths according to the following formula:

[0087]

[0088] In the formula, Indicates the first The fitness of the candidate paths , and These are preset positive weighting coefficients, and Greater than , Greater than This is used to prioritize feasible paths to the target location, and among candidate paths to the target location, priority is given to those shorter and closer to the target location. At the end of each iteration, candidate paths with zero fitness are removed, and the remaining candidate paths are sorted from highest to lowest fitness, retaining the top-fittest paths. The candidate paths form a path population used to construct the prior value map. .

[0089] For the first in the preferred path population 10 candidate paths, based on their fitness Calculate fitness weights The calculation formula is:

[0090]

[0091] In the formula, Indicating the preferred path population The sum of the fitness of each candidate path. The sum of the fitness weights of each candidate path is 1. The higher the fitness of a candidate path, the greater its corresponding fitness weight, and the greater its contribution to the subsequent prior score calculation.

[0092] Step 3: Construct a prior value map based on the path population. The prior value map is used to characterize the potential accessibility value of each grid state in global path search. For example... Figure 5 As shown, the candidate path population generated by the artificial lemming algorithm is first subjected to access frequency statistics, fitness weighting and target distance processing to construct a spatially continuous prior value map; then, the Q table is initialized according to the prior value of the successor state of effective actions and the consistency of action direction, so that Q learning has an initial value bias towards high-potential passage areas and target directions in the early stage of training.

[0093] Specifically, the candidate paths in the path population are traversed, and the access frequency of each accessible raster traversed by a candidate path is counted. For a raster... The prior score is calculated based on the frequency of access, the fitness weight of the candidate path, and the distance from the grid to the target state:

[0094]

[0095] In the formula, Represents a grid Prior scores, Indicates the first Does each candidate path pass through a grid? The path indicator function, when the first A value of 1 is assigned if a candidate path passes through this grid cell, otherwise a value of 0 is assigned. Indicates the first The fitness weights of the candidate paths; Represents a grid To the target state The Euclidean distance or Manhattan distance.

[0096] Furthermore, the prior scores of all passable graticles are normalized:

[0097]

[0098] In the formula, Represents a grid Normalized prior scores, This represents the maximum prior score among all passable grid cells. To prevent positive numbers from being divided by zero, their values ​​are much smaller than 0. This is used to ensure that the calculation remains stable when the path population does not cover a valid raster or the maximum score is zero. After normalization, the prior score of each accessible raster is mapped to the range of 0 to 1, with a larger value indicating that the raster is more likely to be in a high-quality candidate path region.

[0099] Then, neighborhood diffusion or Gaussian smoothing is performed on the normalized prior scores. Specifically, centered on the currently accessible grid, the normalized scores of surrounding accessible grids are weighted and propagated within its neighborhood according to preset weights, while obstacle grids are excluded from propagation to prevent prior values ​​from spreading across obstacles. After at least one diffusion or smoothing operation, a spatially continuous and stable prior value map is obtained. ,in Represents a grid The final prior value. Through this prior value map, grids that frequently appear in high-quality candidate paths and are close to the target location are given higher prior values, and adjacent passable areas are also given continuous value guidance, providing global structural guidance for subsequent Q-learning training.

[0100] Step 4: Initialize the action value function of Q-learning based on the prior value map and action direction consistency to obtain the initial Q-table with prior direction bias.

[0101] Specifically, regarding the state and actions First calculate the action to be performed. The subsequent candidate successor state If the action For actions that cross boundaries, collide with other actions, or are prohibited diagonal actions, the corresponding action value is set to an unselectable state or a preset negative value. If the action is valid, the action direction vector is recorded as... , the current state Pointing to target state The target direction vector is denoted as And calculate the directional consistency score based on the cosine similarity of the two directional vectors:

[0102]

[0103] In the formula, This represents the dot product of the action direction vector and the target direction vector. and These represent the magnitudes of the two direction vectors, respectively. The value ranges from 0 to 1, with a larger value indicating a more consistent action direction with the target direction; current state When the target state is reached, subsequent action selection will not be performed. This is used during the Q-table initialization phase. The directional consistency score is used during the Q-learning training phase. The score represents the consistency score in terms of direction, and both are calculated using the same method.

[0104] Furthermore, the initial action value function is calculated according to the following formula:

[0105]

[0106] In the formula, Representing state Next action The initial value of the action. Represents the successor state Prior value in a prior value map The score indicates the consistency of movement direction. and These are positive weighting coefficients.

[0107] Through the above initialization method, the initial value of the Q table simultaneously includes the global path prior and the direction preference of the action toward the target position provided by the artificial lemming algorithm, avoiding the blind exploration caused by the classic Q learning starting from zero or random values.

[0108] Step 5: Calculate the environmental complexity based on the statistical characteristics of the path population, and adaptively adjust the ε-greedy exploration rate of Q-learning based on the environmental complexity.

[0109] Specifically, first calculate the average path length of the path population:

[0110]

[0111] In the formula, Indicates the first The path length of each candidate path.

[0112] Secondly, calculate the average distance from the endpoint of each candidate path in the path population to the target state:

[0113]

[0114] In the formula, Indicates the first The endpoint of each candidate path.

[0115] Furthermore, regarding the average path length and average distance Normalize each part separately and calculate the initial complexity:

[0116]

[0117] In the formula, and These are the normalized reference values.

[0118] To enhance the differentiation between environments of varying difficulty, the initial complexity was adjusted. By performing nonlinear amplification, the environmental complexity is obtained:

[0119]

[0120] In the formula, The complexity is amplified by an exponent.

[0121] Based on environmental complexity The ε-greedy exploration rate is updated according to the following formula:

[0122]

[0123] In the formula, Indicates the first The exploration rate at each training moment or training round. and These represent the upper limit and lower limit of the exploration rate, respectively. This represents the exploration rate adjustment coefficient. It is applied when the environmental complexity... When the complexity is high, the exploration rate decays more slowly, allowing the mobile robot to maintain a strong exploration capability in complex environments; when the environmental complexity is high... When the rate is smaller, the exploration rate decays faster, allowing the learning process to transition to the utilization phase earlier.

[0124] Step 6: During Q-learning training, select an action based on the current state and perform a state transition to obtain an immediate reward. At the same time, construct a heuristic guide based on the prior value map and the consistency of action direction, and integrate the heuristic guide into the action value function update process.

[0125] Specifically, in each training round, the mobile robot starts from the initial state Departure, in the current state The action is selected according to the adaptive ε-greedy strategy. Execute the action Then, the environment returns to the successor state. and instant rewards Instant rewards This can include positive rewards for reaching the target location, negative penalties for collisions or going out of bounds, and step costs for normal movement.

[0126] Furthermore, construct heuristic guides:

[0127]

[0128] In the formula, Indicates the state Execute action Heuristic guidance at the time Represents the successor state Prior value in a prior value map Indicates action From the current state Pointing to target state Directional consistency score and These represent the weight coefficients of the prior value guiding term and the directional consistency guiding term, respectively.

[0129] By incorporating heuristic guidance into the action value function update process of Q-learning, the following update formula is obtained:

[0130]

[0131] In the formula, Indicates the learning rate. Indicates the discount factor. Represents the successor state The optional actions are described above. Through this update method, the action value function is not only determined by the immediate environmental reward and future return, but is also continuously guided by the prior value of the artificial lemming algorithm and the consistency of the target direction, thereby improving the speed of value propagation and the stability of the strategy.

[0132] Step 7: Repeat steps 5 and 6 until the mobile robot reaches the target position, the training rounds reach the preset upper limit, or the preset convergence condition is met. Output the collision-free path from the starting position to the target position based on the trained Q-table.

[0133] Specifically, after training, the mobile robot is reset to its starting position, and in each state, the effective action with the highest Q-value is selected as the next action. If multiple effective actions have the same Q-value, the action with higher prior value and better directional consistency is prioritized. Finally, a collision-free path from the starting position to the target position is generated.

[0134] Furthermore, the preset convergence condition can be determined using a sliding window method. Within a sliding window consisting of W consecutive training rounds, the proportion of successful arrivals at the target location, the average length of successful paths, and the coefficient of variation of successful path lengths are statistically analyzed. When the success rate within the sliding window is not lower than a preset success rate threshold, the coefficient of variation of successful path lengths is not higher than a preset stability threshold, and the average length of successful paths does not exceed a preset multiple of the current optimal path length, the current strategy is determined to have reached a stable convergence state.

[0135] The following description further illustrates the application of the method of this invention in path planning and the simulation results.

[0136] 1. Environment and Parameter Settings: The simulation environment of this invention is constructed using a rasterization method, such as... Figure 6 As shown in the figure, this embodiment sets up six test maps to verify the path planning adaptability of the present invention under different map sizes, obstacle densities, and obstacle topologies. The specific test environment is shown in Table 1, and the main parameters of the algorithm are shown in Table 2.

[0137] Table 1: Specific Test Environment

[0138]

[0139] Table 2 Main Algorithm Parameters

[0140]

[0141] 2. Simulation Results and Analysis: To verify the effectiveness of the proposed method, it was compared with the classical Q-learning algorithm under the same map, starting position, and target position conditions. Evaluation metrics included path length, search time, and convergence epochs. Table 3 shows the performance improvement of the proposed method compared to the classical Q-learning algorithm in various test environments.

[0142] Table 3 shows the performance improvement of this invention compared to the classic Q-learning algorithm.

[0143]

[0144] like Figure 7 As shown in the figure, the path planning results of the classic Q-learning algorithm and the present invention are compared on six test maps, which is used to intuitively reflect the path search situation under different obstacle distributions, structured obstacles, and trap structures. Figure 8 As shown in the figure, the improvements of this invention over the classic Q-learning algorithm in terms of path length, search time, and convergence rounds under different test environments are summarized. Combined with Table 3, it can be seen that this invention exhibits better overall planning performance under different map sizes and obstacle complexities. Especially in large-scale random obstacle environments and trap structure environments, it can reduce search time and training convergence rounds, demonstrating better path planning efficiency and stability in complex environments.

[0145] The above description is merely a preferred embodiment of the present invention, used to illustrate the technical solution of the present invention, but is not intended to limit this application. The scope of protection of the present invention is not limited to the above embodiments. All equivalent substitutions or modifications made based on the technical concept of the present invention should be included within the scope of protection of the present invention.

[0146] Example 2:

[0147] Corresponding to the aforementioned embodiment of a mobile robot path planning method based on artificial lemming prior-guided Q-learning, this application also provides an embodiment of a mobile robot path planning system based on artificial lemming prior-guided Q-learning.

[0148] Figure 9 This is a schematic diagram of an embodiment of a mobile robot path planning system based on artificial lemming prior-guided Q-learning, provided in this application. Figure 9 As shown, the system provided in this embodiment includes a gridding module, an artificial lemming prior search module, a prior value graph construction module, a Q-table initialization module, an adaptive exploration module, a prior guidance update module, and a path output module; wherein,

[0149] The gridding module is used to build a static grid map model, set the starting and target positions of the mobile robot, and define obstacle areas, passable areas, and action space.

[0150] The Artificial Lemming Prior Search Module is used to perform prior search of the Artificial Lemming Algorithm based on a static grid map model, generate a candidate path population, and calculate the fitness of the candidate paths.

[0151] The prior value graph construction module is used to construct a prior value graph based on the candidate path access frequency, candidate path fitness weight, and target distance.

[0152] The Q-table initialization module is used to initialize the action value function learned by Q based on the prior value map and the consistency of action direction.

[0153] The adaptive exploration module is used to calculate the environmental complexity based on the statistical characteristics of the path population and adaptively adjust the ε-greedy exploration rate based on the environmental complexity.

[0154] The prior guidance update module is used to construct heuristic guidance terms during Q-learning training and integrate these heuristic guidance terms into the action value function update process.

[0155] The path output module is used to output a collision-free path from the starting position to the target position based on the trained Q-table.

Claims

1. A mobile robot path planning method based on artificial lemming prior-guided Q-learning, characterized in that, Includes the following steps: S1. Construct a static grid map model, set the starting position and target position of the mobile robot, and define the obstacle area, passable area and action space. S2. Perform prior search using the artificial lemming algorithm based on the static grid map model to generate a path population consisting of multiple candidate paths, and evaluate the fitness of the candidate paths. S3. Construct a prior value map based on the path population to characterize the potential access value of each grid state in the global path search; S4. Initialize the action value function of Q-learning based on the prior value map and action direction consistency to obtain the initial Q-table with prior direction bias; S5. Calculate the environmental complexity based on the statistical characteristics of the path population, and adaptively adjust the ε-greedy exploration rate of Q-learning based on the environmental complexity. S6. During Q-learning training, select an action based on the current state and perform a state transition to obtain an immediate reward. At the same time, construct a heuristic guidance item based on the prior value map and the consistency of action direction, and integrate the heuristic guidance item into the action value function update process. S7. Repeat S5 and S6 until the mobile robot reaches the target position, the training rounds reach the preset upper limit, or the preset convergence condition is met. Output a collision-free path from the starting position to the target position based on the trained Q table.

2. The mobile robot path planning method based on artificial lemming prior-guided Q-learning according to claim 1, characterized in that, Fitness evaluation in S2 includes: Perform a feasibility assessment on the candidate paths; set the fitness of candidate paths that are determined to be invalid to zero. For valid candidate paths, calculate the fitness of the candidate paths: ； In the formula, Indicates the first The fitness of the candidate paths , and These are preset positive weighting coefficients, and Greater than , Greater than , For the first The arrival indication value for whether each candidate path reaches the target location; when the endpoint of a candidate path is the target location. Select 1 if the value is 1, otherwise select 0. For the first The path length of each candidate path, This represents the distance from the endpoint of the candidate path to the target location. This is a reference value for path length. This is a reference value for the destination distance; After each iteration, candidate paths with zero fitness are removed, and the remaining candidate paths are sorted from highest to lowest fitness, retaining the top-performing paths. The candidate paths form a path population; Based on path population The sum of the fitness of the candidate paths is related to the first... 10 candidate paths, based on their fitness Calculate fitness weights .

3. The mobile robot path planning method based on artificial lemming prior-guided Q-learning according to claim 1, characterized in that, S3 includes: Traverse the candidate paths in the path population and count the access frequency of each accessible grid by the candidate path; for each grid, calculate the prior score based on its access frequency, the fitness weight of the candidate path, and the distance from the grid to the target state; normalize the prior scores of all accessible grids; perform neighborhood diffusion or Gaussian smoothing on the normalized prior scores to obtain a spatially continuous and stable prior value map.

4. The mobile robot path planning method based on artificial lemming prior-guided Q-learning according to claim 3, characterized in that, The prior score in S3 is calculated as follows: ； In the formula, Represents a grid Prior scores, Indicates the first Does each candidate path pass through a grid? The path indicator function, when the first A value of 1 is assigned if a candidate path passes through this grid cell, otherwise a value of 0 is assigned. This represents the fitness weight of the k-th candidate path; Represents a grid To the target state The Euclidean distance or Manhattan distance.

5. The mobile robot path planning method based on artificial lemming prior-guided Q-learning according to claim 3, characterized in that, S3 also includes: Centered on the currently passable grid, the normalized prior scores of the surrounding passable grids are weighted and propagated within its neighborhood according to preset weights, while keeping obstacle grids out of the propagation process; after at least one diffusion or smoothing, a spatially continuous and stable prior value map is obtained.

6. The mobile robot path planning method based on artificial lemming prior-guided Q-learning according to claim 1, characterized in that, In S4, the initial action value function is calculated as follows: ； In the formula, Representing state Next action The initial value of the action. Represents the successor state Prior value in a prior value map The score indicates the consistency of movement direction. and These are positive weighting coefficients, and each passable grid corresponds to a state in Q-learning.

7. The mobile robot path planning method based on artificial lemming prior-guided Q-learning according to claim 1, characterized in that, S5 include: Calculate the average path length of the path population and the average distance from the endpoint of each candidate path to the target state; The initial complexity is defined after normalizing the average path length and average distance respectively; The initial complexity is nonlinearly amplified to obtain the environmental complexity; based on the environmental complexity... The ε-greedy exploration rate is updated according to the following formula: ； In the formula, Indicates the first The exploration rate at each training moment or training round. and These represent the upper limit and lower limit of the exploration rate, respectively. This represents the exploration rate adjustment coefficient.

8. The mobile robot path planning method based on artificial lemming prior-guided Q-learning according to claim 1, characterized in that, S6 include: In each training round, the mobile robot starts from the initial state and selects an action according to the adaptive ε-greedy policy in the current state; after executing the action, the environment returns the subsequent state and an immediate reward. ; The heuristic guiding term is constructed based on the consistency score of successor states and directions as follows: ； In the formula, Indicates the state Execute action Heuristic guidance at the time Represents the successor state Prior value in a prior value map Indicates action From the current state Pointing to target state Directional consistency score and These represent the weight coefficients of the prior value guiding term and the directional consistency guiding term, respectively; By incorporating heuristic guidance into the action value function update process of Q-learning, the following update formula is obtained: ； In the formula, Indicates the learning rate. Indicates the discount factor. Represents the successor state Optional actions below.

9. The mobile robot path planning method based on artificial lemming prior-guided Q-learning according to claim 1, characterized in that, S7 includes: After training, the mobile robot is reset to the starting position. In each state, the effective action with the largest Q value is selected as the next action. If multiple effective actions have the same Q value, the action with higher prior value and better directional consistency is selected, and finally a collision-free path from the starting position to the target position is generated. The preset convergence condition adopts a sliding window determination method: within a sliding window consisting of W consecutive training rounds, the proportion of successful arrivals at the target position, the average length of successful paths, and the coefficient of variation of successful path length are statistically analyzed; when the proportion of successful arrivals at the target position within the sliding window is not lower than the preset success rate threshold, the coefficient of variation of successful path length is not higher than the preset stability threshold, and the average length of successful paths does not exceed a preset multiple of the current optimal path length, the current strategy is determined to have reached a stable convergence state.

10. A mobile robot path planning system based on artificial lemming prior-guided Q-learning, characterized in that, include: The gridding module is used to build a static grid map model, set the starting position and target position of the mobile robot, and define the obstacle area, passable area and action space. The artificial lemming prior search module is used to perform prior search of the artificial lemming algorithm based on a static grid map model, generate a path population consisting of multiple candidate paths, and evaluate the fitness of the candidate paths. The prior value map construction module is used to construct a prior value map based on the path population to characterize the potential access value of each grid state in global path search. The Q-table initialization module is used to initialize the action value function of Q-learning based on the prior value map and action direction consistency, so as to obtain an initial Q-table with prior direction bias. The adaptive exploration module is used to calculate the environmental complexity based on the statistical characteristics of the path population, and adaptively adjust the ε-greedy exploration rate of Q-learning based on the environmental complexity. The prior guidance update module is used to select an action based on the current state and perform a state transition during Q-learning training to obtain an immediate reward. At the same time, it constructs a heuristic guidance term based on the prior value map and the consistency of action direction, and integrates the heuristic guidance term into the action value function update process. The path output module is used to output a collision-free path from the starting position to the target position based on the trained Q-table; The mobile robot path planning system based on artificial lemming prior-guided Q-learning is used to implement the mobile robot path planning method and steps based on artificial lemming prior-guided Q-learning as described in claim 1.

Images