A multi-robot full coverage path planning method based on heuristic Q-learning

Abstract

The present application relates to a kind of heuristic Q-learning-based multi-robot full coverage path planning method, including constructing global gridding map and dividing mutually non-overlapping connected sub-region by DARP algorithm;Each robot independent sub-environment is constructed and Q-learning agent is initialized;Agent uses the enhanced state definition of "absolute coordinate+local environment topological feature", and the optimal action is selected by heuristic strategy of unvisited priority and backtrack distance guidance;The Q value table is updated based on the comprehensive function of coverage reward, dynamic repeated access penalty and completion reward;Path is generated after iteration training to full coverage or reaching preset number.The present application reduces 23.8% of total moving steps under the premise of ensuring 100% coverage, reduces path repetition rate from 26.35% to 3.36%, reduces 44.1% of turning number, significantly improves work efficiency, and can be widely applied to collaborative work scenarios such as cleaning robot cluster and agricultural automatic harvester group.

Description

Technical Field

[0001] This invention relates to a multi-robot full-coverage path planning method based on heuristic Q-Learning, which is applicable to multi-robot collaborative operation scenarios such as cleaning robot clusters, agricultural automated harvester groups, and search and rescue vehicle fleets, and belongs to the field of mobile robot path planning technology. Background Technology

[0002] In fields such as intelligent manufacturing, agricultural automation, public environmental sanitation, and emergency rescue, multi-robot collaborative operations have gradually replaced traditional single-robot operation modes due to their high efficiency, fault tolerance, and flexibility. Multi-robot full-coverage path planning, as a core prerequisite for collaborative operations, aims to plan seamless and low-repetition collaborative paths in complex environments. As operational scenarios become increasingly large-scale and unstructured, higher demands are placed on path planning algorithms regarding coverage completeness, convergence speed, and path repetition rate.

[0003] Existing multi-robot full-coverage path planning methods are mainly divided into two categories:

[0004] Traditional path planning combined with divide-and-conquer methods, represented by the ox-plowing method, the inner spiral method, and the spanning tree covering method (STC), combine divide-and-conquer algorithms such as DARP to divide the global region into sub-regions. However, these methods have significant drawbacks: while the DARP algorithm can achieve balanced region distribution, the STC algorithm relies on 2×2 grid aggregation, which lacks flexibility when facing jagged boundaries generated by DARP or narrow channels with single-grid widths. It is prone to leaving coverage dead zones, causing the robot to backtrack a lot, significantly increasing the path repetition rate, and restricting the operation efficiency.

[0005] Planning methods based on reinforcement learning, represented by the Q-Learning algorithm, learn the optimal strategy through trial and error by interacting with the environment, exhibiting strong environmental adaptability. However, conventional Q-Learning algorithms have key drawbacks: First, they involve initial blind exploration; when the Q-value table is not effectively updated, the agent relies on random strategies, resulting in numerous iterations and slow convergence, making it difficult to meet real-time requirements in large-scale scenarios. Second, the state definition is simplistic, relying solely on absolute coordinates, failing to distinguish between different local topological environments at the same location, leading to state confusion. Third, the path repetition rate is high; the lack of targeted guidance easily creates "islands" of uncovered areas, requiring extensive repeated exploration to reach them. Fourth, dead zone backtracking is difficult; once the agent is trapped in a covered area, it wanders blindly and cannot quickly turn to an uncovered area.

[0006] In summary, existing multi-robot full-coverage path planning methods suffer from several drawbacks. Traditional methods exhibit poor environmental adaptability and high path repetition rates, while conventional reinforcement learning methods such as Q-Learning suffer from slow convergence due to initial blind exploration, state confusion, difficulty in dead zone backtracking, and insufficient integration between multi-robot collaboration and single-robot path generation. These shortcomings make it difficult to meet the needs of efficient and complete full-coverage operations in large-scale and complex environments. Therefore, there is an urgent need for a path planning method that balances environmental adaptability, convergence efficiency, and coverage quality. Summary of the Invention

[0007] To overcome the shortcomings of existing traditional path planning methods, such as poor environmental adaptability and high path repetition rate, as well as the slow convergence, state confusion, and difficulty in dead zone backtracking of conventional Q-Learning algorithms, this invention provides a multi-robot full-coverage path planning method based on heuristic Q-Learning. While ensuring 100% coverage, it reduces the total number of system steps by 23.8%, the path repetition rate from 26.35% to 3.36%, and the number of turns by 44.1%, significantly improving work efficiency. It can be widely applied to collaborative operation scenarios such as cleaning robot clusters and agricultural automated harvester groups.

[0008] A multi-robot full-coverage path planning method based on heuristic Q-Learning includes the following steps:

[0009] Step 1: Construct a global rasterized map, identify obstacles and the robot's initial position, and use the DARP algorithm to divide the global map into multiple non-overlapping and connected sub-regions, generating an allocation matrix for each sub-region;

[0010] Step 2: Construct an independent sub-environment for each robot based on the sub-region allocation matrix, treat the regions allocated to other robots as virtual obstacles, and initialize the Q-Learning agent, Q-value table, and iteration count for each robot;

[0011] Step 3: Obtain the current state of the Q-Learning agent. The state is composed of the agent's absolute coordinates and the topological features of the local environment. Based on the current state, a heuristic strategy including prioritizing unvisited areas and guiding the nearest distance is adopted to select the optimal action.

[0012] Step 4: The agent executes the action selected in Step 3, obtains the comprehensive feedback value of the coverage reward, repeated access penalty and completion reward based on the state after the move, calculates the final reward value and updates the Q value table;

[0013] Step 5: Repeat steps 3 and 4, iteratively train and update the Q-value table until each sub-region is fully covered or the number of iterations reaches the preset value, and generate the full coverage path of each sub-region based on the final Q-value table of each sub-region.

[0014] The construction of the independent sub-environment for each robot in step two is as follows: For the i-th robot, based on the global region allocation matrix obtained by the DARP algorithm in step one, the grid positions with values ​​not equal to i in the allocation matrix and the physical obstacle positions in the global map are uniformly marked as impassable obstacles in the sub-environment of the i-th robot.

[0015] Step three, obtaining the current state of the Q-Learning agent, specifically includes:

[0016] The state is defined as a tuple (P, S), where P is the absolute coordinates of the agent in the grid map, and S is the local environment feature value. The calculation process of the local environment feature value S is as follows: taking the agent's current position as the center, traverse the four adjacent grids (upper, lower, left, and right) in a preset order; if the adjacent grid is in the map and is in an unvisited state, mark it as "new path" and assign it a binary 1; if it is an obstacle or is in a visited state, mark it as "old path" and assign it a binary 0; combine the four binary values ​​into a decimal integer through bitwise operations, which is used as the local environment feature value S.

[0017] Specifically, bitwise operations are used to combine the binary values ​​of four adjacent grid cells into a unique decimal integer as a feature value. This feature value enables the agent to distinguish between entities with the same geometric location but different local topological environments, thereby eliminating state confusion and accelerating algorithm convergence.

[0018] The heuristic strategy in step three specifically includes an unvisited neighbor priority mechanism and a backtracking distance guidance mechanism:

[0019] Unvisited Neighbor Priority Mechanism: Detect the neighbor grids around the current agent's position. If there are unvisited neighbor grids, calculate the "unvisited degree" of each unvisited neighbor grid—that is, the number of unvisited grids around the neighbor grid. Prioritize the neighbor grid with the lowest unvisited degree as the next action to prevent the grid with the lowest unvisited degree from becoming an isolated area, resulting in an excessively high repetition rate.

[0020] Backtracking distance guidance mechanism: If all neighboring grids around the current agent have been visited, it means that the agent has entered a dead end or is surrounded by explored areas. Then, an artificial potential field is constructed to guide backtracking. The attraction term of the artificial potential field is the minimum Manhattan distance from the current position to the nearest uncovered grid, and the repulsion term is the historical visit count of the neighboring grid. The neighboring grid with the smallest distance and the fewest visit counts is selected as the next action.

[0021] The artificial potential field consists of two parts: the minimum Manhattan distance from the current agent's position to the nearest uncovered grid in the map is used as the attraction term; the historical visit count of the current agent's neighboring grids is used as the repulsion term. The neighboring grid with the smallest distance and the fewest visits is selected as the next action to guide the agent to quickly jump out of the locally covered area.

[0022] The final reward value in step four includes coverage reward, negative penalty, and completion reward.

[0023] The final reward value is calculated as follows:

[0024] If an agent enters an unvisited grid, it is given a fixed positive coverage reward;

[0025] If a previously visited grid is entered, a negative penalty is imposed. The penalty value is composed of a base penalty value and the number of times the grid has been visited in the past.

[0026] If all free grid cells in the current sub-region are covered, a one-time high completion reward will be given.

[0027] Step five specifically includes:

[0028] Repeat steps three and four, obtaining feedback values ​​and updating the Q-value table, until full coverage is achieved or the number of steps in the current round exceeds the preset dynamic maximum step limit. After each training round, [the following steps are performed]. The exploration rate in the strategy is decayed;

[0029] After all training rounds are completed, reset the sub-environment state of each robot, move the agent back to its initial position, clear the temporary access records, and switch the agent's action selection strategy to exploitation mode.

[0030] In exploitation mode, the agent is completely shut down. The strategy incorporates a random exploration function, where the agent selects the action with the highest value in the current state based on the Q-value table at each decision step. If multiple actions have the same value, the agent reverts to the unvisited degree-first heuristic strategy described in step three for decision-making. Each agent iteratively makes its next action decision based on the Q-value table and the unvisited degree-first heuristic strategy, ultimately generating a full-coverage path for each sub-region.

[0031] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0032] Divide and conquer strategy reduces planning complexity: The DARP algorithm divides large-scale complex regions into non-overlapping connected sub-regions, achieving load balancing for multiple robots. This transforms the multi-robot collaboration problem into a single-robot local coverage problem, significantly reducing algorithm complexity and ensuring regional connectivity and load balancing.

[0033] Enhanced state definition eliminates confusion: By combining absolute coordinates with local environmental topological features to construct a state space, the agent can distinguish different topological scenes (such as open space, narrow corridor, dead end) at the same location, completely solving the state confusion problem of traditional Q-Learning and improving the algorithm's convergence speed.

[0034] Heuristic strategies avoid backtracking to isolated areas and dead zones: the unvisited degree priority mechanism prioritizes processing unvisited edge grids, reducing the probability of "isolated" areas to 0; the backtracking distance guidance mechanism provides global navigation through an artificial potential field, reducing the agent's dead zone backtracking time by more than 60% and enabling rapid turning to uncovered areas;

[0035] Dynamic penalty mechanism reduces duplication rate: By adopting a dynamic penalty strategy that is linearly linked to the number of visits, the path duplication rate is reduced from 26.35% in traditional Q-Learning to 3.36%, which greatly improves the effectiveness of the path.

[0036] Significantly improved operational efficiency: While ensuring 100% coverage, the total number of steps moved by the system was reduced by 167 (an increase of 23.8%), and the number of turns was reduced by 158 (a decrease of 44.1%), significantly reducing energy consumption and operation time, and meeting the real-time operation needs in large-scale and complex environments. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is an overall flowchart of a multi-robot full-coverage path planning method based on heuristic Q-Learning according to the present invention.

[0039] Figure 2 This is a flowchart of the DARP partitioning algorithm for a multi-robot full-coverage path planning method based on heuristic Q-Learning, as described in this invention.

[0040] Figure 3 This is a schematic diagram of a global gridded map constructed by a multi-robot full-coverage path planning method based on heuristic Q-Learning according to the present invention.

[0041] Figure 4 This is a schematic diagram of the region allocation result after partitioning the DARP algorithm, a multi-robot full-coverage path planning method based on heuristic Q-Learning, according to the present invention.

[0042] Figure 5 This invention provides a multi-robot full-coverage path planning method based on heuristic Q-Learning, where each robot generates a final full-coverage path trajectory map based on traditional Q-Learning.

[0043] Figure 6 This invention relates to a multi-robot full-coverage path planning method based on heuristic Q-Learning, where each robot generates a final full-coverage path trajectory map based on heuristic Q-Learning. Detailed Implementation

[0044] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] To address the issues of slow convergence and high path duplication rates caused by blind exploration in the early stages of reinforcement learning in existing full-coverage path planning methods, this paper refers to... Figure 1-6 A multi-robot full-coverage path planning method based on heuristic Q-Learning includes the following steps:

[0046] Step 1: Construct a global rasterized map, identify obstacles and the robot's initial position, and use the DARP algorithm to divide the global map into multiple non-overlapping and connected sub-regions, generating an allocation matrix for each sub-region;

[0047] Specifically, first, the two-dimensional grid matrix of the area to be covered is read, where obstacle grids are assigned a value of -2, marking them as impassable and unallocable areas; free grids are assigned a value of -1, marking them as areas to be allocated; and then the robot set is initialized. and its initial position coordinates Initialize the correction factors for each robot. Set to 1. Construct an allocation matrix of the same size as the raster matrix. In the initial state, only the initial position of each robot is set. The grid at that location is assigned the corresponding robot index. The remaining grid cells are assigned a value of 0.

[0048] Calculate each free grid cell in the map To the initial position of each robot The Euclidean distance, with a generation dimension of Initial evaluation matrix , where matrix elements Indicates the first A robot to the grid The metric value is calculated using the following formula:

[0049]

[0050] In the formula, For the first The grid coordinates of the robot's initial position.

[0051] Obtain the evaluation matrix for each robot. Then, the core iterative loop of the DARP algorithm is entered until the convergence condition is met. Each iteration includes the following processing:

[0052] First, construct the initial evaluation matrix. ; Traverse all grid cells in the grid matrix, if the grid cell For the free region, from the evaluation matrix Find the robot index i corresponding to the smallest value at that position in the grid, assign the i-th robot to that grid, and update the assignment matrix A:

[0053]

[0054] If grid If it is an obstacle, then in the allocation matrix The middle mark is After calculating the allocation matrix A, the sub-region to be covered assigned to the i-th robot is obtained. Based on the initial allocation strategy described above, there may be instances where some free grid cells are at the same distance to multiple robots. To address this issue, the algorithm introduces a correction factor into the evaluation matrix E. The evaluation matrix of each robot during the iteration process Update according to the following formula:

[0055]

[0056] Then, OpenCV's connected component analysis algorithm is used to check the connectivity of the current assigned region of each robot; if the connectivity of the first robot is detected... The region of a robot contains multiple disconnected components. The connected component where its initial position is located is defined as the main region. The remaining separate connected components are isolated regions. Construct a connectivity penalty matrix. Calculate the distance difference between each point on the map and the main area and isolated areas:

[0057]

[0058] The matrix is ​​normalized and a connectivity adjustment coefficient is introduced. :

[0059]

[0060] Count the total number of grid cells actually assigned to each robot. Calculate its ratio with the expected allocation number. To account for the deviation, the expected allocation number in this experiment is the total number of free grids divided by the total number of robots. The correction factor for each robot is updated using the gradient descent method. :

[0061]

[0062] In the formula, To adjust the step size.

[0063] Then synthesize the connectivity matrix Updated correction factors and random perturbation matrix Update the evaluation matrix as follows :

[0064]

[0065] In the formula, This represents element-wise multiplication.

[0066] Finally, repeat the above iterative steps to continuously optimize the evaluation matrix. The process continues until one of the following termination conditions is met: all robot assignment regions are simply connected regions, and the number of robots assigned to each region deviates from the specified value. The value is less than a preset threshold; or the number of iterations reaches a preset upper limit. After the iteration terminates, the final output is the converged allocation matrix. This serves as the sub-region input for subsequent path planning.

[0067] Step 2: Construct an independent sub-environment for each robot based on the sub-region allocation matrix A, treat the regions allocated to other robots as virtual obstacles, and initialize the Q-Learning agent, Q-value table, and iteration count for each robot;

[0068] Specifically, based on the sub-region allocation matrix Construct independent sub-environments for each robot, treating the areas allocated to other robots as virtual obstacles: For robot i, traverse the allocation matrix. Each grid point in If the value of a grid point is equal to i, then the grid is marked as the robot's free working area; if the value is not equal to i or the location itself is an obstacle in the global map, then the grid coordinates are... Add the obstacle set to the i-th robot, thus treating the work areas of other robots as virtual obstacles, thereby logically isolating the global map into multiple independent sub-environments.

[0069] Initialize the robot's Q-Learning agent and Q-value table, instantiate a heuristically based Q-learning agent object for each robot, and set the learning rate. Discount factor of 0.1 The exploration rate is 0.99 and the greedy strategy. The exploration rate is set to 0.5, and it decreases as training progresses. The agent's action space is defined as four actions: up, down, left, and right. The current training round counter is initialized to 0, and the maximum number of training rounds is set to 600.

[0070] Step 3: Obtain the current state of the Q-Learning agent. The state is composed of the agent's absolute coordinates and the topological features of the local environment. Based on the current state, a heuristic strategy including prioritizing unvisited areas and guiding the nearest distance is adopted to select the optimal action.

[0071] Specifically, the agent first obtains the current state information S through the environment interface. This state is a binary tuple. ,in These are the absolute coordinates of the agent, representing the agent's current position in the grid map; This is a local environment topology feature used to enhance the agent's perception of local micro-environments (such as corridors, intersections, and dead ends), enabling the agent to distinguish different exploration stages under the same coordinates. The agent traverses the four neighboring grids of the current position, checking if they are in a "free grid and unvisited" state. If a neighbor in a certain direction is an unvisited free grid, the corresponding position is set to 1; otherwise, it is set to 0. Then, the exploration states of the four neighboring grids above, below, left, and right of the current position are encoded. Bitwise operations are used to convert the states of the four directions into an integer from 0 to 15, which serves as the local environment topology feature value of the current position.

[0072] When the random number is greater than The agent invokes the heuristic decision-making module. The agent first obtains all free neighbor grids at its current location and then filters out a candidate set that has not yet been visited. .like If the set is not empty, the system calculates the unvisited degree of each neighboring grid in the set. The unvisited degree is defined as the number of unvisited grids in the free space surrounding that neighboring grid. Then, the agent prioritizes selecting the neighboring grid with the lowest unvisited degree as the next action. The lower the unvisited degree, the closer the grid is to a wall, obstacle, or the edge of a covered area. If multiple neighboring grids have the same lowest unvisited degree, the agent further queries the Q-value table and selects the action with the highest Q-value.

[0073] When an agent is surrounded by obstacles or already visited grid cells, resulting in a dead end or encirclement, the system triggers a backtracking mechanism to select the next action from the set of already visited neighbor grid cells. First, the agent calculates the Manhattan distance from each visited neighbor grid cell in its current location to the nearest uncovered grid cell on the map. At the same time, obtain the historical visit count of each neighboring grid. Through a comprehensive scoring formula:

[0074]

[0075] In the formula, The penalty coefficient is 0.5, which was chosen for this experiment.

[0076] Comprehensive score for agent selection The lowest neighbor grid is used as the action output. The distance potential field is used to provide global navigation for the agent. Combined with the visit count penalty, the agent is guided to jump out of the currently covered area in the shortest path and in a way that avoids repetition as much as possible, and head to the nearest unexplored area.

[0077] Step 4: The agent executes the action selected in Step 3, obtains the comprehensive feedback value of the coverage reward, repeated access penalty and completion reward based on the state after the move, calculates the final reward value and updates the Q value table;

[0078] Based on the current coordinates and the optimal action decided in step three, the agent calculates the expected target coordinates. First, it determines whether the target coordinates exceed the sub-region boundary or belong to the area of ​​obstacles or other robots. If so, it indicates that the agent has collided, and the agent remains stationary, receiving a high negative penalty of -10. If not, it indicates that the expected target coordinates are reasonable, and the agent moves to the new coordinates, updates its current position state to the new coordinates, and increments the step counter by 1.

[0079] After updating the current location to the new coordinates, the comprehensive reward value for this optimal action is calculated. This reward consists of three parts: a coverage reward value, a dynamic penalty value for repeated visits, and a completion reward plus step consumption. The coverage reward value is mainly determined based on the access status of the new location. If the access value of the new location is 0, the state is unvisited, and a fixed positive reward of 10 is given for covering the new grid. At the same time, the grid is marked as visited, and the full coverage counter is incremented by one. Conversely, if the access value of the new location is 1, the state is visited, and it is determined to be a repeated path. In this case, a dynamic penalty value for repeated visits is given based on the number of times the grid is visited. The dynamic penalty value increases linearly with the number of times the grid is repeatedly traversed. The formula for calculating the dynamic penalty value is as follows:

[0080]

[0081] In the formula, It is a dynamic penalty value. This represents the number of times the grid has been accessed.

[0082] In addition to the two penalty and reward values ​​mentioned above, the step cost value is a small basic step cost deducted for each move, with a value of -0.1, which is used to encourage the agent to seek the shortest path; if all free grids in the sub-region are covered, a one-time high completion reward of 100 is given, which is used to encourage the agent to achieve full coverage, and at the same time, the training round is marked as over.

[0083] After the agent obtains the comprehensive reward value of this optimal action, it observes the state at the next moment. Update the corresponding entries in the Q-value table using the Bellman equation. That is, the Q value. The Bellman equation is as follows:

[0084]

[0085] In the formula, For learning rate, As a discount factor, This represents an estimate of the maximum potential value of the next state.

[0086] Step 5: Repeat steps 3 and 4, iteratively train and update the Q-value table until each sub-region is fully covered or the number of iterations reaches the preset value, and generate the full coverage path of each sub-region based on the final Q-value table of each sub-region.

[0087] When the training round ends and the flag is true, or when the maximum training steps are reached, the agent returns to the starting point, the grid is reset to unvisited, the current sub-environment state is reset, and a new training round begins. The agent repeats the heuristic policy selection action in step three and the execution action in step four, obtains feedback values, and updates the Q-value table until the full coverage of the region is completed or the number of steps in the current round exceeds the preset dynamic maximum step limit. After each training round, the agent performs a series of actions. Exploration rate in strategy The attenuation process is performed using the following formula:

[0088]

[0089] After all training rounds are completed, reset the sub-environment state of each robot, move the agent back to its initial position, and clear temporary access records. Simultaneously, switch the agent's action selection strategy to exploitation mode.

[0090] In this mode, the agent is completely turned off. The strategy incorporates a random exploration function, where the agent selects the action with the highest value in the current state based on the Q-value table at each step. If multiple actions with the same value exist in the current state, the decision reverts to the unvisited degree-first heuristic strategy described in step three. Each agent iteratively makes its next action decision based on the Q-values ​​in the Q-value table, combined with the unvisited degree-first heuristic strategy, ultimately generating a full-coverage path for each sub-region.

[0091] The effects of this invention can be further illustrated by the following simulation results:

[0092] Tables 1 and 2 show the data results for each robot in its sub-region using the traditional Q-Learning algorithm and the heuristic Q-Learning algorithm, respectively.

[0093] Table 1

[0094]

[0095] Table 2

[0096]

[0097] To verify the superiority of the proposed heuristic Q-Learning-based multi-robot full-coverage path planning method, simulation data of the traditional Q-Learning algorithm (Table 1) and the algorithm of this invention (Table 2) were compared under the condition that the number of robots (4), initial positions, and task areas were exactly the same. The comparison results are as follows: Figure 4 , Figure 5 As shown in the table above.

[0098] As can be seen from the comparison of the data in Tables 1 and 2, although both the traditional Q-Learning algorithm and the algorithm of this invention have completed the full coverage task, the algorithm of this invention reduces the total number of system movement steps by 167 steps, improves the operation efficiency by about 23.8%, the overall path repetition rate of the system drops sharply to 3.36%, a reduction of about 23%, and the total number of system turns is reduced to 200, a reduction of 44.1%.

[0099] In summary, compared with the traditional Q-Learning algorithm, the path planning method based on heuristic Q-Learning proposed in this invention completes the task with fewer steps, extremely low repetition rate and fewer turns while ensuring 100% full coverage. It effectively solves the problems of blind exploration and messy paths in traditional methods.

[0100] This invention utilizes a heuristic approach based on topological unvisitedness to effectively avoid the "island" effect: It introduces an "unvisitedness" priority mechanism into the action selection strategy. When making decisions, the agent calculates the degrees of freedom of all candidate neighboring grids and prioritizes the grid with the lowest degrees of freedom (i.e., the fewest surrounding unvisited grids). This significantly reduces the probability of leaving hard-to-reach, scattered uncovered points (i.e., "islands") in the center of the map, thereby improving the coherence of path coverage and the one-time coverage rate.

[0101] This invention integrates global backtracking guidance using distance potential energy fields to solve the problem of blind wandering after getting stuck in a dead zone: Addressing the common local optimum trap in full-coverage tasks, this invention proposes a dynamic backtracking mechanism based on distance potential energy fields. When the agent gets stuck in a covered area, the system no longer relies solely on a slowly converging Q-value table, but instead calculates the Manhattan distance between the current position and the nearest uncovered grid on the map in real time, and constructs a comprehensive score by combining this distance with the historical access frequency along that path. This mechanism essentially provides the agent with global navigation pointing to a new area, guiding it to quickly escape dead ends with the shortest path and lowest repetition rate, significantly reducing the time wasted wandering within covered areas.

[0102] This invention optimizes the state space definition for reinforcement learning by combining the agent's absolute coordinates with the local micro-environment states of its neighbors. This enhanced state representation enables the agent to distinguish different environmental contexts at the same coordinates (e.g., between being in an open space and being in a narrow corridor), thereby learning more refined and robust movement strategies.

[0103] This invention introduces a dynamic penalty based on the number of visits to reduce the repetition rate: a dynamic penalty mechanism linked to the number of visits is introduced, which enables the agent to dislike high-frequency access areas, thereby planning a path with a lower repetition rate.

[0104] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A multi-robot full-coverage path planning method based on heuristic Q-Learning, characterized in that: Includes the following steps: Step 1: Construct a global rasterized map, identify obstacles and the robot's initial position, and use the DARP algorithm to divide the global map into multiple non-overlapping and connected sub-regions, generating an allocation matrix for each sub-region; Step 2: Construct an independent sub-environment for each robot based on the sub-region allocation matrix, treat the regions allocated to other robots as virtual obstacles, and initialize the Q-Learning agent, Q-value table, and iteration count for each robot; Step 3: Obtain the current state of the Q-Learning agent. The state is composed of the agent's absolute coordinates and the topological features of the local environment. Based on the current state, a heuristic strategy including prioritizing unvisited areas and guiding the nearest distance is adopted to select the optimal action. The heuristic strategy in step three specifically includes an unvisited neighbor priority mechanism and a backtracking distance guidance mechanism: Unvisited Neighbor Priority Mechanism: Detect the neighbor grids around the current agent's position. If there are unvisited neighbor grids, calculate the "unvisited degree" of each unvisited neighbor grid—that is, the number of unvisited grids around the neighbor grid. Prioritize the neighbor grid with the lowest unvisited degree as the next action. Backtracking distance guidance mechanism: If all neighboring grids around the current agent have been visited, an artificial potential field is constructed to guide backtracking. The attraction term of the artificial potential field is the minimum Manhattan distance from the current position to the nearest uncovered grid, and the repulsion term is the historical visit count of the neighboring grid. The neighboring grid with the smallest distance and the fewest visit counts is selected as the next action. Step 4: The agent executes the action selected in Step 3, obtains the comprehensive feedback value of the coverage reward, repeated access penalty and completion reward based on the state after the move, calculates the final reward value and updates the Q value table; Step 5: Repeat steps 3 and 4, iteratively train and update the Q-value table until each sub-region is fully covered or the number of iterations reaches the preset value, and generate the full coverage path of each sub-region based on the final Q-value table of each sub-region.

2. The multi-robot full-coverage path planning method based on heuristic Q-Learning according to claim 1, characterized in that: The construction of the independent sub-environment for each robot in step two specifically includes: for the i-th robot, based on the global region allocation matrix obtained by the DARP algorithm in step one, the grid positions with values ​​not equal to i in the allocation matrix and the physical obstacle positions in the global map are uniformly marked as impassable obstacles in the sub-environment of the i-th robot.

3. The multi-robot full-coverage path planning method based on heuristic Q-Learning according to claim 1, characterized in that: Step three, obtaining the current state of the Q-Learning agent, specifically includes: The state is defined as a tuple (P, S), where P is the absolute coordinates of the agent in the grid map, and S is the local environment feature value. The calculation process of the local environment feature value S is as follows: taking the agent's current position as the center, traverse the four adjacent grids (upper, lower, left, and right) in a preset order; if the adjacent grid is in the map and is in an unvisited state, mark it as "new path" and assign it a binary 1; if it is an obstacle or is in a visited state, mark it as "old path" and assign it a binary 0; combine the four binary values ​​into a decimal integer through bitwise operations, which is used as the local environment feature value S.

4. The multi-robot full-coverage path planning method based on heuristic Q-Learning according to claim 1, characterized in that: The final reward value in step four includes coverage reward, negative penalty, and completion reward.

5. The multi-robot full-coverage path planning method based on heuristic Q-Learning according to claim 4, characterized in that: The calculation rule for the final reward value is as follows: If an agent enters an unvisited grid, it is given a fixed positive coverage reward; If a previously visited grid is entered, a negative penalty is imposed. The penalty value is composed of a base penalty value and the number of times the grid has been visited in the past. If all free grid cells in the current sub-region are covered, a one-time high completion reward will be given.

6. The multi-robot full-coverage path planning method based on heuristic Q-Learning according to claim 1, characterized in that: Step five specifically includes: Repeat steps three and four, obtaining feedback values ​​and updating the Q-value table, until full coverage is achieved or the number of steps in the current round exceeds the preset maximum step limit. After each training round, [the following steps are performed]. The exploration rate in the strategy is decayed; After all training rounds are completed, reset the sub-environment state of each robot, move the agent back to its initial position, clear the temporary access records, and switch the agent's action selection strategy to exploitation mode. In exploitation mode, the agent is completely shut down. The random exploration function in the strategy allows the agent to select the action with the highest value in the current state based on the Q-value table at each step. If multiple actions have the same value, the agent will fall back to the unvisited degree priority heuristic strategy described in step three to make a decision. Each agent iteratively makes the next action decision based on the Q-value table and the unvisited degree priority heuristic strategy, and finally generates a full coverage path for each sub-region.

Images