Path planning method and system fusing hyper-heuristic scheduling and QL

Abstract

The invention discloses a hyperheuristic scheduling and QL integrated path planning method and system, and the method comprises the steps: firstly constructing a rasterized map model, setting a starting position and a target position, and defining the map information of an obstacle and a passing region; secondly, an action value function is initialized according to the constructed rasterized map model, heuristic information is introduced into a QL algorithm, evaluation function values of a heuristic operator LLH from the initial position to the target position are calculated, the maximum value of the evaluation function values serves as a globally optimal reference path, and the QL algorithm is optimized and adjusted. And finally, in the established rasterized environment model, continuously iteratively learning and updating an action value function through the optimized QL to obtain an optimal path. According to the method, efficient and adaptive decision making in a complex environment is realized, and the global search performance and the path planning capability of the algorithm are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of path planning technology, and in particular relates to a path planning method and system that integrates hyperheuristic scheduling and QL. Background Technology

[0002] With the rapid development of artificial intelligence technology, path planning, as a core task in robot autonomous decision-making and motion control, is increasingly becoming a research hotspot. The main goal of path planning is to plan an optimal path from the starting point to the target point for a robot in a known or unknown environment. As a key link in achieving efficient perception and intelligent decision-making in robots, path planning is widely used in various intelligent scenarios such as autonomous driving systems, warehouse logistics robots, and cleaning robots. It plays a crucial role in improving the operational efficiency and safety performance of intelligent devices, possessing extremely high research value and application prospects.

[0003] Before autonomous learning was widely applied to path planning problems, researchers mainly relied on traditional algorithms, heuristic search algorithms, and biomimetic intelligent optimization algorithms to solve path planning tasks. However, traditional and heuristic algorithms mostly depend on predefined models, rules, or heuristic functions, and often struggle to adjust strategies in real time when faced with complex, unstructured, or variable environments. While biomimetic algorithms possess certain global search capabilities, they suffer from strong parameter dependencies, slow convergence speeds, and a tendency to get trapped in local optima.

[0004] Therefore, with researchers' in-depth study of path planning problems, reinforcement learning, as an intelligent algorithm with autonomous learning capabilities, has gradually demonstrated strong potential and broad application prospects in the field of path planning. Reinforcement learning is an intelligent algorithm that uses robots to learn through trial and error in interaction with their environment, enabling the robot to learn an optimal strategy through continuous exploration to maximize its cumulative reward in a specific task. Among them, the QL algorithm is a classic reinforcement learning algorithm, first proposed by Watkins.

[0005] However, in existing technologies, the QL algorithm lacks an effective initialization mechanism and relies entirely on interactive learning from scratch. At the same time, in complex trap environments, it is prone to getting stuck in local optima, resulting in low path planning efficiency and affecting overall performance.

[0006] To address the aforementioned problems, this invention can improve the learning rate, convergence speed, and optimization ability of the algorithm in complex trap environments, as well as enhance path smoothness and shorten path length. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention proposes a path planning method and system that integrates hyperheuristic scheduling and QL (Quick Path Query) to plan the optimal path more scientifically.

[0008] This invention aims to solve the static global path planning problem in complex environments. By introducing a reinforcement learning mechanism and dynamic scheduling of multiple heuristic operators, it achieves global path search and escape, thereby improving the convergence speed and path quality of the algorithm.

[0009] Specifically, the present invention is achieved through the following technical solution:

[0010] The first aspect of this invention provides a path planning method that integrates hyperheuristic scheduling and QL, the method comprising:

[0011] Step 1: Construct a rasterized map model, set the starting and target positions, and define map information for obstacles and passable areas;

[0012] Step 2: Initialize the action value function based on the constructed raster map model, introduce heuristic information into the QL algorithm, calculate the evaluation function value of the heuristic operator LLH from the starting position to the target position, take the maximum value of the evaluation function value as the globally optimal reference path, and optimize and adjust the QL algorithm.

[0013] The optimization and adjustment of the QL algorithm specifically includes the following methods:

[0014] 2.1 Based on the established action value function, the hyperparameters of the QL algorithm are dynamically adjusted using an optimized learning rate model;

[0015] 2.2 In the initialization phase, the starting point and target point are first determined, and the comprehensive performance of the heuristic operators is evaluated using an evaluation function. The optimal heuristic operator is selected based on the output value of the evaluation function to generate the corresponding optimal global reference path. Based on the optimized learning rate model, a hyperheuristic scheduling mechanism is introduced. When trapped and unable to escape, the evaluation functions of different heuristic operators are calculated between the current grid point and the nearest grid point on the global optimal reference path. The operator with the maximum value obtained by the evaluation function is used as the escape path and direction. The intersection point generated by the escape path and the global optimal reference path is used as the escape point.

[0016] 2.3 By moving towards the target point while avoiding obstacles, after successfully reaching the target point for the first time, and before the second iteration, the optimal pruning strategy is generated by taking the maximum evaluation function value of the heuristic operator between the escape points.

[0017] Step 3: Perform path planning in the established gridded environment model. Iteratively learn and update the action value function through the optimized QL to obtain the optimal path.

[0018] A second aspect of this application provides a path planning system integrating hyperheuristic scheduling and QL, the system comprising a gridding module, a path optimization module, and an optimal path module; wherein...

[0019] The rasterization module is used to build a rasterized map model, set the starting and target positions, and define map information for obstacles and passable areas.

[0020] The path optimization module is used to initialize the action value function based on the constructed raster map model, introduce heuristic information into the QL algorithm, calculate the evaluation function value of the heuristic operator LLH from the starting position to the target position, take the maximum value of the evaluation function value as the globally optimal reference path, and optimize and adjust the QL algorithm.

[0021] The optimal path module is used to continuously iterate and update the action value function through the optimized QL in the established gridded environment model to obtain the optimal path.

[0022] Beneficial effects of this invention:

[0023] This invention provides a path planning algorithm and system that integrates hyperheuristic scheduling and QL (Quick Path Level), aiming to improve the global optimization capability and learning efficiency of path planning, enhance the adaptability and stability of the algorithm in complex environments, and dynamically select the optimal heuristic operator through a hyperheuristic scheduling mechanism to achieve intelligent switching of path search strategies. In the global path planning phase, the algorithm uses a grid map as a basis, combined with obstacle distribution and environmental characteristics, to adaptively adjust the learning rate, thereby improving search efficiency and accelerating the learning speed.

[0024] Building upon global path planning, this invention further introduces an escape mechanism and path pruning strategy to address situations with multiple obstacles or trap areas in complex environments. When the algorithm gets stuck in a trap area during the search process, an escape path is generated through an escape point strategy, and the action value function is updated based on feedback from the optimal LLH operator to achieve rapid response and flexible adjustment. Furthermore, the pruning strategy is automatically triggered each iteration when an escape point is generated, ensuring that the final generated path has the shortest distance and optimal smoothness. Through the synergistic effect of global planning, dynamic scheduling, the escape mechanism, and reinforcement learning, this invention achieves efficient and adaptive decision-making in complex environments, significantly improving the algorithm's global search performance and path planning capabilities. Attached Figure Description

[0025] Figure 1 This is a flowchart of a path planning method and system that integrates hyperheuristic scheduling and QL, provided by an embodiment of the present invention;

[0026] Figure 2 This refers to the Q-table initialization information provided in this embodiment of the invention;

[0027] Figure 3 This is a schematic diagram of the system provided in an embodiment of the present invention;

[0028] Figure 4 This refers to the information of simulation environment 1 provided in this embodiment of the invention;

[0029] Figure 5 This refers to the information of simulation environment 2 provided in this embodiment of the invention;

[0030] Figure 6 This is a schematic diagram of the path simulation results of the QL algorithm provided in this embodiment of the invention in simulation environment 1;

[0031] Figure 7 This is a schematic diagram of the path simulation results of the optimized QL algorithm provided in this embodiment of the invention in simulation environment 1;

[0032] Figure 8 This is a schematic diagram of the path simulation results of the QL algorithm provided in this embodiment of the invention in simulation environment 2;

[0033] Figure 9 This is a schematic diagram of the path simulation results of the optimized QL algorithm provided in this embodiment of the invention in simulation environment 2;

[0034] Figure 10 This is a comparative diagram of learning rates in simulation environment 1 provided by an embodiment of the present invention;

[0035] Figure 11 This is a comparative diagram of the learning rate in simulation environment 2 provided by an embodiment of the present invention;

[0036] Figure 12 This is a comparison of the convergence performance in simulation environment 1 provided by the embodiments of the present invention;

[0037] Figure 13 This is a comparison of the convergence performance in simulation environment 2 provided by the embodiments of the present invention;

[0038] Figure 14 This is an embodiment of the invention showing the usage of various heuristic operators in simulation environment 1.

[0039] Figure 15 This describes the usage of various heuristic operators in simulation environment 2, as provided in this embodiment of the invention. Detailed Implementation

[0040] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0041] Figure 1 A flowchart illustrating a path planning method integrating hyperheuristic scheduling and QL provided in this application is shown. Please refer to... Figure 1 The method provided in this embodiment includes:

[0042] Construct a rasterized map model, set the starting and target positions, and define map information for obstacles and passable areas;

[0043] Specifically, the steps for constructing a rasterized map model include:

[0044] (1) Determine the working area and resolution;

[0045] The work area refers to the operating range within the environment, and the resolution is used to determine the accuracy of map division. The application site can be divided into several regional units according to its length and width, and the resolution is set by the task requirements to balance map accuracy and computational cost.

[0046] (2) Based on the resolution, the operating range is divided into a 25×25 grid area;

[0047] Each grid represents the smallest identifiable unit in the environment. The grids are all the same size and are used to record the status information of obstacles or passable areas.

[0048] (3) Obtain information about obstacles within the environmental range;

[0049] In reinforcement learning, the system dynamically learns the specific location information of obstacles through continuous interaction with the unknown environment.

[0050] (4) Assign a status value to each grid cell;

[0051] In the environmental model, unobstructed grids are marked as passable, while grids containing obstacles are marked as impassable, forming a complete environmental map.

[0052] The action value function is initialized based on the constructed raster map model. Heuristic information is introduced into the QL algorithm to provide an initial reference direction for the subsequent learning stage. At the same time, the evaluation function value of the heuristic operator LLH from the starting position to the target position is calculated. The maximum value of the evaluation function value is used as the globally optimal reference path, and the QL algorithm is optimized and adjusted accordingly.

[0053] Specifically, the process of initializing the action value function based on the constructed raster map model and calculating heuristic information in QL is as follows:

[0054] (1) Determine if the grid is an obstacle;

[0055] Specifically, during the initialization phase, when a grid is marked as an obstacle, it is not included in the heuristic calculation of the action value function. Conversely, after determining that a grid is a passable area, the coordinates of the current grid are set to (x, y). The movement direction attributes of up, down, left, right, upper left, lower left, upper right, and lower right are represented as (x, y+1), (x, y-1), (x-1, y), (x+1, y), (x-1, y+1), (x-1, y-1), (x+1, y+1), and (x+1, y-1), respectively.

[0056] (2) If the grid is a passable area, then the target point coordinates are used. Compared with the current state The Euclidean distances of the coordinates of the 8 neighboring grid points are calculated, and the reciprocal of the Euclidean distance exponential function is used as heuristic information.

[0057] Furthermore, heuristic information is calculated using the following formula:

[0058] (1)

[0059] in, For the resolution of the environment, For the set of directions of movement, This is the current state. It can move in one direction. To and Adjacent x-axis coordinate values, To and Adjacent y-axis coordinate values, The x-axis coordinate of the target point. The y-axis coordinate of the target point. This is the initial Q value. Wherein, Figure 2 A heatmap showing the distribution of Q-values ​​after applying the heuristic information provided in this embodiment is displayed. The grayscale from light to dark reflects the change in the magnitude of the Q-values, and the darker the color, the higher the value of the action in the corresponding direction.

[0060] Furthermore, in the QL algorithm, heuristic operators A*, JPS, and Theta* are selected as heuristic operators. For each heuristic operator, based on the path length, number of nodes, number of visited nodes, and running time characteristics obtained after execution, the reciprocal of the weighted composite value of each characteristic is taken as a performance index, representing the overall performance of different heuristic operators. The operator with the largest value among the performance indexes is determined as the current optimal operator, and the maximum value of the evaluation function corresponding to this optimal operator is used as the globally optimal reference path to guide path optimization and strategy updates in subsequent learning processes.

[0061] Specifically, the evaluation function is represented by the following formula:

[0062] (2)

[0063] in, , , and These represent the path length, number of nodes, number of accessed nodes, and running time for each LLH, respectively. , , and These are the corresponding weights ( ). The evaluation value is the heuristic operator with the highest evaluation value, which exhibits the best performance.

[0064] Furthermore, the learning rate model is optimized to dynamically adjust the hyperparameter learning rate of the QL algorithm;

[0065] Specifically, the steps for optimizing the learning rate model include:

[0066] (1) When the number of times the QL algorithm falls into a trap during the learning process exceeds the preset maximum tolerance threshold, the learning rate of the current stage remains unchanged;

[0067] Specifically, at this point, the QL algorithm is in a local optimum, and adjusting the learning rate immediately would lead to learning instability. Therefore, the learning rate is kept constant for a period of time, allowing continuous exploration with a fixed step size and gradual accumulation of empirical data.

[0068] (2) In the subsequent learning process of the QL algorithm, determine whether the number of times the trap is encountered still exceeds the preset threshold; if it does not exceed the threshold, gradually increase the learning rate until the iteration ends.

[0069] Specifically, if a device can continuously escape local trap regions and successfully reach the endpoint in several iterations, it indicates that its adaptability to the environment has improved.

[0070] Furthermore, by dynamically adjusting the learning rate, the Q-table is updated faster, which accelerates later convergence and enables stable learning while ensuring stability.

[0071] Specifically, the optimized learning rate model formula is as follows:

[0072] (3)

[0073] in, It is a constant parameter. It is the total number of iterations. It is the current iteration number. This is the current learning rate. It's the learning rate from the last iteration. It is the number of times you fall into a trap. Maximum tolerance threshold.

[0074] Furthermore, a hyperheuristic scheduling mechanism is introduced. When trapped and unable to escape, the evaluation function of different heuristic operators is calculated between the current grid point and the nearest grid point on the global optimal reference path. The heuristic operator with the maximum value obtained by the evaluation function is used as the escape path and direction. The intersection point generated by the escape path and the global optimal reference path is used as the escape point.

[0075] Specifically, the hyperheuristic scheduling mechanism serves as a control strategy, used to dynamically select multiple heuristic operators (LLHs) based on an evaluation function during path planning. Specifically, A*, JPS, and Theta* are introduced as part of the heuristic operators LLH. By combining the characteristics of different heuristic operators, the complementarity of multiple heuristic operators can be achieved during path planning, enabling the optimized QL algorithm to dynamically select the optimal heuristic operator under different environmental conditions.

[0076] Furthermore, the current Q value is updated based on the selected heuristic operator. This update process is based on the state at time step t. In the t-th iteration, the selected heuristic operator is used to adjust the current Q value, generating the updated Q value.

[0077] Specifically, the formula for updating the current Q value is:

[0078] (4)

[0079] in, It is the time step time surface, It is the first The heuristic operator chosen in the next iteration Is using For the present The table was updated. It is the updated version. surface.

[0080] Furthermore, during the Q-table update process, the optimal heuristic operator is selected based on the current state and environmental information to generate an escape path, and a weighting factor is introduced into the Q-table. The escape path is dynamically adjusted, and the weight factor is adaptively adjusted based on the performance of the heuristic operator and the current environment state to ensure the flexibility and effectiveness of the escape path.

[0081] Furthermore, the movement is performed according to the generated escape path direction. The specific steps in determining the escape path direction include:

[0082] (1) Based on the action value function corresponding to each direction in the current state, compare and select all possible movement directions;

[0083] Specifically, by combining the Q-values ​​of each action in the current state with the update results of the optimal heuristic operator in the hyperheuristic scheduling, the action values ​​in different directions are weighted and corrected. Then, the direction corresponding to the maximum value of the action value function is selected as the optimal movement direction for the next step.

[0084] (2) The next action direction is used to indicate the forward direction under the current position, and the optimal direction indicates the direction with the greatest action value under the current state;

[0085] Specifically, the set of all possible movement directions constitutes the action set, which is used by the QL algorithm to select the optimal action and update the strategy. In each movement, if the current direction is consistent with the optimal direction, then the movement continues in that direction to maintain the continuity of the path and the stability of the learning process; if the current direction is inconsistent with the optimal direction, then the movement is adjusted according to the greedy strategy to maintain the original direction or the optimal direction.

[0086] By moving towards the target point while avoiding obstacles, after successfully reaching the target point for the first time, and before the second iteration, the optimal pruning strategy is generated by maximizing the evaluation function value of the heuristic operator between escape points.

[0087] Specifically, during the initialization phase, the starting position and the target position are first determined in order to generate the corresponding optimal global reference path.

[0088] Furthermore, based on this, the escape point is determined according to the intersection of the escape path and the optimal global reference path. Then, the comprehensive performance of the heuristic operators A*, JPS, and Theta* needs to be evaluated and selected between each escape point using an evaluation function. Optimal pruning is then constructed segment by segment to ensure avoidance of obstacles and trap areas. After pruning, the retained path is taken as the current valid path, completing the online correction of the action value function.

[0089] This embodiment provides a path planning method that integrates hyperheuristic scheduling and QL (Quick Path Learning). By introducing a reinforcement learning mechanism and combining dynamic scheduling with A*, JPS, and Theta* heuristic operators, it achieves integrated control of global path planning and local escape optimization in complex environments. First, using the QL algorithm as the core, a hyperheuristic scheduling mechanism is employed to select among different heuristic operators, improving the search efficiency and flexibility of global path planning. During escape path planning, the escape path is constructed using the trap-triggered grid point as the starting point and the global reference path or target point as the guide, through hyperheuristic scheduling. Simultaneously, weight factors are set. The escape path is dynamically adjusted to achieve a balance between obstacle avoidance and passage efficiency. A pruning process is then performed, removing redundant path segments and updating the action value function for each path segment using an immediate reward function.

[0090] Corresponding to the aforementioned embodiment of a path planning method integrating hyperheuristic scheduling and QL, this application also provides an embodiment of a path planning system integrating hyperheuristic scheduling and QL.

[0091] Figure 3 This is a schematic diagram of an embodiment of a path planning system integrating hyperheuristic scheduling and QL provided in this application. Please refer to... Figure 3 The system provided in this embodiment includes a rasterization module, a path optimization module, and an optimal path module; wherein,

[0092] The rasterization module is used to build a raster map model, set the starting and target positions, and define map information for obstacles and passable areas.

[0093] The path optimization module is used to initialize the action value function based on the constructed raster map model, introduce heuristic information into the QL algorithm, calculate the evaluation function value of the heuristic operator LLH from the starting position to the target position, take the maximum value of the evaluation function value as the globally optimal reference path, and optimize and adjust the QL algorithm.

[0094] The optimal path module is used to continuously iterate and update the action value function through the optimized QL in the established gridded environment model to obtain the optimal path.

[0095] The following section provides a further description of the application and simulation of the QL algorithm and the QL algorithm of this invention in path planning.

[0096] 1. Environment and Parameter Settings: The simulation environment model of this invention is constructed using rasterization. This example provides, for instance... Figures 4-5 The simulation environment shown uses a 25×25 grid resolution, with each grid cell having a side length of 1cm. A relatively complex simulation environment is designed, with white areas representing passable areas and gray areas representing obstacles. The specific parameter settings in the algorithm are shown in Table 1.

[0097] Table 1 Simulation Parameters

[0098]

[0099] 2. Simulation Results and Analysis: Based on the above algorithm steps and simulation parameters, this invention conducted a simulation experiment on the above simulation environment.

[0100] 2.1. Simulation Analysis of Path Results: Figures 6-7The path results of the QL algorithm and the optimized QL algorithm in simulation environment 1 are shown. For a more intuitive comparison, [the following is a list of steps]. Figures 6-7 The performance data is recorded in Table 2.

[0101] Table 2 Performance Comparison

[0102]

[0103] Based on simulation results and data, in Environment 1, the optimized QL algorithm generates smoother paths and reduces redundant paths compared to the standard QL algorithm. This is reflected in the numerical values ​​shown in Table 2: in Environment 1, the optimized QL algorithm reduces time by 17.46 seconds, path length by 10.97 cm, and smoothness by 31.41 compared to the standard QL algorithm. Therefore, it outperforms the QL algorithm comprehensively in this environment.

[0104] Furthermore, Figures 8-9 The path results of the QL algorithm and the optimized QL algorithm in simulation environment 2 are shown; similarly, the path results of the optimized QL algorithm in simulation environment 2 are shown. Figures 8-9 The performance data is recorded in Table 3.

[0105] Table 3 Performance Comparison

[0106]

[0107] Similarly, combining simulation results and data, it can be seen that in Environment 2, the optimized QL algorithm generates smoother paths and reduces the generation of redundant paths compared to the QL algorithm. Furthermore, as shown in Table 3, in Environment 2, the optimized QL algorithm reduces the time by 25.90 seconds, the path length by 10.04 cm, and the smoothness by 53.39 compared to the QL algorithm. In summary, based on the above simulation results and data comparison, the optimized QL algorithm outperforms the QL algorithm in all performance indicators.

[0108] 2.2. Simulation analysis of dynamic learning rate: Figures 10-11 The graph illustrates the learning rate of the QL algorithm and its optimized counterpart in two different environments. As the learning rate curves show, the QL algorithm maintains a constant learning rate of 0.1, failing to dynamically adjust according to different learning stages. Therefore, its adaptability to complex environments is limited, resulting in low learning efficiency. In contrast, the optimized QL algorithm employs an adaptive learning rate strategy, dynamically adjusting the learning rate based on the learning progress to improve learning efficiency.

[0109] Specifically, the core idea of ​​the optimized QL algorithm in terms of dynamic learning rate is to have a certain ability to adapt to and escape from complex environments. When facing trap areas, the learning rate gradually increases during the training process, and after each period of learning, the current learning rate is maintained for a period of time to avoid drastic fluctuations caused by over-adjustment. This allows for better adaptation to the next trap environment and enables the learning of path planning to be completed in less time.

[0110] 2.3. Simulation Analysis of Algorithm Convergence: To verify the convergence performance and stability of the proposed method in complex environments, the convergence curves of the QL algorithm and the optimized QL algorithm were compared in two different environments, such as... Figures 12-13 As shown.

[0111] As can be seen from the curve changes in the figure, under the same initial conditions, the QL algorithm exhibits significant fluctuations in the early stages, with the step count curve showing large fluctuations, indicating that it is prone to getting trapped in local optima during the learning process and has a slow overall convergence speed. In contrast, the optimized QL algorithm, due to the introduction of a hyperheuristic scheduling mechanism and a dynamic learning rate strategy, can quickly establish a stable exploration direction in the early stages of iteration, enabling the algorithm to significantly reduce the number of path steps within a smaller number of iterations and showing a faster convergence trend.

[0112] Overall, the optimized QL algorithm demonstrates higher stability and global convergence capability in both simulation environments. Compared to the QL algorithm, it shows significant improvements in convergence speed, path planning efficiency, and algorithm stability, fully validating the effectiveness of this method in achieving path planning optimization by integrating hyperheuristic scheduling and QL.

[0113] 2.4. Simulation Analysis of Heuristic Operators: To further illustrate the correctness of the proposed hyperheuristic scheduling, Figures 14-15 The use of each heuristic operator is shown in two environments.

[0114] Specifically, simulation results from both environments show that the optimized QL algorithm can automatically schedule heuristic operators based on environmental changes, rationally allocating the usage ratio of different operators, and achieving synergy and balance among heuristic algorithms. The hyperheuristic scheduling mechanism achieves good coordination between global search and local optimization, not only improving the stability and efficiency of path planning but also effectively reducing the risk of the algorithm getting trapped in local optima.

[0115] Furthermore, at the beginning of learning, the operators are used more frequently because the environment is unfamiliar. As learning progresses and the environment is understood, the frequency of operator usage decreases. Simulation results fully verify the adaptability and intelligence of the optimized QL algorithm in multi-heuristic fusion scheduling. By dynamically calling three types of heuristic operators—A*, JPS, and Theta*—it can maintain high search performance and path quality under different environmental complexities, further demonstrating the rationality and superiority of the hyperheuristic scheduling mechanism in this invention.

[0116] The above description is merely a preferred embodiment of the present invention, used to illustrate the technical solution of the present invention, but is not intended to limit this application. The scope of protection of the present invention is not limited to the above embodiments. All equivalent substitutions or modifications made based on the technical concept of the present invention should be included within the scope of protection of the present invention.

Claims

1. A path planning method integrating hyperheuristic scheduling and QL, characterized in that, Includes the following steps: Step 1: Construct a rasterized map model, set the starting and target positions, and define map information for obstacles and passable areas; Step 2: Initialize the action value function based on the rasterized map model, introduce heuristic information into the QL algorithm, calculate the evaluation function value of the heuristic operator LLH from the starting position to the target position, take the maximum value of the evaluation function value as the globally optimal reference path, and optimize and adjust the QL algorithm. Step 3: In the established gridded environment model, the action value function is continuously updated through the optimized QL to obtain the optimal path.

2. The path planning method integrating hyperheuristic scheduling and QL according to claim 1, characterized in that, The specific implementation process of step 1 is as follows: Determine the working area and resolution: The working area represents the operating range in the environment, and the resolution is used to determine the accuracy of map division; Based on the resolution, the operating range is divided into grid areas, each grid representing the smallest identifiable unit in the environment. The grids are of the same size and are used to record the status information of obstacles or passage areas. Obtain information about obstacles within the environmental range; Assign a state value to each grid cell: grid cells without obstacles are marked as passable, and grid cells containing obstacles are marked as impassable, forming a complete environment map.

3. The path planning method integrating hyperheuristic scheduling and QL according to claim 2, characterized in that, The optimization and adjustment of the QL algorithm specifically includes the following methods; Based on the action value function, the hyperparameters of the standard QL algorithm are dynamically adjusted using an optimized learning rate model; In the initialization phase, the starting point and the target point are first determined, and the comprehensive performance of the heuristic operator is evaluated through the evaluation function. Based on the output value of the evaluation function, the optimal heuristic operator is selected to generate the corresponding optimal global reference path. Based on the optimized learning rate model, a hyperheuristic scheduling mechanism is introduced. When trapped and unable to escape, the evaluation function of different heuristic operators is calculated between the current grid point and the nearest grid point on the global optimal reference path. The heuristic operator with the maximum value obtained by the evaluation function is used as the escape path and direction. The intersection point generated by the escape path and the global optimal reference path is used as the escape point. By moving towards the target point while avoiding obstacles, after successfully reaching the target point for the first time, and before the second iteration, the optimal pruning strategy is generated by maximizing the evaluation function value of the heuristic operator between escape points.

4. The path planning method integrating hyperheuristic scheduling and QL according to claim 3, characterized in that, The implementation process of introducing heuristic information into the QL algorithm is as follows: Determining if a grid is an obstacle: During the initialization phase, if a grid is marked as an obstacle, it will not be included in the heuristic calculation of the action value function; If the grid is a passable area, heuristic information is calculated as the initialization action value function.

5. The path planning method integrating hyperheuristic scheduling and QL according to claim 4, characterized in that, The specific process of calculating the heuristic information is as follows: After determining that a grid is a passable area, the geometric distance between the target point and the 8 neighboring grid points centered on that grid point is calculated and incorporated into the action value function as heuristic information; the coordinates of the current grid are set as (x,y), then the movement direction attributes of up, down, left, right, upper left, lower left, upper right, and lower right are represented as (x,y+1), (x,y-1), (x-1,y), (x+1,y), (x-1,y+1), (x-1,y-1), (x+1,y+1), and (x+1,y-1) respectively; Calculate heuristic information: using the coordinates of the target point Compared with the current state The Euclidean distances of the coordinates of the 8 neighboring grid points are calculated, and the reciprocal of the Euclidean distance exponential function is used as heuristic information. Use heuristic information as the value function for initializing actions.

6. The path planning method integrating hyperheuristic scheduling and QL according to claim 5, characterized in that, The specific process for calculating the evaluation function value of the heuristic operator LLH from the starting position to the target position is as follows: In the QL algorithm, heuristic operators A*, JPS, and Theta* are selected. For the path length, number of nodes, number of visited nodes, and running time characteristics obtained after running each heuristic operator, the reciprocal of the weighted comprehensive value of each characteristic is taken as the performance index, which serves as the overall performance of different heuristic operators. The operator with the largest value among the performance indices is determined as the current optimal operator, and the maximum value of the evaluation function corresponding to the optimal operator is taken as the reference path of the global optimum, guiding the path optimization and strategy update in the subsequent learning process.

7. The path planning method integrating hyperheuristic scheduling and QL according to claim 6, characterized in that, The specific implementation process of the optimized learning rate model is as follows: When the QL algorithm falls into traps more than the preset maximum tolerance threshold during the learning process, the learning rate of the current stage remains unchanged. In subsequent learning processes, it is determined whether the number of times the student falls into a trap still exceeds a preset threshold; If the threshold is not exceeded, the learning rate is gradually increased until the iteration ends.

8. The path planning method integrating hyperheuristic scheduling and QL according to claim 7, characterized in that, The hyperheuristic scheduling mechanism is used as a control strategy to dynamically select multiple heuristic operators (LLHs) based on an evaluation function during the path planning process.

9. A path planning method integrating hyperheuristic scheduling and QL according to claim 8, characterized in that, After selecting a heuristic operator, the hyperheuristic scheduling mechanism updates the Q-value in the current action value function. During the Q-table update process, the optimal heuristic operator is selected based on the current state and environmental information to generate an escape path, and a weight factor is introduced into the Q-table to dynamically adjust the escape path.

10. A path planning method integrating hyperheuristic scheduling and QL according to claim 9, characterized in that, The optimal pruning strategy is implemented as follows: Based on the action value function corresponding to each direction in the current state, all possible movement directions are compared and selected; by combining the Q value of each action in the current state with the update result of the optimal heuristic operator in the hyperheuristic scheduling, the action value of different directions is weighted and corrected, and then the direction corresponding to the maximum value of the action value function is selected as the optimal movement direction for the next step. The set of all possible movement directions constitutes the action set, which is used by the QL algorithm to select the optimal action and update the strategy. In each movement, if the current direction is consistent with the optimal direction, then continue to move in that direction; if the current direction is inconsistent with the optimal direction, then adjust according to the greedy strategy to maintain the original direction or the optimal direction. The retained path is used as the current valid path to complete the online correction of the action value function.

11. A path planning system integrating hyperheuristic scheduling and QL, used to implement the path planning method according to any one of claims 1 to 10, characterized in that, Includes the following modules: The rasterization module is used to build a raster map model, set the starting and target positions, and define map information for obstacles and passable areas. The path optimization module initializes the action value function based on the raster map model, introduces heuristic information into the QL algorithm, calculates the evaluation function value of the heuristic operator LLH from the starting position to the target position, takes the maximum value of the evaluation function value as the globally optimal reference path, and optimizes and adjusts the QL algorithm. The optimal path module is used to continuously iterate and update the action value function through the optimized QL in the established gridded environment model to obtain the optimal path.

Images