Path planning method for UAV in hydraulic tunnel surveying based on learning strategy

Abstract

The present invention relates to a method for path planning for a hydraulic tunnel surveying drone based on a learning strategy, which is applicable to the field of surveying and mapping geographic information acquisition technology. The method comprises: generating a grid map based on the environment of a target hydraulic tunnel, wherein the side length of the grid is determined based on the minimum safe distance between the drone and obstacles; treating each traversable grid in the grid map as a state, and constructing an action set based on the movement direction of the drone from the center point of a grid to the center point of an adjacent grid in the grid map; using an improved Q-learning algorithm to perform path planning based on the start and end points of the path to be planned, in combination with the states and action sets in the grid map, to obtain a first path plan; and smoothing the first path plan using a quasi-uniform B-spline function to obtain a second path plan; the improved Q-learning algorithm employing a heuristic reward function.

Description

Technical Field

[0001] The present invention relates to a learning strategy-based hydraulic tunnel surveying and mapping UAV path planning method, which is applicable to the field of surveying and mapping geographic information acquisition technology. Background Art

[0002] Hydraulic tunnels are widely used in hydropower projects for water diversion, flood discharge, and drainage. They are characterized by narrow and long spaces, low light levels, high humidity, poor ventilation, and complex structures. Inclined and vertical shaft sections, in particular, often have large inclinations and lengths, posing significant challenges to manual inspections and detailed surveying. These environments make it difficult or impossible for workers to enter for extended periods, making traditional surveying methods inefficient, inaccurate, and potentially hazardous.

[0003] Unmanned aerial vehicle (UAV) path planning and obstacle avoidance technology is a key branch of automation and control, finding widespread application in industrial production and everyday life. Equipped with task-driven surveying and mapping instruments, UAVs can replace human operators in performing information collection and processing tasks in complex and even harsh environments. In recent years, with the widespread adoption of UAV platforms in industrial inspection, their flexible and efficient flight capabilities have made it possible to carry out hydraulic tunnel surveying using sensors such as lidar and cameras. However, as enclosed indoor spaces, hydraulic tunnels are unable to receive GNSS signals, making it impossible for UAVs to rely on conventional navigation systems for positioning and path control. Furthermore, tunnels often contain dense obstacles such as pipeline supports, temporary facilities, and variable-section structures, posing a threat to UAV flight safety. Therefore, enabling UAVs to perform autonomous path planning and obstacle avoidance in such complex environments is crucial for ensuring the safe, efficient, and accurate completion of tunnel surveying tasks.

[0004] Currently, many mature algorithms are used for UAV path planning, or have been modified to adapt to complex workspaces. These methods include, but are not limited to, algorithms based on search and sampling (ABSS), algorithms based on meta-heuristic intelligence (ABMHI), and algorithms based on Q-learning (ABQL). A review of the literature and various path planning algorithms reveals that, compared to ABSS and ABMHI, reinforcement learning (RL) algorithms, represented by Q-learning, offer significant advantages and represent a promising development direction. Both classic Q-learning (CQL) and various improved Q-learning algorithms aim to address shortcomings in path planning, such as high computational complexity, poor adaptability to multiple evaluation metrics, and convergence difficulties. However, many current Q-learning-based algorithms still only plan paths based on the shortest distance, while ignoring the energy losses associated with UAV motion. Furthermore, while the optimal path was obtained, it lacked smoothness due to discontinuous line segments, which reduced the drone's turning performance. These flaws posed a safety hazard when using surveying and mapping drones in hydraulic tunnels, and also affected the accuracy and efficiency of collecting environmental information. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: in response to the above-mentioned problems, a method for path planning of hydraulic tunnel surveying UAV based on learning strategy is provided.

[0006] The technical solution adopted by the present invention is: a method for path planning of a hydraulic tunnel surveying drone based on a learning strategy, comprising:

[0007] Based on the target hydraulic tunnel environment, a grid map is generated. The side length of the grid in the grid map is determined based on the minimum safe distance between the drone and the obstacle.

[0008] Each traversable grid in the grid map is considered as a state, and an action set is constructed based on the movement direction of the UAV from the center point of the grid to the center point of the adjacent grid in the grid map;

[0009] Based on the starting and ending points of the path to be planned, combined with the state and action set in the grid map, an improved Q-learning algorithm is used for path planning to obtain the first path solution;

[0010] The first path plan is smoothed by using a quasi-uniform B-spline function to obtain a second path plan;

[0011] The improved Q-learning algorithm adopts a heuristic reward function, which is constructed based on a base reward, a path length reward, and a path smoothness reward.

[0012] The side length of the grid in the grid map is determined based on the minimum safe distance between the drone and obstacles, including:

[0013] Assuming that the minimum safe distance between the UAV and the obstacle is a / 2, the side length of the grid in the grid map is a.

[0014] The heuristic reward function is constructed based on the base reward, path length reward, and path smoothness reward, including:

[0015] ;

[0016] in, is the heuristic reward function, R(t) is the basic reward, is the path length reward, is the path smoothness reward, α1 and α2 are the preset reward factors 0<α1, α2<1, and t is the iteration round.

[0017] The basic reward is determined according to the Q-learning strategy function and value function, and is designed based on the coordinates and attitude parameters of the current drone. The path length reward is determined based on path safety and introduces a weighted obstacle avoidance cost parameter. The path smoothness reward is determined based on the angle change when the drone avoids obstacles and incorporates the change in the path curvature radius.

[0018] The improved Q-learning algorithm uses a semi-gradient time-difference learning algorithm with eligibility traces to estimate the Q-value function.

[0019] The method of estimating the Q-value function using a semi-gradient time-difference learning algorithm with a qualification trace includes:

[0020] The Q function is decomposed into state-value function and action-value function for estimation;

[0021] When using the semi-gradient temporal difference learning algorithm with eligibility trace to estimate the Q function, the eligibility trace z(t) is first defined;

[0022] Secondly, the state-value function adds a weight update mechanism to estimate the sum of rewards associated with subsequent states; finally, the action-value function is updated based on the optimized Bellman equation as shown below to implement the iteration of the Q-value:

[0023] ;

[0024] in, is the heuristic reward function, represents the mean estimate, α, β are weight coefficients, represents the state-value function with adjustable weight w(t) under the current state s(t), and the policy represents the adaptive ε-greedy policy; z(t) is the eligibility trace at the current time t.

[0025] A hydraulic tunnel surveying and mapping UAV path planning device based on a learning strategy, comprising:

[0026] The grid generation module is used to generate a grid map based on the environment of the target hydraulic tunnel. The side length of the grid in the grid map is determined based on the minimum safe distance between the UAV and the obstacle.

[0027] The state-action definition module is used to treat each traversable grid in the grid map as a state and construct an action set based on the movement direction of the UAV from the center point of the grid to the center point of the adjacent grid in the grid map;

[0028] The first path module is used to perform path planning based on the starting and ending points of the path to be planned, combined with the state and action set in the grid map, using an improved Q-learning algorithm to obtain a first path solution;

[0029] A second path module is used to smooth the first path plan using a quasi-uniform B-spline function to obtain a second path plan;

[0030] The improved Q-learning algorithm adopts a heuristic reward function, which is constructed based on a base reward, a path length reward, and a path smoothness reward.

[0031] A storage medium stores a computer program that can be executed by a processor, and when the computer program is executed, the steps of the hydraulic tunnel surveying and mapping UAV path planning method based on a learning strategy are implemented.

[0032] A path planning device comprises a memory and a processor, wherein the memory stores a computer program executable by the processor, and when the computer program is executed, the steps of the hydraulic tunnel surveying and mapping UAV path planning method based on a learning strategy are implemented.

[0033] The beneficial effects of the present invention are:

[0034] (1) In view of the characteristics of narrow passages, tortuous structures, and dense obstacles inside hydraulic tunnels, the present invention adopts an improved Q-learning algorithm for path planning, enabling the surveying and mapping UAV to flexibly avoid obstacles such as walls, supporting structures, and equipment, effectively improving flight safety;

[0035] (2) Through the optimization of reinforcement learning strategies and the design of reward functions, the planned path not only meets the obstacle avoidance requirements but also takes into account the path length and curvature smoothness, ensuring that the UAV can move quickly and accurately in the tunnel and improving the efficiency of surveying and mapping operations;

[0036] (3) The present invention uses a quasi-uniform B-spline smoothing algorithm to post-process the path, which greatly improves the continuity and controllability of the path and reduces the risk of collision or increased energy consumption of the UAV due to steering difficulties at complex corners. It is particularly suitable for long-distance, fully enclosed hydraulic tunnel inspection tasks;

[0037] (4) The path planning method proposed in this paper has demonstrated good convergence speed and algorithm stability in multiple sets of simulations and physical experiments, verifying its practical value and promotion potential for autonomous navigation control of surveying and mapping UAVs in hydraulic tunnel scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of an embodiment.

[0039] Figure 2 This is the effect diagram of the embodiment on a two-dimensional grid map. The red solid line represents the path planned by IQLA; the other colored dotted lines represent the paths planned by commonly used intelligent evolutionary algorithms, namely the wolf pack algorithm (blue), sparrow search algorithm (yellow), and particle optimization algorithm (purple).

[0040] Figure 3 The effect diagram in the three-dimensional grid map, where (a) 3D simulation result and (b) horizontal section view. DETAILED DESCRIPTION

[0041] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and are not to be construed as limiting the present invention. The step numbers in the following embodiments are provided for ease of explanation only and do not limit the order of the steps. The order of execution of the steps in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0042] In the description of the present invention, "a plurality" means two or more. The terms "first" and "second" are used solely to distinguish technical features and are not to be construed as indicating or implying relative importance, or as implicitly indicating the number of the indicated technical features, or as implicitly indicating the order of the indicated technical features. Furthermore, unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art.

[0043] Example 1: This example is a method for planning a path for a hydraulic tunnel surveying drone based on a learning strategy, specifically comprising the following steps:

[0044] S100: Generate a grid map based on the environment of the target hydraulic tunnel, where the side length of a grid in the grid map is determined based on a minimum safe distance between the UAV and obstacles.

[0045] In view of the common environmental characteristics of hydraulic tunnels, such as narrow and long passages, frequent cross-section changes, and various types of obstacles (such as pipelines, supports, accumulated water, and equipment residues), this embodiment uses a two-dimensional grid method to model the tunnel working area. The internal space of the tunnel is discretized into N x ×N y The modeling method is used to identify and block obstacles and to provide a constraint boundary for subsequent path search.

[0046] To avoid collisions due to spatial proximity to obstacles, the side length of the grid in the grid map in this embodiment is determined based on the minimum safe distance between the drone and the obstacle. The minimum safe distance is set to a / 2, and the side length of the grid in the grid map is set to a.

[0047] S200: Consider each traversable grid in the grid map as a state, and construct an action set based on the movement direction of the UAV from the center point of a grid to the center point of an adjacent grid in the grid map.

[0048] Reinforcement learning learns knowledge through the interaction between the agent and the environment, selecting actions and iterating states based on feedback from environmental assessments. Given the tortuous paths, frequent obstacles, and high precision requirements of hydraulic tunnel surveying, this embodiment defines each accessible grid as a state S(t) that can be explored by the surveying drone. Incorporating the physical flight constraints and safety distance requirements of the surveying drone, an eight-directional motion scheme is employed to design the action space, enabling the drone to flexibly respond to turns and navigate around obstacles in tunnels. Within the configuration workspace designed in this embodiment, the surveying drone can move in any of eight directions (forward, back, left, right, right front, right back, left front, and left back) to the next feasible grid.

[0049] S300 , based on the starting and ending points of the path to be planned, combined with the state and action set in the grid map, an improved Q-learning algorithm is used to perform path planning to obtain a first path solution.

[0050] In this embodiment, the improved Q-learning algorithm adopts the following strategies to improve path quality and algorithm convergence during the path learning and updating process:

[0051] (1) Action selection strategy:

[0052] A reasonable action selection strategy can achieve a balance between exploration and exploitation in learning algorithms. The widely used ε-greedy strategy easily leads to confusion between exploration and exploitation, making it difficult for the algorithm to converge, resulting in the planned path being only a local optimum.

[0053] To overcome the shortcomings of the ε-greedy strategy, this embodiment adopts an adaptive ε-greedy policy as the action selection strategy, thereby encouraging extensive exploration of the entire hydraulic tunnel area in the initial stage and gradually converging to the optimal path decision in the later stages of training. This balances globality and convergence speed, effectively avoiding exploration obstacles caused by the complex structure inside the tunnel.

[0054] In this embodiment, an adaptive ε-greedy strategy is used to implement the strategy evolution of "extensive exploration in the early stage + tending to the optimal in the later stage", so that the surveying and mapping UAV can quickly identify feasible areas in complex tunnel topology and gradually converge to the shortest path solution.

[0055] (2) Heuristic reward function:

[0056] The reward function, used in the Q-learning algorithm to determine the value of each action, is key to reducing ineffective search and improving convergence speed. In CQL, the reward function is defined as positive when the target is reached, negative when an obstacle is collided, and zero otherwise. Using a constant reward function increases the number of blind steps in the drone's search, resulting in inefficient search.

[0057] Based on the comprehensive requirements of the hydraulic tunnel mission for the shortest path length, smooth turns, and low flight energy consumption, this embodiment constructs a heuristic reward function based on the basic reward, path length reward, and path smoothness reward. Compared with the constant reward design in CQL, the improved Q-learning algorithm can effectively suppress invalid path searches in complex obstacle-dense environments, accelerate convergence, and improve path quality, which is more in line with the dual goals of "both accuracy and safety" in actual surveying and mapping operations.

[0058] In this embodiment, the basic reward is determined based on the Q-learning strategy function and value function, and is designed based on the coordinates and attitude parameters of the current drone. The path length reward is determined based on path safety and introduces a weighted obstacle avoidance cost parameter. The path smoothness reward is determined based on the angle change when the drone avoids obstacles and incorporates the change in the path curvature radius.

[0059] The heuristic reward function in this embodiment is constructed based on the base reward, path length reward, and path smoothness reward, including:

[0060] ;

[0061] in, is the heuristic reward function, R(t) is the basic reward, Path length reward is the path smoothness reward, α1 and α2 are the preset reward factors, α1 and α2 are the preset reward factors, 0<α1, α2<1, and t is the iteration round.

[0062] In this example, a reward function model combining path length and turn smoothness is constructed to guide the drone to prefer decisions with shorter flight paths and softer turns, thereby meeting the requirements of low energy consumption, stable positioning, and continuous image acquisition in tunnel mapping.

[0063] (3) Q-value update mechanism:

[0064] In CQL, when temporal difference learning is used to iteratively approximate the Q function, there are defects such as unstable process and large approximation error.

[0065] Considering the complex structure of hydraulic tunnels, which leads to large state space dimensions and volatile search processes, this embodiment uses a semi-gradient temporal-difference (SGTD) learning algorithm with eligibility traces to replace traditional temporal-difference methods. This method dynamically evaluates subsequent state rewards, significantly improving the Q-function convergence accuracy and training stability, ensuring more reliable path selection for drones in variable-section tunnels.

[0066] In this embodiment, the Q function is decomposed into a state-value function and an action-value function for estimation. When estimating the Q function using SGTD, the eligibility trace z(t) is first defined. Secondly, a weight update mechanism is added to the state-value function to estimate the sum of rewards associated with subsequent states. Finally, the action-value function is updated based on the optimized Bellman equation shown below to implement the iteration of the Q-value:

[0067] ;

[0068] in, is the heuristic reward function, represents the mean estimate, α, β are weight coefficients, represents the state-value function with adjustable weight w(t) under the current state s(t), and the policy represents the adaptive ε-greedy policy; z(t) is the eligibility trace at the current time t.

[0069] This embodiment introduces semi-gradient temporal difference learning with qualification traces to estimate the Q-value function, overcoming the problem of unstable training of traditional methods when the state space is large, and improving the robustness of the algorithm in long tunnels and multi-branch structures.

[0070] S400 : Smoothing the first path plan using a quasi-uniform B-spline function to obtain a second path plan.

[0071] In order to further improve the turning smoothness and flight stability of the surveying and mapping UAV in the narrow space of the tunnel, this embodiment performs quasi-uniform B-spline (QUBS) optimization processing on the first path solution.

[0072] This embodiment decouples the control points through quasi-uniform B-spline functions and introduces control node vectors with higher degrees of freedom, thereby significantly improving the continuity and smoothness of the path while retaining its effectiveness, reducing flight deviations or instability caused by excessively large path turns, and is particularly suitable for long-distance, multi-curve tunnel space scenarios.

[0073] This embodiment uses the quasi-uniform B-spline function (QUBS) to smooth the original planned path to enhance the continuity and executability of the path. It is particularly suitable for operation scenarios with multiple sharp turns or asymmetric structures in the tunnel.

[0074] The second path plan generated in this embodiment is stored in the navigation module and combined with the mapping drone's onboard positioning system (such as an IMU or laser SLAM) for trajectory tracking control, enabling efficient mapping of the tunnel interior during autonomous flight. Experiments have shown that the planned path not only effectively avoids structural obstacles but also exhibits excellent geometric properties and flight stability, adapting to typical hydraulic tunnel operation scenarios such as long distances, low light conditions, and GNSS signal shielding.

[0075] Example 2: This example is a hydraulic tunnel mapping drone path planning device based on a learning strategy, comprising:

[0076] The grid generation module is used to generate a grid map based on the environment of the target hydraulic tunnel. The side length of the grid in the grid map is determined based on the minimum safe distance between the UAV and the obstacle.

[0077] The state-action definition module is used to treat each traversable grid in the grid map as a state and construct an action set based on the movement direction of the UAV from the center point of the grid to the center point of the adjacent grid in the grid map;

[0078] The first path module is used to perform path planning based on the starting and ending points of the path to be planned, combined with the state and action set in the grid map, using an improved Q-learning algorithm to obtain a first path solution;

[0079] A second path module is used to smooth the first path plan using a quasi-uniform B-spline function to obtain a second path plan;

[0080] The improved Q-learning algorithm adopts a heuristic reward function, which is constructed based on a base reward, a path length reward, and a path smoothness reward.

[0081] Example 3: This example is a storage medium on which a computer program that can be executed by a processor is stored. When the computer program is executed, the steps of the hydraulic tunnel surveying drone path planning method based on learning strategy described in Example 1 are implemented.

[0082] Example 4: This example is a path planning device having a memory and a processor. The memory stores a computer program that can be executed by the processor. When the computer program is executed, the steps of the hydraulic tunnel surveying drone path planning method based on learning strategy described in Example 1 are implemented.

[0083] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the above-mentioned functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the routine skills of an engineer. Therefore, a person skilled in the art can implement the present invention set forth in the claims using ordinary skills without undue experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0084] If the above functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the above methods in various embodiments of the present invention. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks.

[0085] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0086] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable media on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other media, and then editing, interpreting, or processing it in another suitable manner as necessary, and then storing it in a computer memory.

[0087] It should be understood that various components of the present invention may be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods may be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof may be used: a discrete logic circuit having logic gate circuits for implementing logic functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0088] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0089] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0090] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for path planning of hydraulic tunnel mapping UAV based on learning strategy, characterized in that: include: Based on the target hydraulic tunnel environment, a grid map is generated. The side length of the grid in the grid map is determined based on the minimum safe distance between the drone and the obstacle. Each traversable grid in the grid map is considered as a state, and an action set is constructed based on the movement direction of the UAV from the center point of the grid to the center point of the adjacent grid in the grid map; Based on the starting and ending points of the path to be planned, combined with the state and action set in the grid map, an improved Q-learning algorithm is used for path planning to obtain the first path solution; The first path plan is smoothed by using a quasi-uniform B-spline function to obtain a second path plan; The improved Q-learning algorithm adopts a heuristic reward function, which is constructed based on a base reward, a path length reward, and a path smoothness reward.

2. The method for hydraulic tunnel mapping UAV path planning based on learning strategy according to claim 1 is characterized in that: The side length of the grid in the grid map is determined based on the minimum safe distance between the drone and obstacles, including: Assuming the minimum safe distance between the UAV and the obstacle is a / 2, the side length of the grid in the grid map is a.

3. The method for path planning of hydraulic tunnel surveying drone based on learning strategy according to claim 1 is characterized in that: The heuristic reward function is constructed based on the base reward, path length reward, and path smoothness reward, including: ； in, is the heuristic reward function, As a basic reward, is the path length reward, is the path smoothness reward, α1 and α2 are preset reward factors, 0<α1, α2<1, and t is the iteration round.

4. The method for path planning of hydraulic tunnel surveying drone based on learning strategy according to claim 1 is characterized in that: The basic reward is determined according to the Q-learning strategy function and value function, and is designed based on the coordinates and attitude parameters of the current drone. The path length reward is determined based on path safety and introduces a weighted obstacle avoidance cost parameter. The path smoothness reward is determined based on the angle change when the drone avoids obstacles and incorporates the change in the path curvature radius.

5. The method for path planning of hydraulic tunnel surveying UAV based on learning strategy according to claim 1 is characterized in that: The improved Q-learning algorithm uses a semi-gradient time-difference learning algorithm with eligibility traces to estimate the Q-value function.

6. The method for path planning of hydraulic tunnel surveying UAV based on learning strategy according to claim 5 is characterized in that: The method of estimating the Q-value function using a semi-gradient time-difference learning algorithm with a qualification trace includes: The Q function is decomposed into state-value function and action-value function for estimation; When using the semi-gradient temporal difference learning algorithm with eligibility trace to estimate the Q function, the eligibility trace z(t) is first defined; Secondly, the state-value function adds a weight update mechanism to estimate the sum of rewards associated with subsequent states; finally, the action-value function is updated based on the optimized Bellman equation as shown below to implement the iteration of the Q-value: ； in, is the heuristic reward function, represents the mean estimate, α, β are weight coefficients, Indicates the current state With adjustable weights The state value function of , this strategy represents an adaptive ε-greedy strategy; is the eligibility trace at the current time t.

7. A hydraulic tunnel mapping drone path planning device based on learning strategy, characterized in that: include: The grid generation module is used to generate a grid map based on the environment of the target hydraulic tunnel. The side length of the grid in the grid map is determined based on the minimum safe distance between the UAV and the obstacle. The state-action definition module is used to treat each traversable grid in the grid map as a state and construct an action set based on the movement direction of the UAV from the center point of the grid to the center point of the adjacent grid in the grid map; The first path module is used to perform path planning based on the starting and ending points of the path to be planned, combined with the state and action set in the grid map, using an improved Q-learning algorithm to obtain a first path solution; A second path module is used to smooth the first path plan using a quasi-uniform B-spline function to obtain a second path plan; The improved Q-learning algorithm adopts a heuristic reward function, which is constructed based on a base reward, a path length reward, and a path smoothness reward.

8. A storage medium having stored thereon a computer program executable by a processor, characterized in that: When the computer program is executed, the steps of the hydraulic tunnel surveying drone path planning method based on learning strategy according to any one of claims 1 to 6 are implemented.

9. A path planning device comprising a memory and a processor, wherein the memory stores a computer program executable by the processor, wherein: When the computer program is executed, the steps of the hydraulic tunnel surveying drone path planning method based on learning strategy according to any one of claims 1 to 6 are implemented.

Images