Unmanned aerial vehicle inspection path planning system based on reinforcement learning and dynamic obstacle avoidance control method

Abstract

The invention relates to the field of unmanned aerial vehicles, and discloses an unmanned aerial vehicle inspection path planning system based on reinforcement learning and a dynamic obstacle avoidance control method. Comprising a multi-source data acquisition module, a hybrid prediction evaluation module, a target scene adaptation module, a hierarchical collaborative decision module, an action dynamic mapping module, a cross-layer consistent verification module, an environment disturbance calibration module and a system local database, and each module and the system local database are in bidirectional communication through a data bus; the multi-source data acquisition module acquires static environment information, historical motion data of dynamic obstacles and state information of the unmanned aerial vehicle, and generates a standardized data matrix; the hybrid predictive evaluation module receives a standardized data matrix. By adopting hybrid prediction logic of a rigid body kinematics equation and a hidden Markov model and combining historical motion data and real-time state parameters of a dynamic obstacle, an obstacle trajectory coordinate sequence and confidence within a future preset duration can be accurately output.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, specifically to a UAV inspection path planning system and dynamic obstacle avoidance control method based on reinforcement learning. Background Technology

[0002] Existing UAV inspection path planning technologies are mostly based on static environment model design, which lacks accuracy in predicting the movement trajectory of dynamic obstacles. They often use a single prediction algorithm, resulting in large trajectory deviations and making it difficult to avoid moving obstacles in advance. At the same time, traditional systems have not established a dynamic threat level assessment mechanism, and the collision probability calculation relies only on a single distance index, which cannot take into account key parameters such as obstacle speed and acceleration. This can easily lead to problems such as untimely obstacle avoidance response or excessive obstacle avoidance, resulting in disordered inspection paths.

[0003] Currently, the weights of the reinforcement learning reward functions in UAV inspection systems are mostly fixed values, which cannot be flexibly adjusted according to the complexity indicators of the scene, such as obstacle density, target point distribution, and dynamic obstacle ratio in the inspection area. This makes it difficult to balance core requirements such as inspection coverage, path length, and energy consumption in complex scenarios. In addition, existing systems are not adaptable enough to environmental disturbances such as wind speed, airflow, and sensor noise, and lack targeted parameter calibration mechanisms. They are prone to decision-making biases in disturbed environments, affecting the stability and reliability of inspection tasks. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a UAV inspection path planning system and a dynamic obstacle avoidance control method based on reinforcement learning, which solves the problem that the weights of the reinforcement learning reward function in current UAV inspection systems are mostly fixed values ​​and cannot be adjusted flexibly and adaptively.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a UAV inspection path planning system based on reinforcement learning. Preferably, the UAV inspection path planning system based on reinforcement learning includes a multi-source data acquisition module, a hybrid prediction and evaluation module, a target scene adaptation module, a hierarchical collaborative decision-making module, a dynamic action mapping module, a cross-layer consistency verification module, an environmental disturbance calibration module, and a system local database. Each module and the system local database communicate bidirectionally via a data bus. The multi-source data acquisition module acquires static environmental information, historical motion data of dynamic obstacles, and the UAV's own state information, and generates a standardized data matrix. The hybrid prediction and evaluation module receives the standardized data matrix, retrieves historical motion data of dynamic obstacles, and outputs a predicted trajectory coordinate sequence of dynamic obstacles, trajectory confidence, and collision detection parameters. The system comprises the following modules: a collision probability and threat level matrix; a target scene adaptation module receiving the threat level matrix, static environment information, and historical motion data of dynamic obstacles to generate a target priority sequence and a reinforcement learning reward function; a hierarchical collaborative decision-making module receiving static environment information, the target priority sequence output by the target scene adaptation module, and the threat level matrix output by the hybrid prediction and evaluation module to generate hierarchical decision results; an action dynamic mapping module receiving the threat level matrix and the UAV's own state information to calculate action granularity and allocate flight commands for adjustment step size; a cross-layer consistency verification module receiving the hierarchical decision results and the target priority sequence to verify consistency and generate reward correction commands and granularity reduction commands; and an environmental disturbance calibration module collecting environmental disturbance data and quantifying its level, combining it with the UAV's own state information to generate calibration commands.

[0006] Preferably, the multi-source data acquisition module includes a lidar, a millimeter-wave radar, an inertial measurement unit, a high-definition camera, and a power sensor; static environmental information includes terrain features of the inspection area, three-dimensional coordinates and contours of fixed obstacles, dynamic environmental information includes the real-time position of moving obstacles, the speed of dynamic obstacles, the acceleration of dynamic obstacles, and the direction of motion, and the UAV's own status information includes flight speed, altitude, attitude angle, and remaining battery power; the standardized data matrix is ​​generated through timestamp synchronization, spatial coordinate calibration, and Kalman filtering.

[0007] Preferably, the hybrid prediction and evaluation module achieves hybrid prediction by combining rigid body kinematics equations and hidden Markov models, calculates the theoretical motion trajectory and corrects deviations; the collision probability is calculated by combining the UAV's own state information, historical motion data of dynamic obstacles and preset attenuation coefficients, wherein the preset attenuation coefficients are manually input by human experience; the threat level is divided according to the collision probability, the speed of dynamic obstacles and the relative distance.

[0008] Preferably, the reinforcement learning reward function weights of the target scene adaptation module include inspection coverage, path length, obstacle avoidance safety distance, and UAV energy consumption, with the sum of the four weights being 1. The inspection coverage, path length, and obstacle avoidance safety distance are calculated based on the standardized data matrix output by the multi-source data acquisition module, and the UAV energy consumption is calculated based on the UAV's own state information output by the multi-source data acquisition module.

[0009] Preferably, the action granularity of the action dynamic mapping module is calculated based on the threat level matrix and the UAV's own state information through a preset proportional relationship; the flight command adjustment step size includes the steering angle adjustment step size and the speed adjustment step size, and is allocated according to the rule that low threat and high speed correspond to coarse step size and high threat and low speed correspond to fine step size.

[0010] The reinforcement learning-based dynamic obstacle avoidance control method for UAVs includes the following steps: S1. Multi-source data acquisition and preprocessing: The multi-source data acquisition module collects static environmental information, dynamic environmental information, and UAV's own status information, generates a standardized data matrix, and stores the historical motion data of dynamic obstacles in the system's local database. S2. Dynamic obstacle trajectory prediction and threat assessment: The hybrid prediction and assessment module outputs the dynamic obstacle prediction trajectory coordinate sequence and trajectory confidence through rigid body kinematics equations and hidden Markov models, calculates the collision probability and classifies the threat level, and generates a threat level matrix, in which the threat level is divided into 1-5 levels. S3. Target Priority Update and Scene Adaptation: The target scene adaptation module adjusts the target priority to generate a target priority sequence, which is divided into 1-5 levels. Based on static environment information and dynamic environment information, the obstacle density, target point density, and dynamic obstacle ratio are calculated, and the weight of the reinforcement learning reward function is dynamically adjusted to generate a dynamic weight reward pool. S4. Hierarchical Collaborative Decision Making and Dynamic Action Mapping: The hierarchical collaborative decision making module outputs the globally optimal path and basic flight commands. The dynamic action mapping module calculates the action granularity and allocates flight commands with adjustment step sizes based on the threat level matrix and the UAV's own state information, generating preliminary flight control commands. S5. Cross-layer Consistency Verification and Environmental Disturbance Calibration: The cross-layer consistency verification module verifies the consistency between the hierarchical decision results and the target priority sequence and generates reward correction commands or granularity reduction commands. The environmental disturbance calibration module collects environmental disturbance data and quantifies it into disturbance intensity levels, generating calibration commands to perform final calibration on the preliminary flight control commands. S6. Command Execution and Data Storage: Execute the calibrated flight control commands, and the hierarchical collaborative decision-making module stores the disturbance intensity and decision correction coefficient mapping relationship in the system's local database.

[0011] Preferably, in step S2, the hybrid prediction and evaluation module calculates the real-time relative distance between the UAV and the obstacle using the Euclidean distance algorithm; the dynamic obstacle prediction trajectory coordinate sequence is a set of continuous coordinates within a preset time period output by the hybrid prediction and evaluation module, and the preset time period is manually input by the operator based on experience.

[0012] Preferably, the target priority adjustment rule executed by the target scene adaptation module in S3 is as follows: when a high-priority target is expected to be subject to a high-level threat of level 4-5 within a preset time period, its priority is increased; when a low-priority target is occupied by a medium-to-high-level threat of level 3-5 for a long period of time, it is marked as pending inspection and its priority is decreased, with the priority adjustment amount being a system preset value.

[0013] Preferably, in step S4, the hierarchical collaborative decision-making module outputs the globally optimal path and basic flight instructions, specifically including the following steps: using the weighted summation of four weight indicators to solve for the optimal path, the four weight indicators include inspection coverage, path length, obstacle avoidance safety distance, and UAV energy consumption; the reinforcement learning algorithm is the PPO algorithm or SAC algorithm adopted by the hierarchical collaborative decision-making module.

[0014] Preferably, the environmental disturbance data collected by the environmental disturbance calibration module in S5 includes wind speed, airflow intensity, and sensor noise amplitude, which are quantified into disturbance intensity levels of 1-3 through preset weighting rules; the calibration instruction is an instruction generated by the environmental disturbance calibration module, including increasing the weight ratio of the rigid body kinematic equation in the hybrid prediction and evaluation module, shortening the decision response cycle of the hierarchical collaborative decision-making module, and reducing the action granularity of the action dynamic mapping module to ensure decision stability under disturbance conditions.

[0015] This invention provides a reinforcement learning-based UAV inspection path planning system and dynamic obstacle avoidance control method. It offers the following advantages: This invention employs a hybrid prediction logic combining rigid body kinematics equations and hidden Markov models, along with historical motion data of dynamic obstacles and real-time state parameters. It can accurately output the trajectory coordinate sequence and confidence level of obstacles within a preset time period. Furthermore, by classifying threat levels from 1 to 5 using multi-dimensional indicators, it significantly improves the prediction accuracy of dynamic obstacle avoidance and effectively reduces the risk of collision.

[0016] This invention dynamically adjusts the weights of the reinforcement learning reward function based on the scenario complexity index, achieving a dynamic balance between inspection coverage, path length, obstacle avoidance safety distance, and energy consumption. Furthermore, through a dynamic target priority update mechanism, the inspection sequence can be optimized according to the threat status, significantly improving inspection efficiency and task completion in different scenarios.

[0017] This invention adds an environmental disturbance calibration module, which quantifies the intensity levels of disturbances such as wind speed, airflow, and sensor noise, and adjusts the prediction model weights, decision response cycles, and action granularity accordingly. At the same time, it combines a cross-layer consistency verification mechanism to correct decision deviations, ensuring the stability of UAV flight control and the reliability of path planning under complex disturbance environments. Attached Figure Description

[0018] Figure 1 This is a structural diagram of the present invention; Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example: Please see the appendix Figure 1 This invention provides a UAV inspection path planning system based on reinforcement learning, including a multi-source data acquisition module, a hybrid prediction and evaluation module, a target scene adaptation module, a hierarchical collaborative decision-making module, an action dynamic mapping module, a cross-layer consistency verification module, and an environmental disturbance calibration module. Each module achieves bidirectional communication through a data bus and transmits data and control commands according to preset logic, forming a closed-loop system of data acquisition, processing, decision-making, and calibration.

[0021] The multi-source data acquisition module is mounted on the UAV fuselage and directly connects to the hybrid prediction and evaluation module and the environmental disturbance calibration module. Its core function is to collect basic operational data of the system. The module includes a lidar, millimeter-wave radar, inertial measurement unit, high-definition camera, and power sensor, and collects three types of information: static environmental information, namely the terrain features of the inspection area and the three-dimensional coordinates and contours of fixed obstacles; dynamic environmental information, namely the real-time position of moving obstacles. ,speed acceleration and direction of motion; the drone's own state information, i.e., flight speed. ,high Attitude angle and remaining battery power During data acquisition, the module achieves consistency alignment of multi-sensor data through timestamp synchronization and spatial coordinate calibration algorithms. After noise interference is eliminated by Kalman filtering, a standardized data matrix is ​​generated and output to the hybrid prediction and evaluation module in real time. Simultaneously, the UAV's own status information is synchronized to the environmental disturbance calibration module, and the historical motion data of dynamic obstacles is stored in the system's local database to provide data support for subsequent prediction logic. The hybrid prediction and evaluation module connects the multi-source data acquisition, target scene adaptation, hierarchical collaborative decision-making, and environmental disturbance calibration modules. Its core functions are dynamic obstacle trajectory prediction and threat level assessment. The module receives the standardized data matrix output by the multi-source data acquisition module, calls the historical obstacle motion data stored in the system's local database, starts the hybrid prediction logic of "rigid body kinematics equations + hidden Markov model", and outputs the future preset duration. Predicted trajectory coordinate sequence and trajectory confidence The module is based on the obstacle locations in a standardized data matrix. ,speed Data calculation of collision probability Combined with the speed of the obstacle movement The system classifies obstacles into threat levels 1-5 according to preset rules, generating a threat level matrix. The module outputs the predicted trajectory coordinate sequence and threat level matrix to the target scene adaptation module and the hierarchical collaborative decision-making module. It receives parameter calibration instructions from the environmental disturbance calibration module and dynamically adjusts the weight ratio of sub-models in the hybrid prediction model. The target scene adaptation module connects to the hybrid prediction evaluation, hierarchical collaborative decision-making, and cross-layer consistency verification modules. Its core function is to dynamically update the priority of inspection targets and the weights of the reinforcement learning reward function. The module presets initial priority quantification scoring rules for inspection targets and assigns initial scores to different types of targets according to the inspection task requirements. By combining the threat level matrix output by the hybrid prediction and assessment module with the dynamic environmental information output by the multi-source data acquisition module, an updated target priority sequence is generated. Based on the static and dynamic environmental information output by the multi-source data acquisition module, the module obtains the scene complexity index through a density statistics algorithm and dynamically adjusts the inspection coverage according to the scene type. Path length Obstacle avoidance safe distance Drone energy consumption The weights of the four indicators are used to generate a dynamic weighted reward pool. The module outputs the target priority sequence and the dynamic weighted reward pool to the hierarchical collaborative decision-making module, and receives reward correction instructions from the cross-layer consistency verification module to adjust the weight allocation in real time.

[0022] The hierarchical collaborative decision-making module is the core decision-making unit of the system, connecting the hybrid prediction and evaluation, target scenario adaptation, dynamic action mapping, cross-layer consistency verification, and environmental disturbance calibration modules. Its core function is to output the UAV flight path and control commands. The module has a built-in electronic map of the inspection area, which is the system's preset basic geographic data. It includes a global path optimization submodule and a local obstacle avoidance control submodule. The two submodules share the dynamic weight reward pool output by the target scenario adaptation module through an internal data interaction interface. The global path optimization submodule receives the electronic map, target priority sequence, and threat level matrix, and optimizes the global path through a reinforcement learning algorithm, outputting the global optimal path node coordinate sequence. The local obstacle avoidance control submodule receives the real-time location of obstacles, threat level, UAV status information and basic flight commands. It calculates the fine-tuning amount of flight commands within the global path framework. The module receives the action granularity parameters output by the action dynamic mapping module, the verification results output by the cross-layer consistency verification module, and the response cycle adjustment command output by the environmental disturbance calibration module. Finally, it outputs the integrated flight control commands to the UAV actuators.

[0023] The motion dynamic mapping module connects the hybrid prediction and assessment, hierarchical collaborative decision-making, cross-layer consistency verification, and environmental disturbance calibration modules. Its core function is to dynamically allocate flight commands and adjust the step size. The module has a preset adaptive motion granularity calculation formula and receives the threat level T and the UAV flight speed amplitude. The module calculates the action granularity G by substituting the values ​​into the formula, and dynamically allocates the steering angle adjustment step size and speed adjustment step size according to the system's preset rules. The module outputs the adjustment step size parameters to the local obstacle avoidance control submodule of the hierarchical collaborative decision-making module. It receives granularity reduction instructions from the cross-layer consistency verification module and granularity refinement instructions from the environmental disturbance calibration module, and adjusts the step size allocation rules in real time. The cross-layer consistency verification module connects to the target scene adaptation, hierarchical collaborative decision-making, and action dynamic mapping modules. Its core function is to verify the consistency and rationality of the hierarchical decision-making results. The module presets path deviation, including deviation angle. Deviation distance Matching degree with target access time sequence The threshold standard is used to receive global path information and local adjustment instructions from the hierarchical collaborative decision-making module in real time, and calculate two verification indicators by combining the target priority sequence output by the target scenario adaptation module. When both indicators meet the threshold requirements, the current system parameters remain unchanged. When either indicator exceeds the threshold, a reward correction instruction is generated and output to the target scene adaptation module, and a granularity reduction instruction is generated and output to the action dynamic mapping module.

[0024] The environmental disturbance calibration module connects the multi-source data acquisition module, the hybrid prediction and evaluation module, the hierarchical collaborative decision-making module, and the action dynamic mapping module. Its core function is to sense environmental disturbances and calibrate system parameters. The module includes a wind speed sensor, an airflow sensor, and a noise detection unit, and collects wind speed data. airflow intensity Sensor noise amplitude Environmental disturbance data are converted into disturbance intensity levels of 1-3 using a quantization and grading algorithm. Based on disturbance intensity level The module generates three calibration commands and outputs them to the hybrid prediction and evaluation module, the hierarchical collaborative decision-making module, and the action dynamic mapping module, respectively. Simultaneously, the module establishes the disturbance intensity... The decision correction coefficient mapping relationship library is stored in the system's built-in reinforcement learning experience replay pool, providing rapid parameter adaptation support for subsequent similar scenarios. It also receives the UAV's own state information output by the multi-source data acquisition module in real time, serving as auxiliary reference data for disturbance calibration.

[0025] Please see the appendix Figure 2 This invention also provides a dynamic obstacle avoidance control method for unmanned aerial vehicles based on reinforcement learning, comprising the following steps: S1: Multi-source data acquisition and preprocessing. The multi-source data acquisition module activates the lidar, millimeter-wave radar, inertial measurement unit, high-definition camera, and power sensor to simultaneously acquire static environmental information, dynamic environmental information, and the UAV's own status information. It achieves consistency and alignment of multi-sensor data through timestamp synchronization and spatial coordinate calibration algorithms. After Kalman filtering to eliminate noise interference, a standardized data matrix is ​​generated and output to the hybrid prediction and evaluation module in real time. The UAV's own status information is synchronized to the environmental disturbance calibration module, and the historical motion data of dynamic obstacles is stored in the system's local database.

[0026] S2: The hybrid prediction and assessment module for dynamic obstacle trajectory prediction and threat assessment receives a standardized data matrix, calls the historical obstacle motion data stored in the system's local database, and initiates the hybrid prediction logic of rigid body kinematics equations and hidden Markov models, based on the dynamic obstacle velocity. acceleration Data, derived from the equations of rigid body kinematics: Calculate the theoretical motion trajectory, where To predict the time step, a hidden Markov model is used to correct trajectory deviations and output the preset future duration. Predicted trajectory coordinate sequence and trajectory confidence Obstacle locations based on standardized data matrix ,speed The data is used to calculate the collision probability using a collision probability algorithm. The formula is: in The real-time relative distance between the drone and the obstacle is calculated using the Euclidean distance formula. calculate, Real-time location of the drone; It is the relative velocity; , The system is pre-set with distance and speed attenuation coefficients, combined with the obstacle's movement speed. The system classifies obstacles into threat levels 1-5 according to preset rules, generates a threat level matrix, and outputs the predicted trajectory coordinate sequence and threat level matrix to the target scene adaptation module and the hierarchical collaborative decision-making module.

[0027] S3: Target Priority Update and Scenario Adaptation. The target scenario adaptation module assigns initial scores to different types of targets according to the inspection task requirements. It receives the threat level matrix output by the hybrid prediction and assessment module, combines it with the dynamic environmental information output by the multi-source data acquisition module, and judges the threat status of the target's surroundings within a preset time period in real time. When a high-priority target is expected to be surrounded by a high-level threat, its priority is increased. When a low-priority target is occupied by a medium- to high-level threat for an extended period, it is marked as "pending inspection" and its priority is lowered. , The system is pre-set with adjustment parameters, and an updated target priority sequence is generated. Based on the static and dynamic environmental information output by the multi-source data acquisition module, the obstacle density is obtained through a density statistics algorithm. Target point density The proportion of dynamic obstacles Scenario complexity metrics, among which The number of dynamic obstacles. Given the total number of obstacles, the inspection coverage rate is dynamically adjusted according to the scenario type using the following formula. Path length Obstacle avoidance safe distance Drone energy consumption The weights of the four indicators satisfy... in Pre-set normalization coefficients for the system. The maximum battery level of the drone is determined; a dynamic weighted reward pool is generated, and the target priority sequence and the dynamic weighted reward pool are output to the hierarchical collaborative decision-making module.

[0028] S4: Hierarchical Collaborative Decision Making and Dynamic Action Mapping. The global path optimization submodule of the hierarchical collaborative decision making module receives an electronic map, target priority sequence, and threat level matrix, and uses dynamic weighted summation to maximize the objective function. The inspection coverage rate is represented by L, and the path length is represented by L. To achieve the obstacle avoidance safety distance target, where E represents actual energy consumption, a reinforcement learning algorithm is used to optimize the global path, outputting a sequence of globally optimal path node coordinates. And basic flight commands, where the reinforcement learning algorithm can be either PPO or SAC algorithm, and the motion dynamic mapping module uses an adaptive motion granularity calculation formula: Where k is the system's preset proportional coefficient based on UAV performance calibration, and T is the threat level (level 1-5) output by the hybrid prediction and evaluation module: The amplitude of the drone's flight speed; the received threat level T and the amplitude of the flight speed. Substituting the values ​​into the formula to calculate the action granularity G, and according to the system's preset rules of coarse step size corresponding to low threat and high speed, and fine step size corresponding to high threat and low speed, the steering angle is dynamically allocated to adjust the step size. Adjusting the speed step size The step size parameter will be adjusted and output to the local obstacle avoidance control submodule of the hierarchical collaborative decision-making module, whereby... , Based on the basic adjustment step size, the local obstacle avoidance control submodule receives the real-time position of obstacles, threat level, UAV's own status information and basic flight commands. Under the global path framework, it optimizes and calculates the fine-tuning amount of flight commands through local reinforcement learning, and adjusts the range according to the action granularity parameters output by the action dynamic mapping module to generate preliminary flight control commands.

[0029] S5: Cross-layer Consistency Verification and Environmental Disturbance Calibration. The cross-layer consistency verification module receives global path information and preliminary flight control commands output by the hierarchical collaborative decision-making module in real time. Combined with the target priority sequence output by the target scenario adaptation module, it calculates two verification indicators using the following formula: in , These are the global path and the locally adjusted velocity direction vectors, respectively. The drone's position after partial adjustments; This refers to the node position corresponding to the global path. This is an indicator function; it takes the value 1 if the access order is consistent, and 0 otherwise. This refers to the actual access order; The preset access order is defined by n, which represents the number of high-priority targets. When both indicators meet the threshold requirements, the current system parameters remain unchanged. If either indicator exceeds the threshold, a reward correction instruction is generated and output to the target scenario adaptation module, while a granularity reduction instruction is generated and output to the action dynamic mapping module. The process returns to step S4 to recalculate the flight control instructions, and the environmental disturbance calibration module collects wind speed data. airflow intensity Sensor noise amplitude Environmental disturbance data, etc., are processed using a quantization and grading algorithm: The disturbance data is converted into a disturbance intensity level of 1-3. ,in These are weighting coefficients, based on the disturbance intensity level. Three calibration instructions are generated: a model weight adjustment instruction is output to the hybrid prediction and evaluation module; a response cycle adjustment instruction is output to the hierarchical collaborative decision-making module to shorten the decision response cycle. , This serves as the initial cycle; it outputs granular refinement commands to the motion dynamic mapping module for final calibration of flight control commands.

[0030] S6: Command Execution and Data Storage. The UAV executes the calibrated final flight control commands, completes inspection path planning and dynamic obstacle avoidance maneuvers, and the environmental disturbance calibration module measures the disturbance intensity. The mapping relationship of decision correction coefficients is stored in the reinforcement learning experience replay pool, providing rapid parameter adaptation support for subsequent similar scenarios.

[0031] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A UAV inspection path planning system based on reinforcement learning, characterized in that, It includes a multi-source data acquisition module, a hybrid prediction and evaluation module, a target scene adaptation module, a hierarchical collaborative decision-making module, an action dynamic mapping module, a cross-layer consistency verification module, an environmental disturbance calibration module, and a system local database. Each module and the system local database communicate bidirectionally through a data bus. The multi-source data acquisition module collects static environmental information, historical motion data of dynamic obstacles, and the UAV's own status information, and generates a standardized data matrix; The hybrid prediction and evaluation module receives a standardized data matrix, calls historical motion data of dynamic obstacles, and outputs a dynamic obstacle predicted trajectory coordinate sequence, trajectory confidence, collision probability, and threat level matrix. The target scene adaptation module receives a threat level matrix, static environment information, and historical motion data of dynamic obstacles, and generates a target priority sequence and a reinforcement learning reward function. The hierarchical collaborative decision-making module receives static environmental information, the target priority sequence output by the target scenario adaptation module, and the threat level matrix output by the hybrid prediction and evaluation module, and generates hierarchical decision-making results. The motion dynamic mapping module receives the threat level matrix and the UAV's own status information, calculates the motion granularity, and allocates flight commands to adjust the step size. The cross-layer consistency verification module receives the hierarchical decision results and target priority sequence, verifies consistency, and generates reward correction instructions and granularity reduction instructions. The environmental disturbance calibration module collects environmental disturbance data and quantifies its level, and generates calibration instructions by combining the UAV's own status information.

2. The UAV inspection path planning system based on reinforcement learning according to claim 1, characterized in that, The multi-source data acquisition module includes a lidar, millimeter-wave radar, inertial measurement unit, high-definition camera, and power sensor; static environmental information includes terrain features of the inspection area, three-dimensional coordinates and contours of fixed obstacles, dynamic environmental information includes real-time position of moving obstacles, speed of dynamic obstacles, acceleration of dynamic obstacles, and direction of motion, and UAV's own status information includes flight speed, altitude, attitude angle, and remaining power; the standardized data matrix is ​​generated after timestamp synchronization, spatial coordinate calibration, and Kalman filtering.

3. The UAV inspection path planning system based on reinforcement learning according to claim 2, characterized in that, The hybrid prediction and evaluation module achieves hybrid prediction by combining rigid body kinematics equations and hidden Markov models, calculates theoretical motion trajectories and corrects deviations. The collision probability is calculated by combining the drone's own state information, historical motion data of dynamic obstacles, and preset attenuation coefficients, where the preset attenuation coefficients are manually entered based on human experience; the threat level is divided according to the collision probability, the speed of dynamic obstacles, and the relative distance.

4. The UAV inspection path planning system based on reinforcement learning according to claim 3, characterized in that, The reinforcement learning reward function weights of the target scene adaptation module include inspection coverage, path length, obstacle avoidance safety distance, and UAV energy consumption. The sum of the four weights is 1. The inspection coverage, path length, and obstacle avoidance safety distance are calculated based on the standardized data matrix output by the multi-source data acquisition module, and the UAV energy consumption is calculated based on the UAV's own state information output by the multi-source data acquisition module.

5. The UAV inspection path planning system based on reinforcement learning according to claim 4, characterized in that, The action granularity of the action dynamic mapping module is calculated based on the threat level matrix and the UAV's own state information through a preset proportional relationship. Flight command adjustment step size includes turn angle adjustment step size and speed adjustment step size, and is allocated according to the rule that low threat and high speed correspond to coarse step size and high threat and low speed correspond to fine step size.

6. A dynamic obstacle avoidance control method for unmanned aerial vehicles (UAVs) based on reinforcement learning, implemented based on the UAV inspection path planning system based on reinforcement learning as described in claims 1-5, characterized in that... Includes the following steps: S1. Multi-source data acquisition and preprocessing: The multi-source data acquisition module collects static environmental information, dynamic environmental information, and UAV's own status information, generates a standardized data matrix, and stores the historical motion data of dynamic obstacles in the system's local database. S2. Dynamic obstacle trajectory prediction and threat assessment: The hybrid prediction and assessment module outputs the dynamic obstacle prediction trajectory coordinate sequence and trajectory confidence through rigid body kinematics equations and hidden Markov models, calculates the collision probability and classifies the threat level, and generates a threat level matrix, in which the threat level is divided into 1-5 levels. S3. Target Priority Update and Scene Adaptation: The target scene adaptation module adjusts the target priority to generate a target priority sequence, which is divided into 1-5 levels. Based on static environment information and dynamic environment information, the obstacle density, target point density, and dynamic obstacle ratio are calculated to dynamically adjust the weight of the reinforcement learning reward function and generate a dynamic weight reward pool. S4. Hierarchical collaborative decision-making and dynamic motion mapping: The hierarchical collaborative decision-making module outputs the global optimal path and basic flight commands, and the dynamic motion mapping module calculates the action granularity and allocates flight commands to adjust the step size based on the threat level matrix and the UAV's own state information, generating preliminary flight control commands. S5. Cross-layer consistency verification and environmental disturbance calibration: The cross-layer consistency verification module verifies the consistency between the hierarchical decision results and the target priority sequence and generates reward correction instructions or granularity reduction instructions. The environmental disturbance calibration module collects environmental disturbance data and quantifies it into disturbance intensity levels, and generates calibration instructions to perform final calibration on the initial flight control instructions. S6. Command Execution and Data Storage: Execute the calibrated flight control commands, and the hierarchical collaborative decision-making module stores the disturbance intensity and decision correction coefficient mapping relationship in the system's local database.

7. The UAV dynamic obstacle avoidance control method based on reinforcement learning according to claim 6, characterized in that, In S2, the hybrid prediction and evaluation module calculates the real-time relative distance between the UAV and the obstacle using the Euclidean distance algorithm; the dynamic obstacle prediction trajectory coordinate sequence is a set of continuous coordinates within a preset time period output by the hybrid prediction and evaluation module, and the preset time period is manually input by the operator based on experience.

8. The UAV dynamic obstacle avoidance control method based on reinforcement learning according to claim 6, characterized in that, The target priority adjustment rule executed by the target scene adaptation module in S3 is as follows: when a high-priority target is expected to be threatened with a level 4-5 threat within a preset time period, its priority will be increased. When a low-priority target is occupied by a medium-to-high level threat of threat level 3-5 for a long period of time, it is marked as pending inspection and its priority is downgraded. The priority adjustment amount is the system preset value.

9. The dynamic obstacle avoidance control method for unmanned aerial vehicles based on reinforcement learning according to claim 6, characterized in that, In S4, the hierarchical collaborative decision-making module outputs the global optimal path and basic flight instructions, specifically including the following steps: using the weighted summation of four weight indicators to solve for the optimal path, the four weight indicators include inspection coverage, path length, obstacle avoidance safety distance, and UAV energy consumption; the reinforcement learning algorithm is the PPO algorithm or SAC algorithm adopted by the hierarchical collaborative decision-making module.

10. The UAV dynamic obstacle avoidance control method based on reinforcement learning according to claim 6, characterized in that, The environmental disturbance data collected by the environmental disturbance calibration module in S5 includes wind speed, airflow intensity, and sensor noise amplitude, which are quantified into disturbance intensity levels of 1-3 through preset weighting rules. The calibration instructions are generated by the environmental disturbance calibration module, including increasing the weight ratio of the rigid body kinematic equation in the hybrid prediction and evaluation module, shortening the decision response cycle of the hierarchical collaborative decision-making module, and reducing the action granularity of the action dynamic mapping module to ensure decision stability under disturbance conditions.

Images