Autonomous unmanned aerial system based on dynamic-static decoupling perception and reinforcement learning strategy

Abstract

The application provides an autonomous unmanned aerial vehicle system based on dynamic and static decoupling perception and reinforcement learning strategy, and relates to the technical field of unmanned aerial vehicle autonomous navigation.The system comprises a sensor data acquisition module, a dynamic and static decoupling perception module, a reinforcement learning strategy generation module and a flight control execution module.The sensor data acquisition module is used to acquire the RGB image, depth information and three-dimensional point cloud data of the environment and generate the ontology state information.The dynamic and static decoupling perception module is used to generate the dynamic obstacle state representation matrix and the static obstacle distance matrix, and is used to construct the static environment map.The reinforcement learning strategy generation module is used to generate the flight strategy instruction.The flight control execution module is used to receive and process the flight strategy instruction.The system exhibits better convergence in a complex mixed obstacle scene, can significantly improve the dynamic obstacle avoidance success rate and shorten the task execution time, and effectively enhances the autonomous flight robustness of the unmanned aerial vehicle in an unknown environment.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation technology for unmanned aerial vehicles (UAVs), specifically to an autonomous UAV system based on a dynamic-static decoupled perception and reinforcement learning strategy. Background Technology

[0002] With the widespread application of drones in search and rescue, security, inspection, logistics and distribution and complex low-altitude missions, autonomous navigation and real-time obstacle avoidance in complex environments have become key issues restricting the improvement of their intelligence level.

[0003] Traditional UAV navigation systems typically combine graph-based planners and optimization-based controllers. While performing well in static obstacle or pre-mapped environments, they often face challenges such as insufficient perception information, inadequate dynamic response capabilities, and unstable control in scenarios with dynamic obstacles, complex environmental structures, and densely distributed obstacles. On the one hand, obstacle avoidance schemes based on a single sensor struggle to simultaneously achieve high-precision perception of both dynamic and static obstacles. On the other hand, traditional path planning and obstacle avoidance schemes usually require explicit mapping, trajectory optimization, or high-frequency local replanning, which suffer from heavy computational burdens, insufficient adaptability, and difficulty in handling sudden obstacles in dynamic environments.

[0004] With the continuous development of artificial intelligence, reinforcement learning-based methods have become a promising paradigm for UAV navigation. These methods map environmental observations into flight control commands in an end-to-end manner, improving decision-making efficiency and adaptability to complex scenarios. However, if the observation space design is unreasonable, the reward function constraints are insufficient, or there is a lack of modeling to differentiate between static and dynamic obstacles, reinforcement learning strategies are prone to undesirable behaviors such as high collision rates, significant trajectory oscillations, or abnormally increasing flight altitude to avoid obstacles. Summary of the Invention

[0005] To address the problems in related technologies, this invention provides an autonomous unmanned aerial vehicle (UAV) system based on a dynamic-static decoupled perception and reinforcement learning strategy.

[0006] To achieve the above objectives, the technical solution adopted by the present invention includes:

[0007] Autonomous unmanned aerial vehicle (UAV) systems based on dynamic and static decoupling perception and reinforcement learning strategies include:

[0008] The sensor data acquisition module is used to acquire RGB images, depth information and 3D point cloud data of the environment, and generate body state information including the UAV's position, speed and relative relationship with the target point.

[0009] The dynamic-static decoupled perception module is used to identify and track dynamic obstacles based on the RGB image and depth information, and generate a dynamic obstacle state representation matrix; and to construct a static environment map based on the three-dimensional point cloud data, and generate a structured static obstacle distance matrix through ray projection.

[0010] The reinforcement learning policy generation module is used to fuse the dynamic obstacle state representation matrix, the static obstacle distance matrix and the ontology state information as the observation state, and to generate flight policy instructions using a policy network based on the near-end policy optimization algorithm.

[0011] The flight control execution module is used to receive and process the flight strategy instructions to control the UAV to perform obstacle avoidance flight.

[0012] Optionally, the body state input generated by the sensor data acquisition module Represented as:

[0013]

[0014] In the formula, Represents the spatial coordinates of the target point. Represents the spatial coordinates of the drone. This represents the drone's current real-time velocity vector. Denotes the Euclidean norm operator, with superscript. This represents the transpose operation of a vector.

[0015] Optionally, when identifying dynamic obstacles, the dynamic-static decoupling sensing module specifically performs the following operations:

[0016] Voxelization downsampling and density clustering are performed on the 3D point cloud transformed from the depth map to construct the initial 3D bounding box of potential dynamic targets;

[0017] By employing time-series tracking and Kalman filtering, data correlation and velocity vector analysis are performed on the detected dynamic targets. Estimate to generate a single dynamic obstacle state vector ;

[0018] The state vector is represented as:

[0019]

[0020] In the formula, Indicates the center position of the dynamic target. Indicates the location of the drone. Indicates the speed of a dynamic target. This indicates the width and height of the bounding box.

[0021] Optionally, when generating the static obstacle distance matrix, the dynamic-static decoupling sensing module specifically performs the following operations:

[0022] A local 3D voxel map is constructed in real time based on the aforementioned 3D point cloud data;

[0023] Using the bounding box set of identified dynamic targets The voxel occupancy data corresponding to the dynamic target position in the 3D voxel map is masked and filtered out to eliminate interference from dynamic objects.

[0024] Using the drone's current location as the origin, 3D ray casting is performed to reduce the dimensionality of the interference-filtered 3D spatial topology and map it into a fixed-dimensional structured distance matrix. .

[0025] Optionally, the structured distance matrix Represented as:

[0026]

[0027] In the formula, Indicates the first Under the angle of a vertical plane, including A set of distances to a horizontal ray. , This represents the number of rays in the vertical direction. This represents the number of rays in the horizontal direction.

[0028] Optionally, the policy network in the reinforcement learning policy generation module is based on a composite reward function. The composite reward function is optimized during training. Represented as:

[0029]

[0030] In the formula, Forward-driving rewards, For static obstacle avoidance rewards, For dynamic obstacle avoidance rewards, To smoothly control rewards, Rewards are highly constrained.

[0031] Optionally, the forward propulsion reward Defined as the decrease in the Euclidean distance between the UAV and the target point within adjacent control cycles:

[0032]

[0033] In the formula, and These represent the Euclidean distances between the drone and the target point at the current time and the previous time, respectively.

[0034] Optionally, the static obstacle avoidance reward Represented as:

[0035]

[0036] in, This represents the dimension of the static sensing grid in the vertical direction. This represents the dimension of the static sensing grid in the horizontal direction. Represents the logarithmic function operator. Represents a structured distance matrix containing i rows and j columns, where the static obstacle avoidance reward is... Used to constrain drones to maintain a safe distance from static obstacles;

[0037] The dynamic obstacle avoidance reward Represented as:

[0038]

[0039] In the formula, This indicates the total number of valid dynamic obstacles within the current sensing range. The dynamic obstacle avoidance reward represents the Euclidean distance between the nth dynamic obstacle and the UAV. Used to guide drones to avoid moving objects.

[0040] Optionally, the smooth control reward Represented as:

[0041]

[0042] In the formula, and These represent the current and previous time step velocity vectors of the UAV, respectively. The Euclidean norm operator represents the smooth control reward. Used to suppress sudden changes in speed commands;

[0043] The highly constrained reward Represented as:

[0044]

[0045] In the formula, Indicates the current altitude of the drone. Indicates the drone's starting altitude. Indicates the target altitude of the drone. Describes the Euclidean norm operator. The square operator represents the height-constrained reward. This is used to prevent the strategy from falling into suboptimal behavior due to abnormal climbing.

[0046] Optionally, the flight control execution module specifically performs the following operations:

[0047] Receive the normalized velocity command output by the policy network and map it into a three-dimensional velocity control command;

[0048] The three-dimensional speed control command is subjected to a safety check. If a collision risk is assessed, the command is modified by solving a linear programming problem to generate a safe speed command.

[0049] The safe speed command is sent to the flight controller for execution.

[0050] Beneficial effects:

[0051] 1. First, the system of the present invention can effectively improve the perception accuracy. Specifically, the present invention achieves real-time tracking of the motion state of dynamic targets and pure modeling of static environmental structures through heterogeneous sensor fusion and dynamic-static decoupling perception mechanism. This can effectively filter out the "artifact" interference generated by dynamic objects in static maps, and improve the integrity and accuracy of perception information in complex environments.

[0052] Second, the system of this invention can effectively enhance the real-time performance and adaptability of decision-making. Specifically, this invention is based on the fusion input of structured perception information after dynamic and static decoupling and the UAV's own state, combined with an end-to-end reinforcement learning policy network, eliminating the explicit mapping and high-frequency replanning steps of traditional methods, resulting in more efficient decision response and significantly improved adaptability to sudden dynamic obstacles and unknown complex scenarios.

[0053] Third, the system of this invention can effectively improve obstacle avoidance success rate and safety. Specifically, this invention achieves a 100% task success rate and an average number of collisions of zero in three types of obstacle avoidance scenarios: static, dynamic, and mixed. This verifies the system's robust obstacle avoidance capability and high safety in complex mixed obstacle environments.

[0054] Fourth, the system of this invention can effectively optimize navigation efficiency. Specifically, compared with traditional benchmark algorithms, this invention significantly reduces task completion time in various scenarios, and significantly improves the flight efficiency of UAVs advancing towards the target point while ensuring safety.

[0055] Fifth, the system of this invention can effectively improve flight stability. Specifically, this invention, through the smooth control reward and altitude constraint reward in the composite reward function, can effectively suppress sudden changes in speed commands and abnormal climb behavior, ensuring flight dynamic stability and trajectory smoothness.

[0056] Sixth, the system of this invention has strong simulation-to-real-world transfer capabilities. Specifically, after large-scale parallel training in a high-fidelity simulation environment, the system of this invention exhibits stable obstacle avoidance capabilities in real outdoor scenes with static trees, dynamic pedestrian interference, and mixed static and dynamic environments, demonstrating excellent "simulation-to-real-world" generalization performance.

[0057] Seventh, the system of the present invention has high reliability. Specifically, the flight control execution module of the present invention introduces a safety verification and command correction mechanism based on linear programming, providing underlying security for the end-to-end learning strategy, which can effectively reduce the potential decision-making risks of the policy network in unknown scenarios and improve the overall engineering reliability of the system.

[0058] 2. Other beneficial effects or advantages of the present invention will be described in detail in the specific embodiments. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] in:

[0061] Figure 1 This is a conceptual diagram of dynamic obstacle avoidance for an autonomous unmanned aerial vehicle (UAV) provided in an exemplary embodiment of the present invention.

[0062] Figure 2 This is a schematic diagram of the framework of an autonomous unmanned aerial vehicle system based on dynamic and static decoupling perception and reinforcement learning strategy provided by an exemplary embodiment of the present invention;

[0063] Figure 3 This is a hardware architecture diagram of an autonomous unmanned aerial vehicle provided in an exemplary embodiment of the present invention;

[0064] Figure 4 This is a schematic diagram of a dynamic and static decoupling sensing method provided in an exemplary embodiment of the present invention;

[0065] Figure 5 This is a reward curve and collision curve diagram of a reinforcement learning obstacle avoidance strategy provided in an exemplary embodiment of the present invention, wherein... Figure 5 'a' represents the return curve. Figure 5 b represents the collision curve;

[0066] Figure 6 This is a schematic diagram of static, dynamic, and hybrid scene and path point configuration in the Gazebo simulation environment provided by an exemplary embodiment of the present invention;

[0067] Figure 7 This is a schematic diagram of the obstacle avoidance trajectory of an autonomous UAV in the Gazebo simulation environment, provided by an exemplary embodiment of the present invention.

[0068] Figure 8 This is a schematic diagram of the obstacle avoidance flight trajectory of an autonomous drone in three scenarios in a real environment, provided as an exemplary embodiment of the present invention. Detailed Implementation

[0069] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0070] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0071] Furthermore, the terms "comprising" and "having," and any variations thereof, used in the description of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. It should also be noted that in embodiments of this invention, the words "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in embodiments of this invention should not be construed as preferred or advantageous over other embodiments or designs. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0072] To facilitate a clearer and more accurate understanding of the technical solution of this invention by those skilled in the art, the technical concept of this invention will be explained below.

[0073] To address the challenge of balancing perception accuracy and real-time decision-making in complex dynamic environments, this invention proposes an autonomous unmanned aerial vehicle (UAV) system based on decoupled dynamic and static perception and reinforcement learning strategies. The technical concept of this invention lies in constructing a two-layer architecture of "perception decoupling-policy fusion" to tackle the autonomous flight difficulties posed by intertwined dynamic and static obstacles in complex environments. At the perception level, heterogeneous sensors acquire the dynamic target motion state and the spatial distance to static structures, achieving decoupling and masking filtering of dynamic and static information at the data level. At the decision-making level, the decoupled structured perception information is fused with the UAV's own state and input into an end-to-end policy network based on a near-end policy optimization algorithm. A scientifically designed composite reward function guides the policy to learn efficient and safe obstacle avoidance behavior. This invention uses the separation and representation of dynamic and static obstacles as the basis for perception and reinforcement learning to directly map perception to action as the core of decision-making, effectively improving the obstacle avoidance robustness and navigation efficiency of UAVs in unknown, dynamic, and complex environments.

[0074] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0075] like Figures 1 to 8 As shown, the autonomous unmanned aerial vehicle system based on dynamic and static decoupled perception and reinforcement learning strategy of the present invention may include a sensor data acquisition module, a dynamic and static decoupled perception module, a reinforcement learning strategy generation module, and a flight control execution module.

[0076] The system comprises the following modules: a sensor data acquisition module for acquiring RGB images, depth information, and 3D point cloud data of the environment, and generating body state information including the UAV's position, speed, and relative relationship with target points; a dynamic-static decoupling perception module for identifying and tracking dynamic obstacles based on RGB images and depth information, and generating a dynamic obstacle state representation matrix; a static-environment map for constructing a static environment map based on 3D point cloud data, and generating a structured static obstacle distance matrix through ray casting; a reinforcement learning policy generation module for fusing the dynamic obstacle state representation matrix, the static obstacle distance matrix, and the body state information as the observation state, and generating flight policy commands using a policy network based on a near-end policy optimization algorithm; and a flight control execution module for receiving and processing flight policy commands to control the UAV to perform obstacle avoidance flight.

[0077] Through this technical solution, firstly, the system of the present invention can effectively improve the completeness and accuracy of perceived information in complex environments. Specifically, in the system of the present invention, the sensor data acquisition module simultaneously acquires environmental RGB images, depth information, and 3D point cloud data. Compared with a single sensor solution, the fusion of heterogeneous data sources enables the system to perceive environmental information from multiple dimensions. Simultaneously, the dynamic-static decoupling perception module, based on this, processes dynamic obstacles and static obstacles separately. On the one hand, it uses RGB images and depth information to identify and track dynamic obstacles and generate their state representation matrix; on the other hand, it uses 3D point cloud data to construct a static environment map and generates a structured static obstacle distance matrix through ray projection. In this way, through this decoupling processing method, the motion state of dynamic targets and the spatial topology of static structures can be separated and represented, avoiding the problem of mutual interference between dynamic and static information in a single perception mode, thereby effectively improving the completeness and accuracy of environmental perception.

[0078] Secondly, in the system of this invention, the reinforcement learning policy generation module can fuse the dynamic obstacle state representation matrix, the static obstacle distance matrix, and the ontological state information as the observation state, and generate flight policy commands using a policy network based on a near-end policy optimization algorithm. Since the observation state already contains decoupled dynamic and static structured perception information, the policy network does not need to implicitly learn obstacle features from the original sensor data, but instead makes decisions directly based on explicit spatial distance and motion state. This effectively reduces the learning difficulty of the policy network. Simultaneously, the end-to-end approach directly maps the perception fusion results to flight policy commands, eliminating intermediate steps such as explicit mapping, trajectory optimization, and high-frequency local replanning in traditional methods. This effectively improves the real-time performance of decision-making and the adaptability to complex dynamic scenarios.

[0079] Third, in the system of this invention, the flight control execution module receives and processes the flight strategy instructions output by the reinforcement learning strategy generation module to control the UAV to perform obstacle avoidance flight. Since the instructions generated by the policy network have integrated decoupled perception information of static and dynamic obstacles, the actions executed by the flight control execution module can simultaneously take into account the avoidance of static structures and the response to dynamic targets. In this way, the UAV can generate coordinated obstacle avoidance behavior when facing mixed obstacle scenarios, ensuring the safety and stability of autonomous flight.

[0080] In one embodiment of the present invention, the body state input generated by the sensor data acquisition module of the present invention It can be represented as:

[0081]

[0082] In the formula, Represents the spatial coordinates of the target point. Represents the spatial coordinates of the drone. This represents the drone's current real-time velocity vector. Denotes the Euclidean norm operator, with superscript. This represents the transpose operation of a vector.

[0083] In this implementation, by combining the target point direction unit vector, the target point distance, and the UAV real-time velocity vector into a structured state vector, the ontological state information can simultaneously characterize the orientation relationship of the navigation target and the kinematic state of the UAV itself. This provides a compact and complete self-state description for the subsequent policy network, which helps the reinforcement learning model to more accurately perceive the navigation progress and motion trend.

[0084] In one embodiment of the present invention, the dynamic-static decoupling sensing module of the present invention may specifically perform the following operations when identifying dynamic obstacles:

[0085] Voxelization downsampling and density clustering are performed on the 3D point cloud transformed from the depth map to construct the initial 3D bounding box of potential dynamic targets;

[0086] By employing time-series tracking and Kalman filtering, data correlation and velocity vector analysis are performed on the detected dynamic targets. Estimate to generate a single dynamic obstacle state vector ;

[0087] The state vector is represented as:

[0088]

[0089] In the formula, Indicates the center position of the dynamic target. Indicates the location of the drone. Indicates the speed of a dynamic target. Indicates the bounding box size.

[0090] In this implementation, an initial 3D bounding box is constructed through voxelization downsampling and density clustering. Combined with temporal tracking and Kalman filtering to estimate the target velocity vector, computational redundancy and noise interference in the point cloud data can be effectively suppressed, while achieving continuous and stable estimation of the dynamic target's motion state. Simultaneously, the generated state vector... By integrating relative position, velocity, and size information, it can provide policy networks with accurate dynamic obstacle kinematic representations.

[0091] In one embodiment of the present invention, the dynamic-static decoupling sensing module of the present invention may specifically perform the following operations when generating a static obstacle distance matrix:

[0092] Real-time construction of local 3D voxel maps based on 3D point cloud data;

[0093] Using the bounding box set of identified dynamic targets Masking is used to filter out the voxel occupancy data corresponding to the location of dynamic targets in the 3D voxel map in order to eliminate interference from dynamic objects.

[0094] Using the drone's current location as the origin, 3D ray casting is performed to reduce the dimensionality of the interference-filtered 3D spatial topology and map it into a fixed-dimensional structured distance matrix. .

[0095] In this implementation, a local 3D voxel map is constructed in real time, and a dynamic set of target bounding boxes is utilized. Performing masking filtering on voxel maps effectively eliminates the "artifacts" interference caused by moving objects in static maps, resulting in cleaner static environment modeling. Furthermore, 3D ray casting performed on this basis accurately reflects the true static structural spatial topology after interference filtering, improving the reliability of static perception information.

[0096] In one embodiment of the present invention, the structured distance matrix of the present invention It can be represented as:

[0097]

[0098] In the formula, Indicates the first Under the angle of a vertical plane, including A set of distances to a horizontal ray. , This represents the number of rays in the vertical direction. This represents the number of rays in the horizontal direction.

[0099] In this embodiment, by organizing the 3D ray projection results into a fixed-dimensional matrix arranged along the vertical and horizontal directions, dimensionality reduction and standardized representation of complex three-dimensional spatial topologies can be achieved. This fixed-dimensional design ensures that static perception outputs have a uniform input format regardless of the environment size, facilitating seamless integration with the observation space of reinforcement learning policy networks and improving the stability and compatibility of the system processing.

[0100] In one embodiment of the present invention, the policy network in the reinforcement learning policy generation module can be based on a composite reward function. Optimize the training and use a composite reward function. It can be represented as:

[0101]

[0102] In the formula, Forward-driving rewards, For static obstacle avoidance rewards, For dynamic obstacle avoidance rewards, To smoothly control rewards, Rewards are highly constrained.

[0103] In this implementation, by linearly combining forward propulsion rewards, static obstacle avoidance rewards, dynamic obstacle avoidance rewards, smooth control rewards, and altitude constraint rewards, a comprehensive balance can be achieved across multiple optimization objectives, including navigation efficiency, static safety, dynamic obstacle avoidance, flight stability, and altitude rationality. This comprehensive balance allows the policy network to simultaneously consider task completion speed and flight safety during the learning process, avoiding policy bias issues caused by a single reward function.

[0104] In one embodiment of the present invention, the forward propulsion reward of the present invention It can be defined as the decrease in the Euclidean distance between the UAV and the target point within adjacent control cycles:

[0105]

[0106] In the formula, and These represent the Euclidean distances between the drone and the target point at the current time and the previous time, respectively.

[0107] In this implementation, the reduction in the Euclidean distance between the UAV and the target point within adjacent control cycles is defined as a reward, directly encouraging the UAV to continuously shorten the spatial distance to the target point. This effectively suppresses inefficient behaviors such as stagnation or detours by the UAV, and drives the policy network to learn efficient flight strategies for advancing towards the target.

[0108] In one embodiment of the present invention, the static obstacle avoidance reward of the present invention It can be represented as:

[0109]

[0110] in, This represents the dimension of the static sensing grid in the vertical direction. This represents the dimension of the static sensing grid in the horizontal direction. Represents the logarithmic function operator. Represents a structured distance matrix containing i rows and j columns, with static obstacle avoidance reward. Used to constrain drones to maintain a safe distance from static obstacles;

[0111] Dynamic obstacle avoidance rewards It can be represented as:

[0112]

[0113] In the formula, This indicates the total number of valid dynamic obstacles within the current sensing range. This represents the Euclidean distance between the nth dynamic obstacle and the drone, and the dynamic obstacle avoidance reward. Used to guide drones to avoid moving objects.

[0114] In this implementation, the static obstacle avoidance reward, calculated by the logarithmic mean of the depth values ​​of each element in the structured distance matrix, enhances the penalty sensitivity to nearby static obstacles, forcing the UAV to maintain a safe distance from static structures. Simultaneously, the dynamic obstacle avoidance reward, calculated by the logarithmic mean of the real-time distances to each dynamic obstacle, guides the UAV to actively avoid moving objects. Furthermore, the nonlinear characteristics of the logarithmic function cause the reward signal to change drastically as obstacles approach, effectively improving the timeliness and sensitivity of the obstacle avoidance response.

[0115] In one embodiment of the present invention, the smooth control reward of the present invention It can be represented as:

[0116]

[0117] In the formula, and These represent the current and previous time step velocity vectors of the UAV, respectively. Represents the Euclidean norm operator, smoothing control of rewards. Used to suppress sudden changes in speed commands;

[0118] Highly constrained rewards It can be represented as:

[0119]

[0120] In the formula, Indicates the current altitude of the drone. Indicates the drone's starting altitude. Indicates the target altitude of the drone. Describes the Euclidean norm operator. Represents the squaring operator, with height-constrained rewards. This is used to prevent the strategy from falling into suboptimal behavior due to abnormal climbing.

[0121] In this implementation, the smooth control reward, by penalizing the L2 norm of the velocity vector difference between adjacent time points, can effectively suppress drastic changes in velocity commands, thereby reducing oscillations in the underlying actuators and ensuring flight dynamics stability and passenger comfort. The altitude constraint reward, by penalizing the squared error of the current altitude deviating from the starting or target altitude, can effectively prevent the policy network from engaging in suboptimal behavior of continuously climbing to avoid obstacles, guiding the UAV to maintain a reasonable flight altitude range.

[0122] In one embodiment of the present invention, the flight control execution module of the present invention may specifically perform the following operations:

[0123] Receive the normalized velocity command output by the policy network and map it into a three-dimensional velocity control command;

[0124] Safety checks are performed on the three-dimensional speed control commands. If a collision risk is assessed, the commands are corrected by solving a linear programming problem to generate safe speed commands.

[0125] The safe speed command is sent to the flight controller for execution.

[0126] In this implementation, by mapping the normalized velocity command output by the policy network to a three-dimensional velocity control command, and then performing safety checks and linear programming-based command corrections, proactive intervention and the generation of safe velocity commands can be achieved when the policy network outputs a collision risk. This provides a fundamental safety mechanism for the end-to-end reinforcement learning policy, effectively mitigating potential decision-making errors in unknown scenarios and thus improving the system's reliability and safety in real flight.

[0127] The technical solution of the present invention will be further described below with reference to an exemplary embodiment.

[0128] It should be noted that this implementation method is mainly verified by combining simulation experiments with real-world outdoor flight experiments. All steps and conclusions have been verified to be correct on the Linux operating system platform through the Isaac Sim environment.

[0129] This implementation study designs an autonomous unmanned aerial vehicle (UAV) system based on dynamic-static decoupled perception and a reinforcement learning flight strategy. Within a heterogeneous sensing framework, RGB-D images and 3D point clouds are acquired using a D455 depth camera and a Mid360 LiDAR, respectively. A dynamic-static decoupled perception mechanism is then used to perform lightweight processing on the heterogeneous data, generating dynamic and static obstacle representation matrices to achieve the collaborative utilization of spatial information of dynamic and static targets in complex environments. For flight decision-making and control execution, a reinforcement learning architecture including independent feature extraction branches and the PPO algorithm is designed. By fusing the decoupled obstacle representation matrix with the UAV's internal state and inputting it into the policy network, the model outputs a real-time desired velocity command, which is then converted into a final drive signal by the PX4 flight controller. This effectively improves the UAV's autonomous navigation and obstacle avoidance performance in dynamic and complex scenarios.

[0130] To facilitate understanding of the technical content of this invention by those skilled in the art, the invention will be further described in detail below with reference to the accompanying drawings. Subsequently, the system performance will be comprehensively evaluated through simulation experiments and real-world flight scenarios. The specific implementation process of this embodiment can be as follows:

[0131] I. Sensor Data Acquisition Module.

[0132] In the data acquisition and state estimation phase, this invention constructs a heterogeneous sensor data stream to support subsequent decoupled sensing. Specifically, an airborne D455 depth camera is used to acquire real-time RGB image and depth information of the environment; this data stream primarily drives subsequent dynamic obstacle perception tasks. Simultaneously, a Mid360 LiDAR combined with the Fast-LIO2 algorithm is used for real-time mapping and state estimation, and the output high-precision 3D point cloud data is directly used for static obstacle structure perception. During this process, the system uses the direction, distance, and relative velocity vector of the target point to constitute the UAV's body state input. .

[0133]

[0134] in, Represents the spatial coordinates of the target point. Represents the spatial coordinates of the drone. This represents the drone's current real-time velocity vector. Denotes the Euclidean norm operator, with superscript. This represents the transpose operation of a vector.

[0135] II. Dynamic and static decoupling sensing module.

[0136] This module performs separation and characterization of dynamic and static obstacles based on the data collected by the sensor data acquisition module, in order to eliminate perception noise and extract structured features.

[0137] In terms of dynamic obstacle perception, the RGB images and depth information acquired by the D455 depth camera are first spatiotemporally aligned, and the depth map is then converted into a 3D point cloud. Voxelization downsampling is performed to suppress computational redundancy, and the DBSCAN density clustering algorithm is combined to adaptively segment the potential obstacle point set based on spatial distribution differences. This allows for the rapid construction of the initial 3D bounding box. The algorithm introduces a time-series tracking mechanism to correlate detection results at different times, and combines this with a Kalman filter model to estimate the target velocity vector. This allows for the accurate identification and separation of real dynamic obstacles, outputting their continuous state representation matrix. With precise 3D bounding box features. Among them, a single dynamic obstacle. state vector It can be represented as a combination of relative position, velocity, and size:

[0138]

[0139] in, and These represent the center position and velocity of a dynamic target, respectively. Location of the drone. This indicates the width and height of the bounding box (i.e., the size of the bounding box).

[0140] In terms of static obstacle perception and decoupled representation, a decoupled representation architecture based on 3D occupancy voxels and ray projection is designed, which integrates data from a Mid360 LiDAR and an Inertial Measurement Unit (IMU) using the Fast-LIO2 algorithm. A local 3D voxel map is constructed in real time by recursively updating the log-odds occupancy probability of each voxel. A data-level decoupling strategy is introduced, utilizing the dynamic target 3D bounding box set obtained from the pre-module. The occupied data within the map is masked and filtered out. That is, when the voxel space coordinates satisfy... season This is to eliminate the "artifacts" caused by moving objects.

[0141] Simultaneously, based on the constructed local 3D occupancy voxels, 3D ray casting is performed with the current position of the UAV as the origin, reducing the complex three-dimensional spatial topology to a fixed-dimensional structured distance matrix. :

[0142]

[0143] in, and These represent the number of rays in the horizontal and vertical directions, respectively. Indicates the first Under the angle of a vertical plane, including A set of distances to a horizontal ray. .

[0144] Finally, the spatial distance matrix provided by static perception will be used. With the kinematic state matrix provided by dynamic perception and the drone's own navigation status By performing dimensional concatenation, a state matrix for dynamic and static decoupling perception is provided for the reinforcement learning decision model. :

[0145]

[0146] In the formula, This represents the feature concatenation operation, which is used to connect state matrices of different dimensions along a specific dimension to form a unified state input matrix.

[0147] III. Reinforcement Learning Strategy Generation Module.

[0148] When constructing the reinforcement learning decision model, this module directly receives the output of the dynamic and static decoupled perception module, defining the observation space O as the fusion state matrix. The core lies in the scientific setting of the reward function. This embodiment constructs a composite reward function composed of a linear combination of five sub-reward components. To balance mission efficiency and flight safety. Composite reward function. The mathematical expression is:

[0149]

[0150] In the formula, Forward-driving rewards, For static obstacle avoidance rewards, For dynamic obstacle avoidance rewards, To smoothly control rewards, Rewards are highly constrained.

[0151] The specific representation and calculation logic of each sub-reward component are as follows:

[0152] Forward Advancement Rewards This is used to encourage drones to continuously and efficiently approach the target point, avoiding stagnation. It is defined as the decrease in the Euclidean distance between the drone and the target point within adjacent control cycles:

[0153]

[0154] In the formula, and These represent the Euclidean distances between the drone and the target point at the current time and the previous time, respectively.

[0155] Static obstacle avoidance rewards To constrain drones to maintain a safe distance from static obstacles, a penalty mechanism is constructed based on the static observation matrix generated by ray casting, using the logarithmic mean of its depth values.

[0156]

[0157] In the formula, This represents the dimension of the static sensing grid in the vertical direction. This represents the dimension of the static sensing grid in the horizontal direction. Represents the logarithmic function operator. This represents a structured distance matrix containing i rows and j columns.

[0158] Dynamic obstacle avoidance rewards Used to guide drones to actively avoid moving obstacles, calculated based on the real-time distance between the drone and the center of gravity of each effective dynamic obstacle within the perception range:

[0159]

[0160] In the formula, This indicates the total number of valid dynamic obstacles within the current sensing range. Indicates the first The Euclidean distance between a dynamic obstacle material center and a drone.

[0161] Smooth control reward Used to suppress oscillations in the underlying actuators caused by sudden changes in speed commands, and to ensure the dynamic stability of actual flight, this item penalizes drastic changes in control speed between adjacent moments:

[0162]

[0163] In the formula, and These represent the current and previous time step velocity vectors of the UAV, respectively. This represents the Euclidean norm operator.

[0164] Highly constrained rewards To prevent the policy network from falling into suboptimal behavior in complex spaces, which involves "avoiding all obstacles by climbing," a negative reward constraint will be imposed to return it to a normal altitude.

[0165]

[0166] In the formula, Indicates the current altitude of the drone. Indicates the drone's starting altitude. Indicates the target altitude of the drone. Describes the Euclidean norm operator. This represents the squaring operator.

[0167] Based on the above architecture, the system uses the PPO algorithm to drive network parameter updates, fuses the extracted obstacle features with the UAV's internal state, and finally outputs the optimal flight strategy.

[0168] IV. Flight Control Execution Module.

[0169] After acquiring the environmental perception results, the trained reinforcement learning policy network first outputs the Beta distribution parameters to infer the normalization rate. And map it into a three-dimensional velocity control command in the target coordinate system. :

[0170]

[0171] in, The maximum speed limit is set.

[0172] Before execution, the initial command must undergo a safety check. If the check assesses that the action poses a future collision risk, the system will construct and solve a linear programming problem to calculate the minimum speed change required to escape the danger zone. This will force the initial command to be projected into an absolutely safe action space, generating the optimal safe speed command. .

[0173] Ultimately, the revised safe speed command is sent to the PX4 flight controller and translated into physical execution signals for safe flight in complex dynamic environments.

[0174] In this embodiment, it is understood that the present invention, through spatiotemporal decoupling perception of lidar and visual sensors, 3D point cloud voxelization, and dynamic target masking technology, accurately filters out dynamic interference, significantly improving the accuracy of static environment modeling and the real-time performance of dynamic target tracking; the end-to-end policy network based on the PPO algorithm achieves a 100% task success rate in static, dynamic, and mixed scenarios, while significantly shortening task completion time and approaching zero collision rate; through simulation and physical verification, the system demonstrates excellent "simulation to reality" transfer capability and strong adaptability to unknown obstacles, fully verifying its robustness and generalization performance in engineering applications, and achieving improvements in perception accuracy, obstacle avoidance efficiency, and system reliability compared to existing technologies.

[0175] To verify the effectiveness and robustness of the system in complex environments, this invention further constructs a high-fidelity simulation environment and real-world scenario testing process, with the specific implementation steps as follows:

[0176] In the Isaac Sim platform, this invention deployed 1024 UAV agents for parallel sampling training to significantly improve data collection efficiency. The deployment step size for a single policy update was set to 1000 steps. The training environment was constructed as a hybrid space containing 350 static obstacles and 60 dynamic obstacles moving at a constant speed along random directions, thereby simulating high-density complex scenes in the real world.

[0177] With this configuration, the system showed a rapid increase in cumulative reward in the early stages of training, and the collision rate showed a continuous downward trend as the number of training steps increased, which verified the effective convergence of the policy network.

[0178] After completing the simulation training, the invention conducted a comprehensive test on the system in a real outdoor environment, verifying the flight performance of the UAV in static tree environment, dynamic pedestrian interference environment, and mixed static and dynamic complex environment.

[0179] Experimental results show that in all three scenarios, the UAV can effectively maintain a safe distance from obstacles and successfully reach the target location. Particularly in scenarios with dynamic pedestrian interference and mixed obstacles, the UAV can perform rapid braking and timely avoidance maneuvers, demonstrating strong dynamic response and environmental adaptability, thus proving the superiority of the proposed method in the "simulation to reality" transfer.

[0180] To verify the effectiveness of the method proposed in this invention, a comprehensive evaluation of each module was conducted, and systematic experiments were carried out by constructing a simulation and physical testing platform.

[0181] The experimental results are explained in detail below with reference to the accompanying figures:

[0182] Figures 1 to 4 It is mainly used to explain the overall architecture, hardware composition, and core algorithm principles of the system of this invention; among which, Figure 1 The concept diagram of dynamic obstacle avoidance of autonomous drones is shown, which intuitively demonstrates how drones can generate action commands for the next moment based on the current observation state through a policy network, thereby achieving safe avoidance of dynamic and static obstacles in continuous time and space. Figure 2 The system framework diagram shows in detail the complete closed-loop control flow of the present invention, which consists of sensing input, feature extraction, reinforcement learning policy network and control output. Figure 3 This is a hardware architecture diagram of an autonomous drone, which specifically shows the physical connections and data interaction relationships between the D455 camera, Mid360 LiDAR, Jetson NX onboard computer, and the drone platform. Figure 4This diagram illustrates a dynamic-static decoupling sensing method. The system demonstrates the entire process of separating and decoupling the perception and representation of dynamic and static obstacles: dynamic obstacles are tracked in real-time using an RGB-D camera and DBSCAN clustering combined with Kalman filtering; static obstacles are transformed into a structured distance matrix through lidar point cloud voxelization, dynamic target masking, and 3D ray projection. By employing spatiotemporal separation and data-level decoupling strategies, dynamic interference is effectively eliminated, enabling accurate modeling of the static environment.

[0183] Figures 5 to 8 Tables 1 to 3 below are mainly used to verify the training convergence, navigation efficiency, and obstacle avoidance robustness of the present invention; among them, Figure 5 To reinforce the reward curve and collision curve of the obstacle avoidance strategy, we show that as the number of training steps increases, the cumulative reward of the system gradually increases and tends to converge. At the same time, we verify the negative correlation trend of the collision rate of the system continuously decreasing with the optimization of the strategy, and verify the effectiveness of the PPO algorithm under this architecture. Figure 6 and Figure 7 The path point configuration maps and flight obstacle avoidance trajectories in static, dynamic and mixed scenarios in the Gazebo simulation environment are shown respectively, verifying the algorithm's path planning capability in all scenarios; Figure 8 The actual flight trajectories of the UAV in real-world environments, including static trees, dynamic pedestrian interference, and mixed static-dynamic scenarios, demonstrate the excellent "simulation-to-reality" transfer capability of this invention. Tables 1 and 2 show the task success rates and completion times of this invention and traditional benchmark algorithms in the three obstacle avoidance scenarios, respectively. Experimental data shows that this invention achieves a high task success rate in all scenarios, and its average completion time is significantly shorter than the comparison algorithms, highlighting the navigation efficiency and anti-interference robustness of this method in complex environments. Table 3 shows the average number of collisions for different methods in the three obstacle avoidance scenarios. This invention maintained a zero-collision record in the test, highlighting the robustness and safety of the system in mixed obstacle environments.

[0184] Table 1. Comparison of task success rates for different methods in three obstacle avoidance scenarios.

[0185]

[0186] Table 2 Comparison of Completion Time for Different Methods in Three Obstacle Avoidance Scenarios (Unit: seconds)

[0187]

[0188] Table 3. Comparison of average collision counts for different methods in three obstacle avoidance scenarios.

[0189]

[0190] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An autonomous unmanned aerial vehicle (UAV) system based on dynamic-static decoupling perception and reinforcement learning strategies, characterized in that, include: The sensor data acquisition module is used to acquire RGB images, depth information and 3D point cloud data of the environment, and generate body state information including the UAV's position, speed and relative relationship with the target point. The dynamic and static decoupled perception module is used to identify and track dynamic obstacles based on the RGB image and depth information, and generate a dynamic obstacle state representation matrix; Furthermore, it is used to construct a static environment map based on the three-dimensional point cloud data, and to generate a structured static obstacle distance matrix through ray projection; The reinforcement learning policy generation module is used to fuse the dynamic obstacle state representation matrix, the static obstacle distance matrix and the ontology state information as the observation state, and to generate flight policy instructions using a policy network based on the near-end policy optimization algorithm. The flight control execution module is used to receive and process the flight strategy instructions to control the UAV to perform obstacle avoidance flight.

2. The autonomous unmanned aerial vehicle system based on dynamic and static decoupling perception and reinforcement learning strategy according to claim 1, characterized in that, The body status input generated by the sensor data acquisition module Represented as: In the formula, Represents the spatial coordinates of the target point. Represents the spatial coordinates of the drone. This represents the drone's current real-time velocity vector. Denotes the Euclidean norm operator, with superscript. This represents the transpose operation of a vector.

3. The autonomous unmanned aerial vehicle system based on dynamic and static decoupling perception and reinforcement learning strategy according to claim 2, characterized in that, When identifying dynamic obstacles, the dynamic-static decoupling sensing module performs the following operations: Voxelization downsampling and density clustering are performed on the 3D point cloud transformed from the depth map to construct the initial 3D bounding box of potential dynamic targets; By employing time-series tracking and Kalman filtering, data correlation and velocity vector analysis are performed on the detected dynamic targets. Estimate to generate a single dynamic obstacle state vector ; The state vector is represented as: In the formula, Indicates the center position of the dynamic target. Indicates the location of the drone. Indicates the speed of a dynamic target. This indicates the width and height of the bounding box.

4. The autonomous unmanned aerial vehicle system based on dynamic and static decoupling perception and reinforcement learning strategy according to claim 3, characterized in that, When generating the static obstacle distance matrix, the dynamic-static decoupling sensing module performs the following operations: A local 3D voxel map is constructed in real time based on the aforementioned 3D point cloud data; Using the bounding box set of identified dynamic targets The voxel occupancy data corresponding to the dynamic target position in the 3D voxel map is masked and filtered out to eliminate interference from dynamic objects. Using the drone's current location as the origin, 3D ray casting is performed to reduce the dimensionality of the interference-filtered 3D spatial topology and map it into a fixed-dimensional structured distance matrix. .

5. The autonomous unmanned aerial vehicle system based on dynamic and static decoupling perception and reinforcement learning strategy according to claim 4, characterized in that, The structured distance matrix Represented as: In the formula, Indicates the first Under the angle of a vertical plane, including A set of distances to a horizontal ray. , This represents the number of rays in the vertical direction. This represents the number of rays in the horizontal direction.

6. The autonomous unmanned aerial vehicle system based on dynamic and static decoupling perception and reinforcement learning strategy according to claim 1, characterized in that, The policy network in the reinforcement learning policy generation module is based on a composite reward function. The composite reward function is optimized during training. Represented as: In the formula, Forward-driving rewards, For static obstacle avoidance rewards, For dynamic obstacle avoidance rewards, To smooth out reward control, Rewards are highly constrained.

7. The autonomous unmanned aerial vehicle system based on dynamic and static decoupling perception and reinforcement learning strategy according to claim 6, characterized in that, The forward progress reward Defined as the decrease in the Euclidean distance between the UAV and the target point within adjacent control cycles: In the formula, and These represent the Euclidean distances between the drone and the target point at the current time and the previous time, respectively.

8. The autonomous unmanned aerial vehicle system based on dynamic and static decoupling perception and reinforcement learning strategy according to claim 6, characterized in that, The static obstacle avoidance reward Represented as: ;in, This represents the dimension of the static sensing grid in the vertical direction. This represents the dimension of the static sensing grid in the horizontal direction. Represents the logarithmic function operator. Represents a structured distance matrix containing i rows and j columns, where the static obstacle avoidance reward is... Used to constrain drones to maintain a safe distance from static obstacles; The dynamic obstacle avoidance reward Represented as: In the formula, This indicates the total number of valid dynamic obstacles within the current sensing range. The dynamic obstacle avoidance reward represents the Euclidean distance between the nth dynamic obstacle and the UAV. Used to guide drones to avoid moving objects.

9. The autonomous unmanned aerial vehicle system based on dynamic and static decoupling perception and reinforcement learning strategy according to claim 6, characterized in that, The smooth control reward Represented as: In the formula, and These represent the current and previous time step velocity vectors of the UAV, respectively. The Euclidean norm operator represents the smooth control reward. Used to suppress sudden changes in speed commands; The highly constrained reward Represented as: In the formula, Indicates the current altitude of the drone. Indicates the drone's starting altitude. Indicates the target altitude of the drone. Describes the Euclidean norm operator. The square operator represents the height-constrained reward. This is used to prevent the strategy from falling into suboptimal behavior due to abnormal climbing.

10. The autonomous unmanned aerial vehicle system based on dynamic and static decoupling perception and reinforcement learning strategy according to claim 1, characterized in that, The flight control execution module performs the following operations: Receive the normalized velocity command output by the policy network and map it into a three-dimensional velocity control command; The three-dimensional speed control command is subjected to a safety check. If a collision risk is assessed, the command is modified by solving a linear programming problem to generate a safe speed command. The safe speed command is sent to the flight controller for execution.

Images