An Autonomous Navigation and Obstacle Avoidance Method for Unmanned Aerial Vehicles in the Face of Real-Time Strong Noise Interference

By employing a spatiotemporally decoupled dual-branch network architecture and a maximum entropy reinforcement learning framework, the problem of state estimation error caused by sensor noise interference in UAV autonomous navigation is solved, achieving efficient and safe navigation in noisy environments.

CN121185306BActive Publication Date: 2026-11-17HARBIN INST OF TECH
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202511378489.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2025-09-25
Publication Date
2026-11-17
Estimated Expiration
2045-09-25

AI Technical Summary

Technical Problem

Existing DRL-based UAV autonomous navigation methods suffer from large state estimation errors when faced with sensor noise interference, making it difficult to effectively utilize historical observation information, resulting in degraded navigation performance. Furthermore, they lack explicit modeling of perception uncertainties, making them unreliable in complex environments.

Method used

A spatiotemporally decoupled dual-branch network architecture is adopted. The Transformer module processes temporal features and spatial difference matrices, and combined with the maximum entropy reinforcement learning framework, an end-to-end navigation model is constructed to directly optimize the success and safety of navigation tasks.

Benefits of technology

The navigation success rate is significantly improved under strong noise interference, maintaining high safety and efficiency. It solves the performance degradation problem caused by perception distribution offset in existing technologies and realizes reliable navigation in complex environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN121185306B_ABST
    Figure CN121185306B_ABST
Patent Text Reader

Abstract

This invention proposes an autonomous navigation and obstacle avoidance method for unmanned aerial vehicles (UAVs) under real-time strong noise interference, belonging to the field of UAV autonomous navigation. It solves the problem of large state estimation errors in existing methods under real-time strong noise interference. The method includes: acquiring the temporal state information of the UAV at multiple consecutive time points; constructing a temporal feature matrix and a spatial difference matrix based on the temporal state information at multiple consecutive time points; inputting the temporal feature matrix and spatial difference matrix into a temporal branch sub-network and a spatial branch sub-network respectively for feature extraction to obtain temporal branch features and spatial branch features; embedding a Transformer module in both the temporal branch sub-network and the spatial branch sub-network; fusing the temporal branch features and spatial branch features to obtain a joint feature representation; and outputting flight control commands for the UAV based on the joint feature representation. It is mainly applicable to fields such as urban inspection, disaster relief, and agricultural monitoring.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of autonomous navigation for unmanned aerial vehicles (UAVs), and in particular relates to an autonomous navigation obstacle avoidance method for UAVs facing real-time strong noise interference. Background Technology

[0002] In recent years, with the rapid development of the low-altitude economy, unmanned aerial vehicles (UAVs) have been increasingly widely used in many industries, such as urban inspection, disaster relief, and agricultural monitoring. Autonomous navigation systems are crucial for ensuring the safe flight of UAVs. However, many complex mission environments pose severe challenges to the perception and decision-making capabilities of UAVs. In particular, the problem of autonomous navigation under sensor noise interference has become a key bottleneck restricting the reliable application of UAVs in real-world scenarios. Airborne lidar, as a mainstream environmental perception sensor, is susceptible to various interferences: system noise caused by hardware limitations such as device jitter and beam divergence; random noise caused by environmental factors such as rain and fog scattering and reflection from dynamic obstacles; and quantization noise introduced during signal processing. According to the latest research data, under rain, snow, and fog conditions, the intensity loss of lidar ranging data can exceed 30%, leading to measurement errors and severely affecting the reliability of UAV autonomous navigation.

[0003] Reinforcement learning (DRL) has become an important method for solving the autonomous navigation problem of unmanned aerial vehicles (UAVs) due to its end-to-end training characteristics and low dependence on environmental models. Currently, most research focuses on optimizing navigation algorithms under ideal perception conditions, neglecting the sensor noise problem prevalent in practical applications. Current DRL-based navigation methods face three key problems when dealing with sensor perception uncertainty: First, sensor noise causes a shift in the distribution of perception data, resulting in inconsistencies between the state distribution during training and actual deployment. This distribution shift can significantly degrade the navigation performance of DRL algorithms. Second, most existing methods assume that sensor data is accurate and reliable, lacking explicit modeling of perception uncertainty, which makes current DRL-based navigation methods exhibit significant limitations when facing sensor perception uncertainty. Third, in terms of temporal processing, current methods struggle to effectively utilize historical observation information to suppress the impact of noise. Experimental data shows that under real-time strong noise interference, existing methods exhibit large state estimation errors. These limitations severely restrict the application effectiveness of DRL-based navigation systems in complex real-world environments. Summary of the Invention

[0004] In view of this, the present invention aims to propose an autonomous navigation and obstacle avoidance method for UAVs under real-time strong noise interference, so as to solve the problem that existing methods will have large state estimation errors under real-time strong noise interference.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: An autonomous navigation and obstacle avoidance method for unmanned aerial vehicles (UAVs) accommodating real-time strong noise interference, the method comprising: Acquire the time-series state information of the UAV at multiple consecutive moments, the time-series state information including lidar observation data affected by noise interference and the UAV's own state data; Based on the temporal state information of the consecutive multiple time points, a temporal feature matrix and a spatial difference matrix are constructed respectively; The temporal feature matrix and spatial difference matrix are respectively input into the temporal branch sub-network and spatial branch sub-network of the pre-trained navigation model for feature extraction to obtain temporal branch features and spatial branch features; wherein, both the temporal branch sub-network and the spatial branch sub-network embed a Transformer module for feature processing through an attention mechanism; By fusing the temporal branch features and spatial branch features, a joint feature representation is obtained; The flight control commands for the UAV are output based on the joint feature representation, and these commands are used for autonomous navigation in environments where there is uncertainty in sensor perception.

[0006] Furthermore, a preferred method is proposed, wherein the construction of the temporal feature matrix and the spatial difference matrix is ​​specifically as follows: The time-series feature matrix Represented as: , in, The drone status information at time t, The information sensed by the airborne lidar at time t. This provides the UAV status information at time t-1. For the airborne lidar sensing information at time t-1, This provides the UAV status information at time t-2. The information sensed by the airborne lidar at time t. Bounded workspace; The spatial difference matrix Represented as: , in, , It represents the change in state.

[0007] Furthermore, a preferred method is proposed, in which the temporal branch sub-network and the spatial branch sub-network perform feature extraction through the Transformer module, including: A linear layer is used to map the 3×37-dimensional states in the temporal feature matrix and spatial difference matrix to a 3×64-dimensional space, and sinusoidal position encoding is added. ; The encoded features are then used for spatiotemporal feature extraction via a Transformer layer: , , in, As initial features, The features are after position encoding and layer normalization. Features encoded by Transformer; Each encoder layer contains a multi-head attention and feedforward network: , , in, The feature matrix to be processed. To process feature X using a multi-head attention mechanism, The features are processed by multi-head attention mechanism and layer normalization. These are the features after processing by a feedforward neural network and layer normalization.

[0008] Furthermore, a preferred approach is proposed, wherein the method is trained based on a maximum entropy reinforcement learning framework, including: Maximize cumulative reward and policy entropy: , in, It is the entropy of the strategy. It is the weight of the entropy regularization term. For state-space information, For dynamic spatial information, As a reward, Let be the discount factor at time t, where t represents time. Calculate the Q-value estimates for each current state-action pair: , in, , The two Q values ​​are obtained by the target network based on the new state-action pair. As a discount factor, Indicates the task completion status; upon completion... It is 1 if it is true, otherwise it is 0; The weights of the Critic network, Actor network, and entropy regularization term are updated by calculating the loss. :

[0009] in, For Critic network loss, For state-based action pairs The first Q value obtained, For the target Q value, For state-based action pairs The second Q value obtained, For actor network loss, For state-based action pairs The i-th Q-value obtained, For action information, For status information, For mathematical expectation calculation, Weights of the entropy regularization term loss; Final soft update target network: , in, For the target network.

[0010] Furthermore, a preferred method is proposed, wherein the drone's own state data is: , in, Angular velocity, Linear velocity, Angular acceleration, For linear acceleration, Let be the distance between the UAV and the target point at time t. This is the angle between the current heading and the target direction.

[0011] Furthermore, a preferred method is proposed, wherein the reward function for training the navigation model is:

[0012]

[0013]

[0014]

[0015] in, For the reward function, This represents the difference in distance between the drone and the target at adjacent time points. To reach the reward, This represents the number of steps the drone has already flown. For collision rewards, Let be the distance between the UAV and the target point at time t-1. Let be the distance between the UAV and the target point at time t. To determine whether the target threshold has been reached, To measure the minimum value of a set of measurement data for lidar, The threshold for determining whether a collision with an obstacle has occurred.

[0016] Furthermore, a preferred method is proposed, wherein the fusion of the temporal branch features and spatial branch features to obtain the joint feature representation is as follows: , in, For time branching features, This is a spatial branching feature.

[0017] Furthermore, a preferred embodiment is proposed, wherein the flight control commands include the linear velocity and angular velocity of the UAV, and its action space is defined as: .

[0018] Based on the same inventive concept, the present invention also proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes an autonomous navigation and obstacle avoidance method for unmanned aerial vehicles (UAVs) oriented to real-time strong noise interference as described in any of the preceding claims.

[0019] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of an autonomous navigation and obstacle avoidance method for unmanned aerial vehicles (UAVs) accustomed to real-time strong noise interference as described in any of the above-mentioned methods.

[0020] Compared with the prior art, the beneficial effects of the present invention are: This invention proposes an autonomous navigation and obstacle avoidance method for UAVs in the face of real-time strong noise interference. In both noise-free and complex environments with varying noise levels, the navigation performance of this invention significantly outperforms existing mainstream algorithms. Specifically, in noise-free tests in high-density obstacle environments, the navigation success rate of this method reached 72%, surpassing the comparative algorithms. More importantly, under the harsh conditions simulating extreme sensor noise interference, the navigation success rate of this method remains at 73%, with a performance drop of only 18.9% from the baseline level in a noise-free environment. This attenuation rate is far lower than that of TS-SAC (63.5%) and SD-SAC (42.1%). This fully demonstrates that the method of this invention can maintain highly reliable decision-making capabilities even in the face of severe perception distortion, solving the problem of drastic performance degradation caused by perception distribution shifts in existing technologies.

[0021] This invention not only demonstrates superior success rate but also achieves an optimal balance in safety metrics. The "Minimum Average Distance (AMD)" metric in system testing shows that this invention maintained the maximum average safe distance between the drone and obstacles in all test scenarios. This indicates that the navigation strategy generated by this invention has an inherent "safety awareness," tending to choose more conservative but safer paths, thereby effectively avoiding misjudgments and collision risks caused by perceived noise. This characteristic of simultaneously improving success rate and enhancing safety is something that existing technologies have failed to achieve, highlighting the invention's outstanding comprehensive robustness in complex interference environments.

[0022] This invention ensures safety without sacrificing flight efficiency excessively. Test results show that the "Average Flight Efficiency (AFE)" of this invention consistently remains at a high level. Compared to some schemes that are slightly more efficient but significantly reduce safety, this invention achieves the best balance between safety and efficiency. Crucially, even with increasing noise levels, the growth rate of the average flight distance (AFD) and average flight steps (AFS) of this invention is far less than that of other comparative algorithms, indicating more stable decision-making and preventing the generation of excessive redundancy or oscillating path planning due to noise interference, thus ensuring the efficient completion of the navigation task.

[0023] This invention pioneers a spatiotemporally decoupled dual-branch network architecture, explicitly decoupling and fusing environmental information. The spatial branch sub-network processes the state difference matrix, capturing the relative changes and local dynamic trends of environmental features. The temporal branch sub-network processes the temporal feature matrix in parallel, representing the original continuous temporal state sequence. It preserves the absolute values ​​of the states and global evolution information, providing a holistic understanding of the long-term evolution of the environment. Finally, the features from the two branches are fused to jointly guide decision-making. This design enables the model to simultaneously utilize both static snapshots and dynamic changes in the environment, resulting in more comprehensive and accurate judgments. It fundamentally solves the shortcomings of existing technologies, which rely on a single method for processing spatiotemporal information and fail to effectively utilize relative change features.

[0024] Existing technologies for handling perceived uncertainty often employ a separate approach: "preprocessing and denoising first, then making a decision." For example, noise is first denoised using a filter, and then the result is input into a DRL model. This approach separates denoising from decision-making, causing the loss of crucial details beneficial to the decision process during denoising. This invention proposes an end-to-end joint optimization paradigm. The entire model is trained jointly with the reinforcement learning algorithm SAC, directly using the success and safety of the navigation task as the final optimization objective. The feature extraction and fusion processes within the model are not aimed at perfectly reconstructing noise-free data, but rather at directly extracting the most beneficial and interference-resistant spatiotemporal features for the final decision, achieving a denoising effect highly coordinated with the decision-making task. Attached Figure Description

[0025] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the autonomous flight mission of the UAV described in this invention, where Obstacle represents an obstacle, Goal represents a target point, and Flight path represents a flight path. Figure 2 This is a schematic diagram of the LiDAR interference sensing described in this invention, where Perceptual Error represents the sensing error; Figure 3 This is a schematic diagram of the STERL algorithm architecture described in this invention, where Agent represents an intelligent agent, Noise represents perceived noise, Environment represents the task environment, Actor Network represents the actor network, Critic Target Network represents the critic target network, Soft update represents soft update, Replay Buffer represents the experience replay pool, and Critic Network represents the critic network. Figure 4 This is a schematic diagram of the spatiotemporal dual-branch network architecture described in this invention; Figure 5 This is a schematic diagram of the reward convergence curves of different algorithms described in this invention, where Episodes represents the number of training rounds and Reward represents the reward obtained by the agent. Figure 6 These are schematic diagrams illustrating the flight trajectories of different algorithms described in this invention under four experimental scenarios; wherein, Figure 6 (a) is a schematic diagram of the flight trajectory of a UAV based on the SAC navigation model in different scenarios; Figure 6 (b) is a schematic diagram of the flight trajectory of the UAV based on the TS-SAC navigation model in different scenarios; Figure 6 (c) is a schematic diagram of the flight trajectory of the UAV based on the SD-SAC navigation model in different scenarios; Figure 6 (d) is a schematic diagram of the flight trajectory of the UAV based on the STERL navigation model in different scenarios; Figure 7 This is a schematic diagram showing the distribution of the minimum distance between the drone and the obstacle under the four test scenarios described in this invention; wherein, Figure 7 (a) is a schematic diagram of the MD distribution in the ENV-1 experiment; Figure 7 (b) is a schematic diagram of the MD distribution in the ENV-2 experiment; Figure 7 (c) is a schematic diagram of the MD distribution in the ENV-3 experiment; Figure 7(d) is a schematic diagram of the MD distribution in the ENV-4 experiment; Figure 8 These are schematic diagrams illustrating the flight trajectories of the four algorithms described in this invention under different intensities of noise interference; wherein, Figure 8 (a) is a schematic diagram of the flight trajectory of a UAV based on the SAC navigation model under different intensities of noise interference; Figure 8 (b) is a schematic diagram of the flight trajectory of the UAV based on the TS-SAC navigation model under different intensities of noise interference; Figure 8 (c) is a schematic diagram of the flight trajectory of the UAV based on the SD-SAC navigation model under different intensities of noise interference; Figure 8 (d) is a schematic diagram of the flight trajectory of the UAV based on the STERL navigation model under different intensities of noise interference; Figure 9 This is a schematic diagram illustrating the navigation performance retention rate of the four algorithms described in this invention under different intensities of noise interference; Figure 10 This is a schematic diagram showing the flight trajectories of STERL and the comparison algorithm described in this invention under different environments; wherein, Figure 10 (a) is a schematic diagram of the flight trajectory of a UAV based on the STERL navigation model under different intensities of noise interference; Figure 10 (b) is a schematic diagram of the flight trajectory of a UAV based on the Frame-5 navigation model under different intensities of noise interference; Figure 10 (c) is a schematic diagram of the flight trajectory of a UAV based on the Frame-7 navigation model under different intensities of noise interference; Figure 10 (d) is a schematic diagram of the flight trajectory of a UAV based on the w / o Transformer navigation model under different intensities of noise interference. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other, and the described embodiments are only some embodiments of the present invention, not all embodiments.

[0027] Implementation Method 1, see Figures 1 to 10 This embodiment describes an autonomous navigation and obstacle avoidance method for unmanned aerial vehicles (UAVs) facing real-time strong noise interference. The method includes: Step 1: Obtain the time-series status information of the UAV at multiple consecutive moments. The time-series status information includes lidar observation data affected by noise interference and the UAV's own status data. In practical applications, a drone's flight mission can be viewed as operating at a fixed altitude, requiring the drone to autonomously navigate from a designated starting point to a target point within a flight environment that includes obstacles. The mission scenario is as follows: Figure 1 As shown.

[0028] Its flight mission can be represented as: (1) in: Indicates a bounded workspace. This is the initial position of the drone. Represents a set of obstacles. As a navigation constraint, the target region is defined as follows: (2) In this implementation, navigation is considered successful if the drone reaches the target point within a certain number of steps. Therefore, successful drone navigation is defined as follows: (3) in: This indicates the maximum permitted flight time for the drone. Safety constraints during flight are represented as follows: (4) In DRL-based autonomous navigation missions for unmanned aerial vehicles (UAVs), airborne lidar, as a means of perceiving environmental information, is crucial for the safety and reliability of UAV navigation. For example... Figure 2 As shown, during flight, the UAV perceives the surrounding environment in real time through airborne lidar. However, due to sensor hardware limitations and environmental interference (such as rain, fog, vibration, electromagnetic noise, etc.), the lidar measurement data has significant errors. These noise interferences can lead to two typical problems: (1) false detection of obstacle positions, misjudging safe areas as obstacles; (2) missed detection of real obstacles, putting the UAV at risk of collision. Especially in dense obstacle scenarios, perception noise can further amplify the uncertainty of motion prediction and seriously affect the reliability of path planning.

[0029] To simulate perceived noise in a real-world environment, this implementation method employs a method of superimposing uniform random noise onto ideal lidar measurements. Let... t The ideal measurement value of time is The actual observed values ​​affected by noise are: (5) in, This represents the maximum noise amplitude. This modeling method can effectively simulate sensor measurement errors, and its uncertainty level can be adjusted. To control.

[0030] In the simulation experiment of this embodiment, the field of view of the UAV lidar was set to ±2π / 3, and the angular resolution was 8°. Therefore, the distance measurement data obtained at each moment had 31 dimensions. The effective measurement range of the lidar was limited to [0.2m, 3.5m]. Based on actual flight safety requirements, a collision avoidance safety threshold was set. =0.25m, the target has reached the judgment threshold. =0.4m.

[0031] Step 2: Based on the temporal state information of the multiple consecutive time points, construct the temporal feature matrix and the spatial difference matrix respectively; The problem of autonomous navigation for unmanned aerial vehicles (UAVs) can be formalized as a Markov Decision Process (MDP), defined as a quintuple: (6) in, The state space is used to describe the known state information of the UAV. In the method proposed in this embodiment, it mainly includes lidar perception information and the UAV's own state information. This represents the state space, i.e., the motion control commands for the UAV; Indicates the state transition probability; This represents the reward function, used to guide the agent's learning process; This represents the discount factor, used to balance immediate and long-term returns.

[0032] At any moment t The 31-dimensional range observation data provided by lidar is represented as follows: (7) To eliminate the influence of dimensions and promote model convergence, we normalize the observations by dividing them by the maximum measurement range of the lidar.

[0033] In addition to the perception information from the lidar, the state space also includes the UAV's own state information, which contains six key parameters.

[0034] (8) in, These represent the speed and acceleration information of the drone, respectively. Indicates the distance between the drone and the target point. This is the angle between the current heading and the target direction.

[0035] Currently, most studies construct a complete state space based on the perceived information and the UAV's own state information at the current moment, while ignoring historical information. However, state evolution in dynamic environments exhibits significant Markov properties, meaning that the current state depends on a finite number of historical states.

[0036] To fully utilize historical time-series information and capture the dynamic evolution of the environment, this implementation method constructs a temporal feature matrix and a spatial difference matrix, respectively:

[0037] in, , It represents the change in state.

[0038] The temporal feature matrix and spatial difference matrix are constructed based on spatial information from three consecutive frames of observations. The temporal feature matrix provides more information on continuous changes and effectively mitigates some of the impact of observability. Furthermore, in the presence of perceived noise, the absolute numerical reliability of the original observations decreases, while the spatial difference matrix, containing relative information on state changes, is more robust and can provide the model with information on relative environmental changes, thereby improving the autonomous decision-making performance of the UAV in complex environments.

[0039] Therefore, in this embodiment, the temporal feature matrix and spatial difference matrix are used as inputs to the DRL model in the state space to obtain the drone's actions. The action space includes flight commands such as the drone's linear velocity and angular velocity, as shown below: (11) In the formula, (m / s) and The constraint of (rad / s) ensures the stability and safety of the drone's movement.

[0040] In the Deep Reinforcement Learning (DRL) framework, the reward function, as the core mechanism of environmental feedback, directly influences the direction of policy optimization and the final performance. From the perspective of Markov decision processes, the reward function defines the optimization objective: (12) in, τ Represents the trajectory. This is the discount factor.

[0041] For UAV navigation tasks, the reward function is mainly based on several considerations: (1) Multi-objective coordination: simultaneously optimizing multiple competing objectives such as path length, safety, and energy consumption; (2) Sparse reward mitigation: solving the sparse reward problem of target arrival through dense sub-reward terms; (3) Safety constraint embedding: transforming hard safety constraints into differentiable penalty terms; (4) Training stability guarantee: maintaining gradient magnitude stability through reasonable reward scaling. Based on the above considerations, the final designed reward function is as follows: (13) (14) (15) (16) in, This represents the number of steps the drone has already flown. The distance between the drone and the nearest obstacle. The collision threshold is set to 0.25 in this embodiment.

[0042] Step 3: Input the temporal feature matrix and spatial difference matrix into the pre-trained navigation model's temporal branch sub-network and spatial branch sub-network respectively for feature extraction to obtain temporal branch features and spatial branch features; wherein, both the temporal branch sub-network and the spatial branch sub-network embed Transformer modules for feature processing through an attention mechanism; Step 4: Fuse the temporal branch features and spatial branch features to obtain a joint feature representation; Step 5: Output the flight control command of the UAV based on the joint feature representation. The flight control command is used for autonomous navigation in an environment where there is uncertainty in sensor perception.

[0043] The spatiotemporal Transformer-enhanced reinforcement learning method proposed in this implementation is based on the Soft Actor-Critic (SAC) algorithm framework. It enhances the extraction capability of temporal information and spatial variation features by introducing a Transformer module. The STERL algorithm architecture is as follows: Figure 3 As shown in the diagram. The core idea of ​​this method is to decompose the state information into two complementary feature spaces: a time dimension and a spatial dimension. These are processed separately and then fused together to improve the navigation performance of the UAV in noisy environments. Its network structure is shown in the diagram. Figure 4 As shown.

[0044] The temporal feature matrix and spatial difference matrix constructed based on continuous observation information are input into the temporal branch sub-network and spatial branch sub-network, respectively, and then processed by the Transformer module. The core of the Transformer architecture lies in its self-attention mechanism, which captures long-range dependencies by dynamically calculating the correlations between elements of the input sequence. Given an input matrix... Its self-attention operation can be expressed as: (17) Among them, the query matrix Key matrix Value matrix They were obtained through linear transformations, Learnable parameters. Scaling factor. This is used to prevent the gradient from vanishing due to an excessively large dot product result.

[0045] Multi-head attention mechanisms achieve this through parallel execution. Group attention operations enhance model capacity: (18) Among them, each attention head Calculated independently, and finally output matrix Integrate.

[0046] like Figure 4 In the network structure shown, the two branches share the same feature extraction process: first, a linear layer maps the 3×37-dimensional states in the temporal feature matrix and spatial difference matrix to a 3×64-dimensional space and adds sinusoidal position encoding. The encoded features are then used for spatiotemporal feature extraction via a Transformer layer: (19) (20) Each encoder layer contains a multi-head attention and feedforward network: (twenty one) (twenty two) Finally, the features from the temporal and spatial branches are concatenated to obtain a joint representation: (twenty three) After passing through two fully connected layers, the vector generates action distribution parameters through the Actor network. Meanwhile, the Critic network uses this feature to evaluate state-action value.

[0047] This design enables the Transformer to effectively capture the spatiotemporal dynamics of state sequences, explicitly modeling complex interactions between state elements through a self-attention mechanism. The introduction of positional encoding preserves temporal information, while the dual-branch structure simultaneously captures both absolute state values ​​and relative state changes—two key features. Compared to traditional fully connected networks, this architecture learns more generalizable feature representations in complex control tasks.

[0048] In the maximum entropy reinforcement learning framework, the optimal policy It is necessary to maximize both cumulative reward and policy entropy simultaneously.

[0049] in: It is the entropy of the strategy. This is the weight of the entropy regularization term, balancing the importance of reward and entropy. SAC calculates the Q-value estimate of the current state-action pair separately through a dual Critic network to avoid overestimation problems that may arise from a single network.

[0050] (25) in, , The target network is based on the new state-action pair Obtained. The target value includes an entropy regularization term. To explore and utilize in an automatic balance.

[0051] Then, the weights of the Critic network, Actor network, and entropy regularization term are updated by calculating the loss. :

[0052]

[0053]

[0054] Finally, perform a soft update on the target network.

[0055] (29) By combining SAC with a Transformer-enhanced spatiotemporal dual-branch network, the agent's spatiotemporal modeling ability in complex dynamic environments is improved, combining the maximum entropy exploration advantage of SAC with the long-range dependency capture ability of Transformer.

[0056] To comprehensively evaluate and verify the effectiveness of the proposed algorithm, this implementation method employs two sets of systems for testing. First, a test environment with varying obstacle densities is constructed to verify the autonomous navigation performance of the proposed method in the absence of noise interference. Then, a series of noise interference environments with increasing intensity are constructed to verify the robustness of the proposed method against noise. Finally, a set of ablation experiments is conducted to investigate the impact of historical information length and the Transformer module on the proposed method architecture. In all experiments, multi-dimensional evaluation metrics are used to comprehensively assess the algorithm's effectiveness, safety, and navigation efficiency.

[0057] An experimental environment for autonomous navigation of a quadcopter was built based on the Gazebo simulation platform and the ROS robot operating system. The experimental task required the quadcopter to autonomously navigate from its starting position to a target endpoint 7 meters away, dynamically avoiding randomly distributed cylindrical and cubic obstacles. All algorithms underwent 500 rounds of standardized training. To ensure generalization ability and prevent overfitting, obstacle positions were randomly reset at the start of each round, and the target point was randomly selected from four preset coordinates. The hyperparameter settings for STERL are shown in Table 1.

[0058] Table 1 STERL hyperparameters.

[0059] To comprehensively evaluate the navigation performance of different algorithms, this implementation method constructs multi-dimensional quantitative indicators: (1) Mission success rate: The percentage of drones that successfully reach their destination within the specified number of steps: (30) in, Indicates the total number of trials. This indicates the number of times the drone successfully navigated.

[0060] (2) Mission collision rate: The percentage of drones that successfully reach their destination within a specified number of steps: (31) in, This indicates the number of times the drone collided with obstacles. It should be noted that during the experiment, there were also instances where the drone neither reached its destination nor collided with any obstacle within the specified number of steps; this is referred to as flight loss. Since such situations were rare in the experiments presented in this paper and could be mitigated... , This was deduced, therefore it was not listed as a separate indicator.

[0061] (3) Average flight distance: The ratio of the total flight distance when the UAV successfully navigates to the number of successful navigation attempts during the navigation test. (32) in, Indicates the first i The distance of the first successful flight.

[0062] (4) Average number of flight steps: In the navigation test, the ratio of the total number of flight steps when the UAV successfully navigated to the number of successful navigation attempts: (33) in, Indicates the first i The number of steps taken in a successful flight.

[0063] (5) Average flight efficiency: In navigation tests, the ratio of the total flight distance to the total number of flight steps when the UAV successfully navigates: (34) (6) Average minimum distance: In navigation tests, the ratio of the sum of the minimum distances from the UAV to the obstacle in all successful navigation flights to the number of successful flights: (35) in, Indicates the first i The minimum distance between the drone and an obstacle during the first successful flight is used to assess the safety of drone flight.

[0064] The aforementioned quantitative evaluation indicators can provide a comprehensive and reliable assessment of the algorithm from multiple perspectives, including flight efficiency, flight performance, and flight safety.

[0065] To comprehensively evaluate the performance of the STERL algorithm, this implementation method selected three reinforcement learning algorithms, each with its own characteristics, for comparative experiments: (1) SAC: SAC combines the maximum entropy reinforcement learning framework and the Actor-Critic method. It is a high-performance DRL algorithm. In this scheme, only the observation information of the UAV at a single moment is used as the input of the model. Both the policy network and the value network adopt 3 fully connected layers.

[0066] (2) SAC based on dual-stream network (TS-SAC): TS-SAC replaces the fully connected layer with a dual-stream network on the basis of the SAC algorithm, and takes the UAV observation information at the current time and the difference between the observation information at adjacent times as input.

[0067] (3) SAC based on state decomposition network (SD-SAC): Based on the SAC algorithm, SD-SAC replaces the fully connected layer with a state decomposition network and uses the current UAV's perception-related information and target-related information as input. In addition, this implementation also embeds a Transformer module in the state decomposition network.

[0068] All algorithms were configured using the same simulation environment and maintained consistency in their hyperparameters. The convergence curves of the reward functions obtained after 500 training iterations for the four algorithms are shown below. Figure 5 As shown in the figure, the convergence trend of the reward function indicates that STERL has the best convergence performance and the fastest convergence trend. Compared to SAC, TS-SAC converges more slowly because its network structure is more complex. Both STERL and SD-SAC embed Transformer modules, enabling them to effectively capture the correlations between historical information, thus achieving rapid convergence.

[0069] To comprehensively verify and evaluate the performance of navigation models trained based on various algorithms, this implementation method constructed four experimental environments with different obstacle densities within a standard 20m×20m test space. 120, 150, 180, and 210 obstacles were randomly placed, forming a progressively challenging scenario from sparse to dense. Each algorithm performed 100 independent navigation tasks in each of the four scenarios, and their flight trajectories are as follows: Figure 6 As shown in Table 2, this implementation method recorded and statistically analyzed six core evaluation indicators.

[0070] Table 2. Experimental results of the four algorithms under different obstacle density scenarios.

[0071] As shown in Table 2, STERL achieved the highest navigation success rate in all scenarios. In low-complexity scenarios with an obstacle density of 120, the compared algorithms also demonstrated high success rates. However, as the obstacle density increased to a high-complexity scenario of 210, the navigation success rates of all four algorithms decreased to varying degrees. SAC and TS-SAC decreased by 29% and 25% respectively, while SD-SAC and STERL both decreased by 22%. STERL, thanks to its spatiotemporal feature extraction capabilities, still achieved a success rate exceeding 70% even in scenarios with the highest obstacle density.

[0072] STERL achieved near-optimal results in both average flight steps and average flight efficiency, with a very small difference from SD-SAC and significantly better performance than SAC and TS-SAC. Furthermore, STERL's average flight distance is relatively large, indicating that the UAV chooses a longer navigation path, but its average flight steps did not increase significantly. This means that the STERL model can complete navigation tasks with higher flight efficiency.

[0073] Of particular note is that during flight, STERL selects relatively longer flight paths for safety reasons, which is further verified in the average minimum distance metric. Figure 7 The figure shows the distribution of the minimum distance between the UAV and the obstacle in the successful navigation trajectory under different experimental scenarios and different algorithms.

[0074] Based on Table 2 and Figure 7 As can be seen, with the increase in the difficulty and complexity of the flight scenario, the AMD index of all algorithms declined. STERL achieved the maximum safe distance in all four flight tests. In terms of specific data distribution, STERL's median and main distribution range were higher than the comparison methods. This indicates that it ensures flight safety by increasing the flight distance to stay away from obstacles.

[0075] The above analysis and test results demonstrate that the method proposed in this invention exhibits outstanding autonomous navigation performance, achieving an optimal balance between safety and efficiency. Test results verify the spatiotemporal dual-branch network's ability to resolve complex spatial structures, proving that STERL possesses reliable safety, excellent robustness, and superior decision-making rationality in high-density obstacle environments.

[0076] Uncertainty in sensor perception poses a significant challenge to the autonomous navigation performance of UAVs. To evaluate the robustness of the proposed method under noise interference, this embodiment introduces progressive sensor noise interference based on the ENV-2 test scenario. Different intensities of noise interference are added to the lidar sensing data to simulate perception uncertainty during flight, as shown in formula (5). The added noise is uniformly distributed noise with different amplitudes. This embodiment constructs four amplitude ranges respectively... , , and The test scenarios were used to test four algorithms, with each algorithm performing 100 independent navigation tasks under various noise conditions. In all tests, the drone's flight trajectory was as follows: Figure 8 As shown in Table 3, the six core statistical indicators are presented.

[0077] Table 3. Experimental results of the four algorithms under different noise levels.

[0078] from Figure 8 The flight trajectories shown and the test results statistically presented in Table 3 demonstrate that the STERL algorithm proposed in this invention achieved the highest navigation success rate in all tests with different noise interference levels. Even in the test with the strongest noise interference, STERL still achieved a navigation success rate of 73%, far exceeding the compared algorithms. In terms of flight efficiency, STERL is only slightly lower than SD-SAC. However, comparing AFS and AFD shows that as the noise interference intensity increases, the average flight distance and average number of flight steps of SD-SAC increase by 22.0% and 17.3% respectively, while the average flight distance and average number of flight steps of STERL do not change significantly. This indicates that STERL can maintain high flight efficiency even under strong noise interference. Furthermore, STERL maintains good performance in terms of safety indicators.

[0079] Based on the above statistical results, we further analyzed the maintenance of navigation performance of different algorithms as noise interference intensity increases by calculating the ratio of the navigation success rate of different algorithms in the noise interference test to the corresponding algorithm navigation success rate in ENV-2. The results are as follows: Figure 9 As shown.

[0080] from Figure 9 As can be seen, STERL maintained minimal performance degradation in all noise interference tests. Under extreme noise conditions, STERL still maintained more than 80% of its navigation performance. Its performance degradation rate was only 18.9%, lower than SAC (22.4%) and significantly lower than TS-SAC (63.5%) and SD-SAC (42.1%).

[0081] The test results and analysis above demonstrate that the STERL algorithm proposed in this invention still exhibits optimal autonomous navigation performance even when airborne sensors are subjected to strong noise interference, and demonstrates outstanding comprehensive advantages in navigation efficiency and safety. This fully verifies the superior robustness of STERL under sensor uncertainty conditions, providing technical assurance for reliable navigation of UAVs in real and complex environments.

[0082] To deeply analyze the contributions of key components in the STERL architecture, this implementation design includes ablation experiments, focusing on the impact mechanism of historical information length and the Transformer module on the navigation performance of the STERL algorithm. Three comparative models were trained for testing: the 5-Frame and 7-Frame models used five and seven consecutive frames of spatial observation information, respectively, to construct temporal feature matrices and state difference matrices to evaluate the impact of historical information length; the w / o Transformer model removed the Transformer module from the STERL architecture and replaced it with a fully connected layer to process features, verifying the necessity of the spatiotemporal modeling component. The three algorithms were thoroughly tested in five different experimental scenarios, and the flight trajectories of all tests are shown below. Figure 10 As shown in Table 4, the statistical results obtained are presented.

[0083] Table 4. Experimental results of STERL and the comparison algorithm under different environments.

[0084] The experimental results shown in Table 4 reveal the deep-seated impact mechanisms of key design choices. Regarding the impact of historical information length, the experiments found that extending the historical window to 5 or 7 frames can slightly improve the minimum safe distance (…). AMD increased by 2.2% and 10.9% respectively, but this caused the navigation success rate under strong noise interference to drop from 73% to 59% and 63%. Furthermore, from Figure 10 The flight trajectories shown become more complex after increasing the length of historical information. The average flight distance (AFD) increased from 7.23 meters to 8.97 meters and 8.56 meters, representing increases of 24.1% and 18.4%, respectively; the average flight steps (AFS) increased from 82.64 to 105.44 and 95.25, representing increases of 27.6% and 15.3%, respectively. This indicates that the algorithm produces path detours in complex scenarios. Notably, the flight efficiency index (AFE) did not show a linear change, indicating that historical information exceeding three frames introduces redundant noise rather than effective features. These results validate that the three-frame temporal window represents the optimal balance between information timeliness and historical dependence, meeting the intelligent decision-making requirements of UAV dynamic obstacle avoidance.

[0085] Ablation experiments on the Transformer module revealed its crucial role in decision-making efficiency. Removing the module did not significantly decrease success rate across all test environments. However, it had a significant impact on flight safety and flight quality. In all test environments, average flight distance and average flight steps increased significantly, while average flight efficiency and safety indicators decreased. Specifically, average flight distance increased by a maximum of 10.8%, average flight steps increased by a maximum of 23.4%, average flight efficiency decreased by a maximum of 13.3%, and average minimum distance decreased by a maximum of 14.5%. (Reference) Figure 10 As shown in the flight trajectory, the Transformer's self-attention mechanism can effectively focus on noise-invariant features such as obstacle edges, while the simplified model, due to the lack of long-range dependency modeling, exhibits trajectory oscillations in continuous obstacle regions.

[0086] The results of the ablation experiment provide important design insights at the mechanistic level: First, the three-frame historical window is an optimal choice for dynamic obstacle avoidance decision-making, effectively utilizing environmental perception information while achieving optimal decision-making performance; second, while ensuring autonomous navigation performance, the Transformer's attention mechanism effectively improves the UAV's navigation efficiency. This ablation study not only verifies the rationality of the STERL component design but also establishes a key theoretical benchmark for the development of lightweight UAV navigation systems.

[0087] This invention proposes a novel autonomous navigation and obstacle avoidance method (STERL) for UAVs facing real-time strong noise interference, addressing the autonomous navigation problem of UAVs under the constraint of sensor perception uncertainty. This method innovatively designs a decoupled spatiotemporal dual-branch network architecture: the temporal branch network processes continuous temporal state feature matrices to capture global information about environmental evolution; the spatial branch network analyzes the state difference matrix to extract local features of environmental changes. By embedding Transformer modules in the dual branches, its self-attention mechanism is utilized to enhance long-term dependency modeling and key state screening, effectively distinguishing noise interference from real-world environmental features. Based on the SAC algorithm framework, STERL combines the advantages of maximum entropy exploration with spatiotemporal feature fusion capabilities, significantly improving decision robustness. In system experiments on the Gazebo simulation platform, this method achieved a 72% navigation success rate in a noise-free, high-density obstacle environment, outperforming comparative algorithms; under extreme noise interference, its success rate remained at 73%, with a performance degradation rate of only 18.9%, far lower than the baseline model, while also demonstrating comprehensive advantages in flight safety distance and path efficiency metrics. Ablation experiments further validated the crucial role of the 3-frame history window design and the Transformer module in improving noise immunity and decision-making efficiency. Extensive testing and evaluation analysis demonstrate that STERL possesses excellent autonomous navigation performance and flight efficiency, enabling robust and safe autonomous navigation of UAVs under perceptual uncertainty.

[0088] Implementation Method 2: A computer device according to this implementation method includes a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the autonomous navigation and obstacle avoidance method for UAVs facing real-time strong noise interference described in Implementation Method 1.

[0089] Implementation Method 3: A computer-readable storage medium as described in this implementation method stores a computer program, which, when executed by a processor, performs the steps of an autonomous navigation and obstacle avoidance method for unmanned aerial vehicles (UAVs) accommodating real-time strong noise interference as described in any one of Implementation Method 1.

[0090] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0092] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure and not to limit its protection scope. Although this disclosure has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading this disclosure, they can still make various changes, modifications or equivalent substitutions to the specific implementation of the invention, but these changes, modifications or equivalent substitutions are all within the protection scope of the published pending claims.

Claims

1. A method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles (UAVs) in the face of real-time strong noise interference, characterized in that, The method includes: Acquire the time-series state information of the UAV at multiple consecutive moments, the time-series state information including lidar observation data affected by noise interference and the UAV's own state data; Based on the temporal state information of the consecutive multiple time points, a temporal feature matrix and a spatial difference matrix are constructed respectively; The temporal feature matrix and spatial difference matrix are respectively input into the temporal branch sub-network and spatial branch sub-network of the pre-trained navigation model for feature extraction to obtain temporal branch features and spatial branch features; wherein, both the temporal branch sub-network and the spatial branch sub-network embed a Transformer module for feature processing through an attention mechanism; By fusing the temporal branch features and spatial branch features, a joint feature representation is obtained; The flight control commands for the UAV are output based on the joint feature representation, and the flight control commands are used for autonomous navigation in an environment where there is uncertainty in sensor perception. The method is trained based on a maximum entropy reinforcement learning framework, including: Maximize cumulative reward and policy entropy: , in, It is the entropy of the strategy. It is the weight of the entropy regularization term. For state-space information, For dynamic spatial information, As a reward, Let be the discount factor at time t, where t represents time. Calculate the Q-value estimates for each current state-action pair: , in, , The two Q values ​​are obtained by the target network based on the new state-action pair. As a discount factor, Indicates the task completion status; upon completion... It is 1 if it is true, otherwise it is 0; The weights of the Critic network, Actor network, and entropy regularization term are updated by calculating the loss. : in, For Critic network loss, For state-based action pairs The first Q value obtained, For the target Q value, For state-based action pairs The second Q value obtained, For actor network loss, For state-based action pairs The i-th Q-value obtained, For action information, For status information, For mathematical expectation calculation, Weights of the entropy regularization term loss; Final soft update target network: , in, For the target network.

2. The method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles (UAVs) in response to real-time strong noise interference as described in claim 1, characterized in that, The construction of the temporal feature matrix and spatial difference matrix is ​​specifically as follows: The time-series feature matrix Represented as: , in, The drone status information at time t, The information sensed by the airborne lidar at time t. This provides the UAV status information at time t-1. For the airborne lidar sensing information at time t-1, This provides the UAV status information at time t-2. For the airborne lidar sensing information at time t-2, Bounded workspace; The spatial difference matrix Represented as: , in, , , , , , It represents the change in state.

3. The method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles (UAVs) in response to real-time strong noise interference as described in claim 1, characterized in that, The temporal branch subnetwork and spatial branch subnetwork perform feature extraction through the Transformer module, including: A linear layer is used to map the 3×37-dimensional states in the temporal feature matrix and spatial difference matrix to a 3×64-dimensional space, and sinusoidal position encoding is added. ; The encoded features are then used for spatiotemporal feature extraction via a Transformer layer: , , in, As initial features, The features are after position encoding and layer normalization. Features encoded by Transformer; Each encoder layer contains a multi-head attention and feedforward network: , , in, The feature matrix to be processed. To process feature X using a multi-head attention mechanism, The features are processed by multi-head attention mechanism and layer normalization. These are the features after processing by a feedforward neural network and layer normalization.

4. The method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles (UAVs) in response to real-time strong noise interference as described in claim 1, characterized in that, The drone's own status data is as follows: , in, Angular velocity, Linear velocity, Angular acceleration, For linear acceleration, Let be the distance between the UAV and the target point at time t. The angle between the current heading and the target direction. Bounded workspace.

5. The method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles (UAVs) in response to real-time strong noise interference as described in claim 1, characterized in that, The reward function for training the navigation model is: in, For the reward function, This represents the difference in distance between the drone and the target at adjacent time points. To reach the reward, This represents the number of steps the drone has already flown. For collision rewards, Let be the distance between the UAV and the target point at time t-1. Let be the distance between the UAV and the target point at time t. To determine whether the target threshold has been reached, To measure the minimum value of a set of measurement data for lidar, The threshold for determining whether a collision with an obstacle has occurred.

6. The method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles (UAVs) in response to real-time strong noise interference as described in claim 1, characterized in that, The fusion of the temporal branch features and spatial branch features yields a joint feature representation as follows: , in, For time branching features, It is a spatial branching feature. Bounded workspace.

7. The method for autonomous navigation and obstacle avoidance of unmanned aerial vehicles (UAVs) in response to real-time strong noise interference as described in claim 1, characterized in that, The flight control commands include the linear velocity and angular velocity of the UAV, and its action space is defined as follows: 。 8. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes an autonomous navigation and obstacle avoidance method for unmanned aerial vehicles (UAVs) resistant to real-time strong noise interference as described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of an autonomous navigation and obstacle avoidance method for unmanned aerial vehicles (UAVs) accommodating real-time strong noise interference as described in any one of claims 1-7.

Citation Information

Patent Citations

  • Unmanned aerial vehicle autonomous navigation method and system based on deep reinforcement learning

    CN118550320A

  • Unmanned aerial vehicle high-speed obstacle avoidance system and method based on optoelectronic end-to-end network

    WO2025161022A1