Real-time obstacle avoidance method and device for unmanned aerial vehicle and medium

Abstract

The invention provides a real-time obstacle avoidance method and device for an unmanned aerial vehicle and a medium. The method comprises the following steps: acquiring a synchronous binocular image of the unmanned aerial vehicle; obtaining a corresponding disparity map according to the synchronous binocular image, and carrying out optimization processing to obtain an optimized disparity map; acquiring a flight state vector of the unmanned aerial vehicle; inputting the optimized disparity map and the flight state vector into an end-to-end decision network to obtain a control instruction; and performing security constraint processing on the control instruction, and sending the control instruction after the security constraint processing to the flight control, so that the flight control executes closed-loop control based on the control instruction after the security constraint processing, thereby realizing real-time obstacle avoidance of the unmanned aerial vehicle. According to the method, the device and the medium, the problem that the autonomous obstacle avoidance capability and the reliability of the unmanned aerial vehicle are not high in a resource-limited scene according to an existing unmanned aerial vehicle real-time obstacle avoidance method can be solved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) perception and control technology, and in particular to a method, device and medium for real-time obstacle avoidance by UAVs. Background Technology

[0002] In the field of autonomous obstacle avoidance for UAVs, the mainstream approach often adopts a cascaded architecture of "mapping - localization - path planning - control": the front end uses binocular / monocular vision or visual inertial odometry to complete feature extraction, matching and pose estimation, and combines dense / semi-dense depth calculation or stereo parallax to generate local / global maps; the back end maintains consistency through graph optimization or sliding window optimization; on this basis, global / local planning is run to obtain the desired trajectory, and finally the trajectory tracker outputs control commands.

[0003] However, existing methods for real-time obstacle avoidance by drones are not very reliable or capable of autonomous obstacle avoidance in resource-constrained scenarios. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the above-mentioned shortcomings of the prior art by providing a method, device and medium for real-time obstacle avoidance by unmanned aerial vehicles (UAVs) to solve the problem that the existing real-time obstacle avoidance methods for UAVs have low autonomous obstacle avoidance capability and reliability in resource-constrained scenarios.

[0005] In a first aspect, the present invention provides a real-time obstacle avoidance method for unmanned aerial vehicles (UAVs). The method is deployed on a preset platform via a PyTorch→ONNX→RKNN model conversion process, employing a multi-threaded pipeline and a zero-copy buffer, and includes: Acquire synchronized stereo images from the drone; Based on the synchronized stereo image, a disparity map corresponding to the synchronized stereo image is obtained, and the disparity map is optimized to obtain the optimized disparity map. Obtain the flight state vector of the UAV, wherein the flight state vector includes at least one of the following: velocity, attitude, and target distance; The optimized disparity map and flight state vector are input into an end-to-end decision network to obtain control commands, wherein the control commands include at least one of the following: forward / lateral / vertical velocity, yaw rate, and attitude setpoint. The control commands are subjected to safety constraints, and the control commands with safety constraints are sent to the flight controller so that the flight controller can perform closed-loop control based on the control commands with safety constraints, so as to realize the UAV obstacle avoidance in real time.

[0006] Furthermore, acquiring the synchronized binocular images of the UAV specifically includes: The synchronized stereo images are acquired through a hardware triggering mechanism or timestamp synchronization.

[0007] Further, obtaining the disparity map corresponding to the synchronized stereo image based on the synchronized stereo image specifically includes: The disparity map is obtained by performing depth estimation on the synchronous stereo image using a stereo depth estimation algorithm or a lightweight convolutional network.

[0008] Furthermore, the stereo depth estimation algorithm employs StereoBM combined with WLS filtering to perform depth estimation on the synchronized stereo image, thereby obtaining the disparity map. Specifically, this includes: Histogram equalization or local mean removal is performed on the left and right images of the synchronized stereo images; A window of a preset size is selected centered on each pixel in the processed left image, and the absolute difference and SAD between each window and the corresponding position window in the processed right image are calculated within a preset disparity search interval to obtain the matching cost for each pixel corresponding to different disparities. Based on the matching cost of different disparities corresponding to each pixel, the winner-take-all (WTA) strategy is executed on each pixel to obtain the initial disparity of each pixel; Each of the initial disparities is subjected to subpixel refinement, and each of the initial disparities after subpixel refinement is subjected to left-right consistency check (LRC). The weighted least squares (WLS) algorithm is used to perform guided filtering on each of the initial disparities after inspection to obtain the disparity map.

[0009] Further, the optimization processing of the disparity map to obtain the optimized disparity map specifically includes: The disparity map is cropped and preprocessed, and then downsampled to a preset resolution size to obtain the optimized disparity map.

[0010] Furthermore, the end-to-end decision network is a network structure combining lightweight convolutional and recurrent units. The step of inputting the optimized disparity map and flight state vector into the end-to-end decision network to obtain control commands specifically includes: The optimized disparity map and flight state vector are input into the end-to-end decision network. The end-to-end decision network performs lightweight convolutional feature extraction and fusion of recurrent unit historical information on the optimized disparity map and flight state vector, and outputs the control command.

[0011] Furthermore, the end-to-end decision network is pre-trained using a simulation environment with domain randomization and fine-tuned based on real flight data. The network operators of the end-to-end decision network are the basic operators of RKNN.

[0012] Secondly, the present invention provides a real-time obstacle avoidance device for unmanned aerial vehicles (UAVs). The device is set up on a preset platform via a PyTorch→ONNX→RKNN model conversion process, employing a multi-threaded pipeline and a zero-copy buffer, and includes: The first acquisition module is used to acquire synchronized stereo images of the UAV; A processing module is connected to the first acquisition module and is used to obtain a disparity map corresponding to the synchronized stereo image based on the synchronized stereo image, and to optimize the disparity map to obtain the optimized disparity map. The second acquisition module, connected to the acquisition processing module, is used to acquire the flight state vector of the UAV, wherein the flight state vector includes at least one of the following: speed, attitude, and target distance; The input module, connected to the second acquisition module, is used to input the optimized disparity map and flight state vector into the end-to-end decision network to obtain control commands, wherein the control commands include at least one of the following: forward / lateral / vertical velocity, yaw rate, and attitude setpoint; The processing and sending module, connected to the input receiving module, is used to perform safety constraint processing on the control commands and send the safety constraint-processed control commands to the flight controller, so that the flight controller can perform closed-loop control based on the safety constraint-processed control commands to achieve real-time obstacle avoidance by the UAV.

[0013] Thirdly, the present invention provides a real-time obstacle avoidance device for unmanned aerial vehicles (UAVs), including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to implement the real-time obstacle avoidance method for UAVs described in the first aspect above.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the real-time obstacle avoidance method for unmanned aerial vehicles described in the first aspect.

[0015] The present invention provides a method, apparatus, and medium for real-time obstacle avoidance by unmanned aerial vehicles (UAVs). First, a synchronized binocular image of the UAV is acquired. Then, a disparity map corresponding to the synchronized binocular image is obtained, and the disparity map is optimized to obtain an optimized disparity map. Next, a flight state vector of the UAV is acquired, wherein the flight state vector includes at least one of the following: velocity, attitude, and target distance. The optimized disparity map and flight state vector are then input into an end-to-end decision network to obtain control commands, wherein the control commands include at least one of the following: forward / lateral / vertical velocity, yaw rate, and attitude setpoint. Finally, the control commands are subjected to safety constraint processing, and the safety-constrained control commands are sent to the flight controller, so that the flight controller performs closed-loop control based on the safety-constrained control commands to achieve real-time obstacle avoidance by the UAV. This invention uses an optimized disparity map as the core input, integrates the UAV flight state vector, and directly outputs speed / attitude / yaw rate control commands through an end-to-end decision network. Compared with existing solutions, this invention avoids the high overhead of map building and global planning, significantly shortens the perception-to-control link and latency, and ensures end-to-end latency on resource-constrained platforms through RKNN edge deployment and multi-threaded pipeline. This effectively improves the autonomous obstacle avoidance capability and reliability of UAVs in resource-constrained scenarios, solving the problem of low autonomous obstacle avoidance capability and reliability of existing real-time obstacle avoidance methods for UAVs in resource-constrained scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present drawings or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present drawings. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0017] Figure 1 This is a flowchart of a real-time obstacle avoidance method for unmanned aerial vehicles (UAVs) according to Embodiment 1 of the present invention; Figure 2 This is a general block diagram of the real-time obstacle avoidance system for unmanned aerial vehicles (UAVs) according to an embodiment of the present invention. Figure 3 This is a flowchart of the binocular parallax estimation for UAVs according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an end-to-end decision network according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a real-time obstacle avoidance device for unmanned aerial vehicles (UAVs) according to Embodiment 2 of the present invention; Figure 6 This is a schematic diagram of the structure of a real-time obstacle avoidance device for unmanned aerial vehicles (UAVs) according to Embodiment 3 of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0019] It is understood that the specific embodiments and accompanying drawings described herein are merely for explaining the invention and are not intended to limit the invention.

[0020] It is understood that, without conflict, the various embodiments and features in the embodiments of the present invention can be combined with each other.

[0021] It is understood that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, while the parts unrelated to the present invention are not shown in the drawings.

[0022] It is understood that each unit or module involved in the embodiments of the present invention may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.

[0023] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this invention may occur in a different order than that marked in the accompanying drawings.

[0024] It is understood that the flowcharts and block diagrams of this invention illustrate the possible architecture, functions, and operations of systems, apparatuses, devices, and methods according to various embodiments of this invention. Each block in the flowchart or block diagram may represent a unit, module, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagram and flowchart can be implemented using a hardware-based system to achieve the specified function, or using a combination of hardware and computer instructions.

[0025] It is understood that the units and modules involved in the embodiments of the present invention can be implemented by software or by hardware. For example, the units and modules can be located in a processor.

[0026] Application Overview In the field of autonomous obstacle avoidance for UAVs, the mainstream approach often adopts a cascaded architecture of "mapping—localization—path planning—control": the front end uses binocular / monocular vision or visual inertial odometry (VIO) to complete feature extraction, matching, and pose estimation, combined with dense / semi-dense depth calculation or stereo parallax to generate local / global maps; the back end maintains consistency through graph optimization or sliding window optimization; and then global / local planning (such as...) is performed on this basis. The desired trajectory is obtained through methods such as sampling, curve optimization, or MPC, and finally the trajectory tracker outputs control commands.

[0027] While this link is generally applicable, it often leads to high computational burden and system coupling on edge platforms with poor texture, drastic lighting changes, shutter jitter, and limited computing power. Depth estimation, optimization, and planning all consume significant computational resources, and data copying and synchronization between modules introduce additional latency. In practical engineering, to ensure robustness, the front end often requires a larger window or a more complex cost function, and the planning end needs more frequent collision detection and replanning, further compressing the available frequency. This results in the entire link operating only at a few Hz to tens of Hz, making it difficult to maintain sub-millisecond closed-loop latency. Insufficient frequency directly leads to control lag and reduced safety margin: when the flight speed increases to 5-10 m / s and the obstacle distance shortens, the cumulative delay of perception-decision-execution causes the obstacle avoidance window to narrow rapidly, resulting in a "seeing-decision-outside-bounds" situation. At the same time, mapping and relocation are prone to degradation or drift in dynamic / low-texture environments, causing the planning results to deviate from the real environment. Coupled with low-frequency control, this makes it more prone to oscillation or over-conservatism, limiting the real-time obstacle avoidance capability of traditional methods on resource-constrained platforms.

[0028] In end-to-end vision control, existing publicly available solutions mostly use depth maps as network input. This approach introduces additional steps such as pixel-by-pixel inversion, invalid disparity filling, and scale calibration. In the long-range, small disparity range, it suffers from numerical amplification and quantization sensitivity issues. Furthermore, pixel-level consistency / confidence information from stereo matching is often not preserved. On computationally limited platforms, these intermediate processes further increase link latency and energy consumption, affecting closed-loop frequency and stability.

[0029] In summary, existing real-time obstacle avoidance methods for UAVs have low autonomous obstacle avoidance capabilities and reliability in resource-constrained scenarios.

[0030] To address the aforementioned technical problems, this application provides a method, apparatus, and medium for real-time obstacle avoidance by unmanned aerial vehicles (UAVs). By using an optimized disparity map as the core input and fusing the UAV's flight state vector, speed / attitude / yaw rate control commands are directly output via an end-to-end decision network. Compared to existing solutions, this invention avoids the high overhead of map building and global planning, significantly shortening the perception-to-control link and latency. Furthermore, by deploying RKNN on the edge and using a multi-threaded pipeline, end-to-end latency is ensured on resource-constrained platforms, thereby effectively improving the autonomous obstacle avoidance capability and reliability of UAVs in resource-constrained scenarios. This at least solves the problem of low autonomous obstacle avoidance capability and reliability of existing real-time obstacle avoidance methods for UAVs in resource-constrained scenarios.

[0031] After introducing the basic principles of this application, various non-limiting embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0032] Example 1: This embodiment provides a real-time obstacle avoidance method for UAVs. The method is deployed on a preset platform through a PyTorch→ONNX→RKNN model conversion process, employing a multi-threaded pipeline and a zero-copy buffer. Figure 1 As shown, the method includes: Step S101: Acquire synchronized stereo images from the UAV.

[0033] It should be noted that the synchronous binocular images of a drone refer to a pair of left and right images obtained by the binocular camera (consisting of two cameras with identical parameters and fixed installation positions) on the drone at the same time and simultaneously capturing the same scene. The preferred preset platform is the RK3588 platform.

[0034] In one optional embodiment, acquiring the synchronized binocular images of the UAV specifically includes: The synchronized stereo images are acquired through a hardware triggering mechanism or timestamp synchronization.

[0035] Specifically, synchronized stereo images are acquired and calibrated and corrected: camera intrinsic and extrinsic parameters and distortion parameters are obtained through stereo calibration. The camera synchronization mechanism is first hardware-triggered, and secondarily timestamp alignment is performed. The RMS reprojection error of the camera intrinsic parameters is preferably ≤0.30px, and the stereo RMS of the extrinsic parameters is preferably ≤0.50px. The temperature drift coefficient is recorded for temperature compensation. Distortion correction and epipolar rectification are performed on the left and right images (i.e., synchronized stereo images) to generate neatly aligned images for subsequent processing (such as disparity / depth estimation). Simultaneously, brightness equalization and light noise reduction are performed.

[0036] It's important to note that epipolar correction utilizes extrinsic and intrinsic parameters obtained from bi-target calibration to align the original left and right images to the same scan system through geometric transformation. This ensures that corresponding points fall on the same horizontal epipolar line, simplifying the two-dimensional matching problem into a one-dimensional search along the line, significantly reducing computational complexity and mismatch rate. Quality metrics include line alignment residuals (e.g., ≤0.5px) and stereo RMS reprojection error. Effective ROIs obtained through epipolar correction can also mask invalid edge regions, reducing subsequent computation.

[0037] Step S102: Based on the synchronized stereo image, obtain the disparity map corresponding to the synchronized stereo image, and optimize the disparity map to obtain the optimized disparity map.

[0038] It's important to note that disparity refers to the horizontal pixel displacement of the same 3D point in the left and right camera images after epipolar correction, expressed in pixels (px). Disparity maps can be obtained from traditional stereo matching (such as StereoBM) or directly estimated by lightweight edge networks. Compared to depth maps, disparity maps operate on a linear scale in the pixel domain, avoiding the numerical amplification and quantization sensitivity associated with mapping small disparities at long distances to depth, making them more suitable for low-bit inference methods like INT8.

[0039] In one optional embodiment, obtaining the disparity map corresponding to the synchronized stereo image based on the synchronized stereo image specifically includes: The disparity map is obtained by performing depth estimation on the synchronous stereo image using a stereo depth estimation algorithm or a lightweight convolutional network.

[0040] It should be noted that the stereo depth estimation algorithm uses StereoBM combined with WLS filtering. Specifically, the depth estimation uses StereoBM+WLS filtering, combined with left-right consistency checking and sub-pixel fitting, and enables speckle filtering; or an existing lightweight stereo network can be used, and the output is a disparity map that is directly fed into the network without the need for point cloud generation and dense mapping; at the same time, invalid regions of the disparity map are filled with holes, and a connected component algorithm is used to remove disparity inconsistencies.

[0041] In one optional embodiment, depth estimation is performed on the synchronized stereo images using a stereo depth estimation algorithm to obtain the disparity map, specifically including: Histogram equalization or local mean removal is performed on the left and right images of the synchronized stereo images; A window of a preset size is selected centered on each pixel in the processed left image, and the absolute difference and SAD between each window and the corresponding position window in the processed right image are calculated within a preset disparity search interval to obtain the matching cost for each pixel corresponding to different disparities. Based on the matching cost of different disparities corresponding to each pixel, the winner-take-all (WTA) strategy is executed on each pixel to obtain the initial disparity of each pixel; Each of the initial disparities is subjected to subpixel refinement, and each of the initial disparities after subpixel refinement is subjected to left-right consistency check (LRC). The weighted least squares (WLS) algorithm is used to perform guided filtering on each of the initial disparities after inspection to obtain the disparity map.

[0042] Specifically, the generation of disparity maps using StereoBM combined with WLS edge-preserving filtering includes: Use ≥20 sets of calibration materials, checkerboard patterns or AprilTags at different distances (0.3~3m) and different orientations (±60°) to complete the calibration and epipolar correction of the binocular camera.

[0043] After epipolar correction is completed, the left and right pixel rows of the image from the stereo camera are aligned. Let the left and right images of the stereo camera (i.e., the synchronized stereo images) be respectively... I L ( u , v ), I R ( u , v When the imaging coordinates of the same spatial point are on the same row, their parallax is defined as:

[0044] in, d (u, v (Image of the left eye) u , v Parallax at ) u L This represents the x-coordinate of the pixel in the left-eye image. u R This is the x-coordinate of the pixel in the right eye image.

[0045] The depth can be further obtained from the pinhole model:

[0046] in, Z ( u , v ) is the left eye image ( u , v The depth value at ) f Focal length (in pixels). B This represents the baseline length.

[0047] Select with ( u , v Centered on, window W (e.g. (2) r +1)×(2 r The cost is calculated within the disparity search interval for blocks of +1). The sum of absolute differences (SAD) is commonly used.

[0048] in, W To match the window, To match the sum of absolute differences (SAD) within the window. d Assuming parallax, I L andI R These are the grayscale values ​​of the left and right eye images from a binocular camera. i and j This represents the relative pixel offset within the window.

[0049] To suppress changes in illumination, histogram equalization / local mean removal is performed before the cost.

[0050] Execute the Winner-Take-All (WTA) strategy for each pixel:

[0051] in, For pixels ( x , y The optimal integer disparity is obtained by finding the independent variable that minimizes the objective function.

[0052] To improve accuracy, utilize Sub-pixel thinning is performed using parabolic interpolation of three neighboring points, taking the value of its three adjacent points:

[0053] The sub-pixel offset can be directly obtained from three-point interpolation:

[0054] This yields the subpixel disparity estimate:

[0055] in, d L To estimate the obtained disparity value, For optimal integer disparity, δ This is a sub-pixel correction amount relative to integer disparity. C -1 , C 0、 C +1 They represent parallax respectively. - 1、 , The matching cost at +1 can effectively reduce disparity quantization error and improve the continuity and accuracy of depth estimation.

[0056] The disparity value of a certain pixel in the left eye can be calculated using the formula above. d L The same method can be used to calculate the corresponding pixels for the right eye. d R For the left-eye view pixels ( u , v The corresponding pixel coordinates of the right view are:

[0057] in, u R This represents the coordinates of the pixel in the left eye within the right eye.

[0058] For the same spatial point, it should point back to each other in the left and right viewpoints; therefore, a left-right consistency check (LRC) is performed.

[0059] in, d L ( u , v ) is the left eye view pixel ( u , v The parallax of ) d R ( u R , v () represents the parallax of the right-eye view at the corresponding position. τ This is the threshold for left-right consistency.

[0060] It should be noted that the Left-Right Consistency Check (LRC) involves cross-checking left-to-right and right-to-left matching results to remove inconsistent pixels. This method effectively removes mismatches, low-texture pixels, and spurious disparities at occluded edges, significantly improving subsequent depth / disparity quality and the reliability of safety decisions.

[0061] To preserve discontinuities at object edges and smooth noise within the texture, weighted least squares (WLS) is used to adjust the initial disparity. d L Perform guided filtering, here we use d L ( u , v ) indicates that the objective function is minimized:

[0062] in, d L ( u , v The original left view disparity after LRC is represented. ( u , v () represents the disparity after WLS filtering to be solved. p and q For the index of adjacent pixels in the neighborhood, This represents the difference in disparity between adjacent pixels p and q within the neighborhood after WLS filtering. For a pixel, it is the local neighborhood set. λ As a smoothing weighting factor, w pqThe weights are based on the guide map (i.e., based on the grayscale difference of the left eye image).

[0063] The objective function attempts to strike a balance between two mutually constraining requirements. On the one hand, the filtered disparity should be as close as possible to the original disparity after left-right consistency checks to ensure that the disparity information is not overly modified; on the other hand, the disparity at adjacent pixels should remain smooth to suppress matching noise and local outliers.

[0064] Weight w pq Gradient of the left eye image I L The typical approach is to (weaken and smooth at the edges):

[0065] in, I L ( p (The left eye image is in pixels) p grayscale value at that location I L ( q (The left eye image is in pixels) q grayscale value at that location σ This is the grayscale sensitivity coefficient. The greater the grayscale difference, the smaller the weight, thus suppressing smoothing across edges.

[0066] Finally, a disparity map is generated and used as input to the end-to-end decision network.

[0067] In one optional embodiment, optimizing the disparity map to obtain an optimized disparity map specifically includes: The disparity map is cropped and preprocessed, and then downsampled to a preset resolution size to obtain the optimized disparity map.

[0068] Specifically, a lightweight preprocessing (i.e. optimization) process is performed on the disparity map generated from the binocular images: stereo matching (such as StereoBM combined with WLS filtering) or a lightweight binocular depth network is used to obtain the disparity map of the current frame. Then, cropping and preprocessing are performed (e.g., cropping the disparity map to...). It can also employ logarithmic enhancement of near-range resolution, and then downsample to a preset low-resolution input size (such as 12×16~64×64). The input size is preferably 12×16, 32×48 or 64×64 to reduce the computational burden. Morphological filling can be optionally performed on isolated holes to improve robustness.

[0069] Step S103: Obtain the flight state vector of the UAV, wherein the flight state vector includes at least one of the following: speed, attitude, and target distance.

[0070] It should be noted that the flight state vector of a drone has a dimension of 7 to 10, and includes at least one of speed, attitude and altitude / relative distance (i.e., target distance).

[0071] Specifically, the flight state vector is acquired and tuned by collecting at least one of the following: the speed, attitude, altitude, or relative distance to the target of the unmanned aerial vehicle system. The vector is then combined to form a state vector and subjected to dimensional normalization / amplitude limiting processing, and aligned with the timestamp of the depth input.

[0072] Step S104: Input the optimized disparity map and flight state vector into the end-to-end decision network to obtain control commands, wherein the control commands include at least one of the following: forward / lateral / vertical speed, yaw rate, and attitude setpoint.

[0073] It should be noted that End-to-End Decision Network (E2E) refers to a learning model that directly maps perceived inputs to control outputs. Compared to the traditional cascaded "mapping-localization-planning-control" process, E2E significantly shortens the link and reduces coupling and latency.

[0074] In one optional embodiment, the end-to-end decision network is a network structure combining lightweight convolutional and recurrent units. The step of inputting the optimized disparity map and flight state vector into the end-to-end decision network to obtain control commands specifically includes: The optimized disparity map and flight state vector are input into the end-to-end decision network. The end-to-end decision network performs lightweight convolutional feature extraction and fusion of recurrent unit historical information on the optimized disparity map and flight state vector, and outputs the control command.

[0075] Specifically, the end-to-end decision network infers control commands: the downsampled disparity map and flight state vector are used as inputs, and temporal information (i.e., historical information) is fused through existing lightweight convolutional feature extraction and recurrent units (such as GRUs) to output action vectors (preferably including forward / lateral / vertical velocity, yaw rate, and attitude setpoint). The network structure adopts existing lightweight combinations and simultaneously implements a systematic approach to end-side RKNN deployment and real-time control pipeline.

[0076] It should be noted that the preferred real-time performance metrics are a control frequency ≥15Hz and an end-to-end latency ≤100ms; thread contention and cache jitter are reduced through pipelined parallelism, priority scheduling, and zero-copy.

[0077] In one optional embodiment, the end-to-end decision network is pre-trained using a simulation environment with domain randomization and fine-tuned based on real flight data. The network operators of the end-to-end decision network are the basic operators of RKNN.

[0078] It should be noted that behavior cloning combined with out-of-bounds penalty and temporal smoothing is commonly used in the training of end-to-end decision networks. First, domain randomization (texture, lighting, noise, occlusion, wind disturbance, baseline error) is performed in the simulation for pre-training, and then fine-tuning is performed using real flight data.

[0079] Specifically, the end-to-end decision network takes the optimized disparity map and flight state vector as input, performs lightweight convolutional feature extraction and temporal unit fusion, and directly outputs control quantities such as velocity, attitude, and yaw rate. The network operators selected are basic operations adapted to RKNN (such as Conv2D, ReLU, BN / folding, Add, MatMul, etc.) to facilitate INT8 quantization and NPU acceleration. It should be noted that quantization (INT8 Quantization) maps the weights and activations of the trained floating-point model (FP32 / FP16) to an 8-bit integer representation (INT8) to reduce storage, bandwidth, and computational costs, and improve the throughput and energy efficiency of the edge NPU / CPU. On NPUs such as the RK3588, INT8 often brings a 2-4x speedup and a significant reduction in power consumption.

[0080] Taking a three-layer lightweight convolution as an example (small convolutional kernel, stride downsampling, suitable for real-time inference on the edge), the network forward propagation can be represented as:

[0081] Where D is the input disparity map after LRC+WLS filtering optimization. For two-dimensional convolution operations, W i No. i Layer convolution kernel weights, b i For the first i Layer convolution bias, σ For non-linear activation functions, choose ReLU or LeakyReLU, F i For the first i Layer convolutional feature map, s t For a moment t The flight state vector, Φ is the feature fusion and temporal unit, h t For the fused temporal hidden state features, W o b represents the weights of the fully connected output layer. o For output layer bias, u t This is the control vector output by the network.

[0082] Typical channel counts for the convolutional feature extraction part are (32, 64, 128). The features output by the third convolutional layer are flattened to form a dimension of 192. Typical input dimensions for the disparity map D are (12×16, 32×48, 64×64).

[0083] Timing fusion uses lightweight GRU units, and single-step updates are as follows:

[0084]

[0085]

[0086] Where ⊙ represents element-wise multiplication, x t For a moment t The fused input feature vector, h t-1 h is the hidden state from the previous moment. t The current hidden state. For the candidate hidden state, z t To update the gate vector, r t To reset the gate vector, σ Let be the Sigmoid activation function, tanh be the hyperbolic tangent activation function, W be the weight matrix from input to gate and candidate states, U be the weight matrix from hidden state to gate and candidate states, and b be the bias vector. The hidden dimension is typically 192, i.e., h t ∈ .

[0087] Step S105: Perform safety constraint processing on the control command and send the safety constraint-processed control command to the flight controller so that the flight controller can perform closed-loop control based on the safety constraint-processed control command to realize real-time obstacle avoidance of the UAV.

[0088] Specifically, it performs safety constraints and command shaping, and controls the interface for issuing and executing commands in a closed loop: based on safety constraints, it performs speed / angular velocity limiting, boundary safety margin checks, and low-confidence emergency stop strategies on control commands, and issues them to the flight controller for execution via MAVLink or similar protocols, in a closed loop.

[0089] It should be noted that the robustness mechanism of the present invention includes perception degradation detection, NPU fault and overload detection, rollback and rate limiting strategies, communication out-of-step protection and attitude anomaly recovery.

[0090] It is worth mentioning that the deployment process of this invention is PyTorch→ONNX→RKNN, and the quantization calibration set covers boundary conditions such as low texture / strong light / weak light, ensuring the accuracy and stability of edge inference. This invention is compatible with common flight control / ground station protocols and can be quickly integrated under ROS / MAVROS architecture, which is convenient for engineering implementation and maintenance.

[0091] In one specific embodiment, the real-time obstacle avoidance method for UAVs provided in this embodiment is applied to the real-time obstacle avoidance system for UAVs based on lightweight binocular depth estimation and end-to-end provided in this embodiment, such as... Figure 2 As shown, Figure 2 This is a general block diagram of the real-time obstacle avoidance system for unmanned aerial vehicles (UAVs) of the present invention. Starting from a binocular camera, after epipolar correction and disparity estimation, a downsampled disparity map is obtained and sent to an end-to-end decision network; state estimation generates a state vector input in parallel. The control quantity output by the network is processed by safety constraints and gating, then encapsulated as MAVLink commands and sent to the flight controller to drive the propulsion system to achieve closed-loop control. Specifically, this real-time obstacle avoidance system for UAVs includes: Binocular camera: global shutter, resolution 640×480~1280×720, frame rate 15~30fps; intrinsic, extrinsic and distortion parameters are obtained through binocular calibration, and hardware trigger / timestamp synchronization is supported.

[0092] The disparity estimation module performs distortion correction and epipolar correction on the left and right images based on calibration parameters, ensuring that corresponding points are in the same row and outputting effective ROIs. It performs matching along the epipolar direction based on StereoBM and / or a lightweight CNN to obtain an initial disparity map and optional confidence / consistency information. It uses the grayscale of the left image as a guide to perform edge-preserving smoothing of the disparity, suppressing noise while maintaining edges. It performs neighborhood / guided filling on invalid / occluded regions and uses temporal median filtering to stabilize the temporal sequence.

[0093] End-to-end decision network: Inputs are a downsampled disparity map and a state vector. Historical information is fused using lightweight convolution and GRU / LSTM. Outputs forward / lateral / vertical velocities and yaw rates or attitude setpoints, along with action confidence / uncertainty. Simultaneously, it performs amplitude / acceleration / jump limiting, boundary safety margin checks, and low-confidence deceleration / emergency stop on the control variables.

[0094] Communication and Flight Control Module: Encapsulates control commands into MAVLink and sends them to the flight controller. The flight controller executes closed-loop control, driving the ESC and motors to complete obstacle avoidance flight.

[0095] Edge computing unit: RK3588 (including NPU / CPU) handles inference and preprocessing, with a multi-threaded zero-copy pipeline for parallel scheduling of camera, inference, and deployment. The system uses INT8 quantization for NPU inference on the RK3588 platform, with a control frequency of 50Hz and end-to-end latency ≤10ms.

[0096] Specifically, the real-time obstacle avoidance method for drones may include: Step 1: Acquire synchronized left and right view images (i.e., synchronized binocular images) and perform camera intrinsic and extrinsic parameter calibration and epipolar correction; Specifically, the left and right eye images are acquired through a hardware triggering mechanism, and brightness equalization and light noise reduction are performed simultaneously.

[0097] Step 2: Perform depth estimation on the left and right images obtained in Step 1 based on the binocular depth estimation algorithm and / or lightweight convolutional network to obtain disparity maps, and preprocess and downsample the disparity maps; Specifically, the depth estimation algorithm employs StereoBM combined with WLS guided filtering, and the output is a disparity map directly input to the end-to-end decision network. The StereoBM's SAD window is preferably 5–15, the disparity search range is preferably 64 or 128, and left-right consistency checks are used to eliminate occlusions and incorrect matches. The input size of the depth map (disparity map) is 12×16–64×64, preferably 12×16 or 32×48. The StereoBM output undergoes left-right consistency checks and sub-pixel fitting (parabolic or correlation peak interpolation) to improve effective disparity density and accuracy.

[0098] Specifically, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the binocular disparity estimation (i.e., depth estimation) process. Starting from the binocular camera, after epipolar correction and disparity estimation, the process sequentially passes through left-right consistency checks, WLS guided filtering, hole filling, and temporal filtering to obtain a disparity map, which is then fed into the end-to-end decision network.

[0099] Step 3: Obtain the UAV flight status, which includes at least one of velocity, attitude and / or target distance, and form a state vector; Specifically, the state vector has a dimension of 7 to 10 and includes at least one of the following: body velocity, attitude quaternion, and relative distance to the target.

[0100] Step 4: Input the preprocessed disparity map obtained in Step 2 and the state vector obtained in Step 3 into the end-to-end decision network. After extracting historical information through convolutional features and fusing with recurrent units, output at least one of forward, lateral and vertical velocities and yaw rate or attitude set point. Specifically, the end-to-end decision network is a network structure combining existing lightweight convolutional and recurrent units, using 2×2 or 3×3 convolutional kernels, with batch=1 executed on the edge, a control frequency of 10-30Hz, and an end-to-end latency ≤100ms. The end-to-end decision network uses a simulation environment for policy pre-training and domain randomization, followed by fine-tuning using real flight data. The domain randomization covers at least one of texture, illumination, camera noise, baseline error, wind disturbance, and obstacle shape distribution. The fine-tuning stage uses at least one of behavior cloning and reinforcement learning hybrid loss.

[0101] Specifically, such as Figure 4 As shown, Figure 4 This is a schematic diagram of an end-to-end decision network structure. Left branch processing: Input disparity map, perform preprocessing, run the visual backbone network (perform multi-layer lightweight convolution and downsampling on the input single-channel disparity map, gradually extract spatial perception features, and flatten the two-dimensional feature map into a fixed-dimensional one-dimensional feature vector). The right-side branch processes the following: input state, running state encoding module (mapping the low-dimensional flight state vector to a representation space of the same dimension as the image features, providing a consistent feature scale for subsequent fusion), feature fusion module (fusing image features and state features by element-wise addition, and enhancing feature expressiveness through nonlinear activation), running sequence modeling module (using lightweight GRU units to perform temporal modeling of the fused features, and combining historical hidden states to capture system dynamic information), and output action. Specifically, the downsampled disparity map output by the binocular depth estimation module and the state vector generated by state estimation are used as inputs into the end-to-end decision network. After disparity perception feature extraction (anisotropic lightweight convolution), camera scale modulation, and state vector encoding, historical information is fused in the temporal unit (GRU / LSTM) to output action control quantity and action confidence. The control quantity is encapsulated into MAVLink commands and sent to the flight controller after safety constraints, gating, and boundary margin checks.

[0102] Step 5: Based on safety constraints, the control commands output in Step 4 are subjected to amplitude limiting, boundary safety margin checks, and low-confidence emergency stop strategies. These are then sent to the flight controller for execution via MAVLink or a similar protocol, forming a closed-loop cycle.

[0103] Specifically, the safety constraints include a speed limit of 1 to 5 m / s and a boundary safety margin of 0.1 to 0.3 field of view ratio.

[0104] It should be noted that the method described is deployed on the RK3588 platform via a PyTorch→ONNX→RKNN model conversion process. Inference runs on the NPU using INT8 quantization, while preprocessing and post-processing are performed on the CPU. Quantization employs either a symmetric or asymmetric scheme and is calibrated using a calibration set of 200–1000 frames. A multi-threaded pipeline and zero-copy buffer are used for parallel scheduling among camera acquisition, deep inference, decision inference, and control delivery.

[0105] It is worth mentioning that this invention uses the disparity map generated by a binocular camera as the core input (without the need to build point clouds / dense maps), integrates the UAV flight state vector, and directly outputs control commands such as speed / attitude / yaw rate through an end-to-end decision network. Furthermore, through RKNN edge deployment and multi-threaded real-time pipeline, it ensures a control frequency of ≥15Hz and an end-to-end latency of ≤100ms on resource-constrained platforms, thereby achieving stable and low-latency autonomous obstacle avoidance even in scenarios without high computing power support.

[0106] Compared with the traditional long-link scheme of "mapping-localization-planning-control", this invention avoids the high overhead of map building and global planning, and significantly shortens the link and latency from perception to control. On the edge computing platform, a stable real-time frequency is achieved through RKNN quantization and pipeline parallelism, thereby effectively improving the autonomous obstacle avoidance capability and reliability of UAVs in resource-constrained scenarios.

[0107] Meanwhile, unlike existing schemes that mostly use depth maps as end-to-end inputs, this method directly uses disparity maps as the network's native input, eliminating the need for pixel-by-pixel conversion and scale calibration from disparity to depth; combined with camera scale conditions and a lightweight CNN+GRU structure, it reduces latency and energy consumption on edge platforms.

[0108] The real-time obstacle avoidance method for unmanned aerial vehicles (UAVs) provided in this embodiment of the invention first acquires a synchronized binocular image of the UAV; then, based on the synchronized binocular image, a disparity map corresponding to the synchronized binocular image is obtained, and the disparity map is optimized to obtain an optimized disparity map; next, the flight state vector of the UAV is acquired, wherein the flight state vector includes at least one of the following: velocity, attitude, and target distance; then, the optimized disparity map and the flight state vector are input into an end-to-end decision network to obtain control commands, wherein the control commands include at least one of the following: forward / lateral / vertical velocity, yaw rate, and attitude setpoint; finally, the control commands are subjected to safety constraint processing, and the safety constraint-processed control commands are sent to the flight controller, so that the flight controller performs closed-loop control based on the safety constraint-processed control commands to achieve real-time obstacle avoidance by the UAV. This invention uses an optimized disparity map as the core input, integrates the UAV flight state vector, and directly outputs speed / attitude / yaw rate control commands through an end-to-end decision network. Compared with existing solutions, this invention avoids the high overhead of map building and global planning, significantly shortens the perception-to-control link and latency, and ensures end-to-end latency on resource-constrained platforms through RKNN edge deployment and multi-threaded pipeline. This effectively improves the autonomous obstacle avoidance capability and reliability of UAVs in resource-constrained scenarios, solving the problem of low autonomous obstacle avoidance capability and reliability of existing real-time obstacle avoidance methods for UAVs in resource-constrained scenarios.

[0109] Example 2: like Figure 5 As shown, this embodiment provides a real-time obstacle avoidance device for unmanned aerial vehicles (UAVs). The device is set up on a preset platform via a PyTorch→ONNX→RKNN model conversion process, employing a multi-threaded pipeline and a zero-copy buffer to execute the aforementioned real-time obstacle avoidance method for UAVs, including: The first acquisition module 11 is used to acquire synchronized stereo images of the UAV; The processing module 12 is connected to the first acquisition module 11 and is used to obtain the disparity map corresponding to the synchronized stereo image based on the synchronized stereo image, and to optimize the disparity map to obtain the optimized disparity map. The second acquisition module 13, connected to the acquisition processing module 12, is used to acquire the flight state vector of the UAV, wherein the flight state vector includes at least one of the following: speed, attitude, and target distance; The input module 14, connected to the second acquisition module 13, is used to input the optimized disparity map and flight state vector into the end-to-end decision network to obtain control commands, wherein the control commands include at least one of the following: forward / lateral / vertical speed, yaw rate, attitude setpoint; The processing and sending module 15 is connected to the input obtaining module 14. It is used to perform safety constraint processing on the control command and send the safety constraint processed control command to the flight controller so that the flight controller can perform closed-loop control based on the safety constraint processed control command to realize the UAV obstacle avoidance in real time.

[0110] Furthermore, the first acquisition module 11 specifically includes: The acquisition unit is used to acquire the synchronized stereo image synchronously through a hardware triggering mechanism or a timestamp.

[0111] Furthermore, the processing module 12 specifically includes: The depth estimation unit is used to perform depth estimation on the synchronous stereo image using a stereo depth estimation algorithm or a lightweight convolutional network to obtain the disparity map.

[0112] Furthermore, the stereo depth estimation algorithm employs StereoBM combined with WLS filtering, and the depth estimation unit specifically includes: The equalization removal unit is used to perform histogram equalization or local mean removal processing on the left and right images of the synchronized stereo images. A calculation unit is selected to select a window of a preset size centered on each pixel in the processed left image, and to calculate the absolute difference and SAD between each window and the corresponding position window in the processed right image within a preset disparity search interval, so as to obtain the matching cost for each pixel corresponding to different disparities. The execution unit is used to execute the Winner-Take-All (WTA) strategy on each pixel based on the matching cost of different disparities corresponding to each pixel, and obtain the initial disparity of each pixel; The processing and inspection unit is used to perform sub-pixel refinement processing on each of the initial disparities, and to perform left-right consistency check (LRC) on each of the initial disparities after sub-pixel refinement processing. The guided filtering unit is used to perform guided filtering on each of the initial disparities after inspection using the weighted least squares algorithm (WLS) to obtain the disparity map.

[0113] Furthermore, the processing module 12 further includes: The processing downsampling unit is used to crop and preprocess the disparity map, and downsample the cropped and preprocessed disparity map to a preset resolution size to obtain the optimized disparity map.

[0114] Furthermore, the end-to-end decision network is a network structure combining lightweight convolutional and recurrent units, and the input acquisition module 14 specifically includes: The input / output unit is used to input the optimized disparity map and flight state vector into the end-to-end decision network, and to perform lightweight convolutional feature extraction and fusion of historical information from recurrent units on the optimized disparity map and flight state vector through the end-to-end decision network, and output the control command.

[0115] Furthermore, the end-to-end decision network is pre-trained using a simulation environment with domain randomization and fine-tuned based on real flight data. The network operators of the end-to-end decision network are the basic operators of RKNN.

[0116] Example 3: refer to Figure 6 This embodiment provides a real-time obstacle avoidance device for unmanned aerial vehicles (UAVs), including a memory 21 and a processor 22. The memory 21 stores a computer program, and the processor 22 is configured to run the computer program to execute the real-time obstacle avoidance method for UAVs in Embodiment 1.

[0117] The memory 21 is connected to the processor 22. The memory 21 can be a flash memory, a read-only memory or other memory, and the processor 22 can be a central processing unit or a microcontroller.

[0118] Example 4: This embodiment provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the real-time obstacle avoidance method for unmanned aerial vehicles in Embodiment 1 above.

[0119] The computer-readable storage medium includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, computer program modules or other data. Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other memory technologies, CD-ROM (Compact Disc Read-Only Memory), DVD or other optical disc storage, cartridges, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer.

[0120] In summary, the real-time obstacle avoidance method, apparatus, and medium for unmanned aerial vehicles (UAVs) provided in this embodiment of the invention first acquires a synchronized binocular image of the UAV; then, based on the synchronized binocular image, a disparity map corresponding to the synchronized binocular image is obtained, and the disparity map is optimized to obtain an optimized disparity map; next, the flight state vector of the UAV is acquired, wherein the flight state vector includes at least one of the following: velocity, attitude, and target distance; then, the optimized disparity map and flight state vector are input into an end-to-end decision network to obtain control commands, wherein the control commands include at least one of the following: forward / lateral / vertical velocity, yaw rate, and attitude setpoint; finally, the control commands are subjected to safety constraint processing, and the safety constraint-processed control commands are sent to the flight controller, so that the flight controller performs closed-loop control based on the safety constraint-processed control commands to achieve real-time obstacle avoidance by the UAV. This invention uses an optimized disparity map as the core input, integrates the UAV flight state vector, and directly outputs speed / attitude / yaw rate control commands through an end-to-end decision network. Compared with existing solutions, this invention avoids the high overhead of map building and global planning, significantly shortens the perception-to-control link and latency, and ensures end-to-end latency on resource-constrained platforms through RKNN edge deployment and multi-threaded pipeline. This effectively improves the autonomous obstacle avoidance capability and reliability of UAVs in resource-constrained scenarios, solving the problem of low autonomous obstacle avoidance capability and reliability of existing real-time obstacle avoidance methods for UAVs in resource-constrained scenarios.

[0121] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for real-time obstacle avoidance for unmanned aerial vehicles, the method comprising: The method is deployed on a preset platform through a PyTorch→ONNX→RKNN model conversion process, employing a multi-threaded pipeline and a zero-copy buffer. The method includes: Acquire synchronized binocular images from the drone; Based on the synchronized stereo image, a disparity map corresponding to the synchronized stereo image is obtained, and the disparity map is optimized to obtain the optimized disparity map. Obtain the flight state vector of the UAV, wherein the flight state vector includes at least one of the following: velocity, attitude, and target distance; The optimized disparity map and flight state vector are input into an end-to-end decision network to obtain control commands, wherein the control commands include at least one of the following: forward / lateral / vertical velocity, yaw rate, and attitude setpoint. The control commands are subjected to safety constraints, and the control commands with safety constraints are sent to the flight controller so that the flight controller can perform closed-loop control based on the control commands with safety constraints, so as to realize the UAV obstacle avoidance in real time.

2. The method of claim 1, wherein, The acquisition of synchronized binocular images from the UAV specifically includes: The synchronized stereo images are acquired through a hardware triggering mechanism or timestamp synchronization.

3. The method of claim 1, wherein, The step of obtaining the disparity map corresponding to the synchronized stereo image based on the synchronized stereo image specifically includes: The disparity map is obtained by performing depth estimation on the synchronous stereo image using a stereo depth estimation algorithm or a lightweight convolutional network.

4. The method of claim 3, wherein, The stereo depth estimation algorithm employs StereoBM combined with WLS filtering. It performs depth estimation on the synchronized stereo image using this algorithm to obtain the disparity map, specifically including: Histogram equalization or local mean removal is performed on the left and right images of the synchronized stereo images; A window of a preset size is selected centered on each pixel in the processed left image, and the absolute difference and SAD between each window and the corresponding position window in the processed right image are calculated within a preset disparity search interval to obtain the matching cost for each pixel corresponding to different disparities. Based on the matching cost of different disparities corresponding to each pixel, the winner-take-all (WTA) strategy is executed on each pixel to obtain the initial disparity of each pixel; Each of the initial disparities is subjected to subpixel refinement, and each of the initial disparities after subpixel refinement is subjected to left-right consistency check (LRC). The weighted least squares (WLS) algorithm is used to perform guided filtering on each of the initial disparities after inspection to obtain the disparity map.

5. The method of claim 1, wherein, The optimization process of the disparity map to obtain the optimized disparity map specifically includes: The disparity map is cropped and preprocessed, and then downsampled to a preset resolution size to obtain the optimized disparity map.

6. The method of claim 1, wherein, The end-to-end decision network is a network structure combining lightweight convolutional and recurrent units. The process of inputting the optimized disparity map and flight state vector into the end-to-end decision network to obtain control commands specifically includes: The optimized disparity map and flight state vector are input into the end-to-end decision network. The end-to-end decision network performs lightweight convolutional feature extraction and fusion of recurrent unit historical information on the optimized disparity map and flight state vector, and outputs the control command.

7. The method of claim 1, wherein, The end-to-end decision network is pre-trained using a simulation environment with domain randomization and fine-tuned based on real flight data. The network operators of the end-to-end decision network are the basic operators of RKNN.

8. A real-time obstacle avoidance device for a drone, comprising: The device is set up on a preset platform via a PyTorch→ONNX→RKNN model conversion process, employing a multi-threaded pipeline and a zero-copy buffer, including: The first acquisition module is used to acquire synchronized stereo images of the UAV; A processing module is connected to the first acquisition module and is used to obtain a disparity map corresponding to the synchronized stereo image based on the synchronized stereo image, and to optimize the disparity map to obtain the optimized disparity map. The second acquisition module, connected to the acquisition processing module, is used to acquire the flight state vector of the UAV, wherein the flight state vector includes at least one of the following: speed, attitude, and target distance; The input module, connected to the second acquisition module, is used to input the optimized disparity map and flight state vector into the end-to-end decision network to obtain control commands, wherein the control commands include at least one of the following: forward / lateral / vertical velocity, yaw rate, and attitude setpoint; The processing and sending module, connected to the input receiving module, is used to perform safety constraint processing on the control commands and send the safety constraint-processed control commands to the flight controller, so that the flight controller can perform closed-loop control based on the safety constraint-processed control commands to achieve real-time obstacle avoidance by the UAV.

9. A real-time obstacle avoidance device for a drone, comprising: It includes a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to implement the real-time obstacle avoidance method for unmanned aerial vehicles as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the real-time obstacle avoidance method for unmanned aerial vehicles as described in any one of claims 1-7.

Images