An unmanned aerial vehicle autonomous obstacle avoidance method based on deep learning and binocular vision

Abstract

The application relates to the technical field of unmanned aerial vehicle obstacle avoidance, and discloses an unmanned aerial vehicle autonomous obstacle avoidance method based on deep learning and binocular vision. The method acquires left view and right view original data streams through binocular vision acquisition terminals, extracts stereoscopic vision feature representation through parallel convolution coding branches of a deep neural network model, inputs a three-dimensional space reconstruction module to generate a dense depth map and an initial position coordinate set of obstacles, combines a dynamic obstacle analysis engine with real-time flight attitude parameters of the unmanned aerial vehicle to calculate a dynamic threat evaluation index, calculates a future moving path probability distribution of the obstacle according to the space-time trajectory prediction model, generates a three-dimensional obstacle avoidance heading correction vector by fusing the distribution and preset navigation path planning data, converts the three-dimensional obstacle avoidance heading correction vector into a flight control instruction set, and transmits the flight control instruction set to an execution module. The method improves the reliability and adaptability of autonomous obstacle avoidance of the unmanned aerial vehicle in a complex dynamic environment, and provides a powerful guarantee for safe navigation of the unmanned aerial vehicle.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle obstacle avoidance, in particular to an unmanned aerial vehicle autonomous obstacle avoidance method based on deep learning and binocular vision. BACKGROUND

[0002] With the continuous expansion of the application scenarios of unmanned aerial vehicles, from low-altitude reconnaissance, logistics distribution to agricultural plant protection and other fields, the requirements for the autonomous obstacle avoidance capability of unmanned aerial vehicles are increasingly stringent. Traditional unmanned aerial vehicle obstacle avoidance methods mostly rely on a single sensor, such as ultrasonic wave, laser radar or monocular vision, which often faces limitations in complex environments. The ultrasonic wave sensor is limited by the detection distance and angle, and is prone to miss detection in open space; although the laser radar can obtain accurate depth information, the device cost is relatively high, and the data stability decreases in strong light or rainy and snowy weather.

[0003] Monocular vision estimates depth through motion parallax, but the accuracy is greatly reduced in static scenes or slow movement, which is difficult to meet the real-time obstacle avoidance requirements. Although the existing binocular vision-based method can directly calculate the parallax to obtain the depth, it is limited by the efficiency of the traditional feature matching algorithm, and when dealing with dynamic obstacles, the trajectory prediction deviation is often caused by feature point loss or mismatch. At the same time, most obstacle avoidance systems only consider the current position of the obstacle, without combining its motion trend and the navigation parameters of the unmanned aerial vehicle itself, which is prone to cause unreasonable obstacle avoidance path in complex environments, and even causes collision risk.

[0004] Traditional obstacle avoidance decisions mostly rely on preset rules, lack the self-adaptive ability to the dynamic changes of the environment, and in the scene of the cross movement of multiple obstacles, it is difficult to quickly generate the optimal obstacle avoidance strategy, which limits the application range of unmanned aerial vehicles in complex dynamic environments. SUMMARY

[0005] The purpose of the present application is to provide an unmanned aerial vehicle autonomous obstacle avoidance method based on deep learning and binocular vision to solve the problems raised in the background.

[0006] To achieve the above purpose, the present application provides an unmanned aerial vehicle autonomous obstacle avoidance method based on deep learning and binocular vision, which comprises:

[0007] The binocular vision acquisition terminal synchronously obtains left view and right view original data streams in the flight environment;

[0008] The deep neural network model synchronously processes the left view and right view original data streams, and extracts the stereoscopic vision feature representation in the binocular image through a parallel convolutional coding branch;

[0009] The stereoscopic vision feature representation is input into a three-dimensional space reconstruction module to generate a flight environment dense depth map and an obstacle initial position coordinate set;

[0010] The dynamic obstacle analysis engine receives the dense depth map and the set of initial position coordinates of the obstacle, and calculates an obstacle dynamic threat evaluation index in combination with real-time flight attitude parameters of the UAV;

[0011] Based on the obstacle dynamic threat evaluation index, a future moving path probability distribution of the obstacle is calculated through a space-time trajectory prediction model;

[0012] The future moving path probability distribution of the obstacle and preset navigation path planning data of the UAV are fused to generate a three-dimensional obstacle avoidance heading correction vector;

[0013] The three-dimensional obstacle avoidance heading correction vector is converted into a flight control instruction set and is transmitted to a UAV power system execution module.

[0014] Preferably, the left view and right view original data streams contain RGB color channel information and infrared thermal imaging data, and the real-time flight attitude parameters of the UAV include a pitch angle, a roll angle, a yaw angle, an altitude, and a ground speed vector.

[0015] Preferably, the extraction of the stereo vision feature representation in the binocular image through the parallel convolution coding branch includes:

[0016] A multi-scale feature pyramid convolution operation is performed on the left view original data stream to generate a left view multi-level feature tensor sequence;

[0017] A feature enhancement processing of a cross-channel attention mechanism is performed on the right view original data stream to generate a right view optimized feature tensor sequence;

[0018] The left view multi-level feature tensor sequence and the right view optimized feature tensor sequence are cross-view feature matched and aligned to form a stereo vision feature representation matrix fused with binocular disparity.

[0019] Preferably, the dynamic obstacle analysis engine receives the dense depth map and the set of initial position coordinates of the obstacle, and calculates an obstacle dynamic threat evaluation index in combination with real-time flight attitude parameters of the UAV, including:

[0020] The spatial distribution density of the set of initial position coordinates of the obstacle in the dense depth map is analyzed;

[0021] The angle between the ground speed vector direction in the real-time flight attitude parameters of the UAV and the surface normal vector of the obstacle is associated;

[0022] According to the spatial distribution density of the obstacle and the angle between the surface normal vectors, in combination with a preset collision risk level mapping table, an obstacle dynamic threat evaluation index is generated.

[0023] Preferably, the process of generating the flight environment dense depth map by the three-dimensional space reconstruction module includes:

[0024] decomposing the stereovision feature representation matrix into a disparity feature channel and a texture feature channel;

[0025] performing sub-pixel level interpolation calculation on the disparity feature channel through a spatial gridding algorithm to construct an initial depth probability distribution field;

[0026] performing noise filtering optimization on the initial depth probability distribution field by fusing edge gradient information in the texture feature channel to output a dense depth map.

[0027] Preferably, the stereovision feature representation is input into a three-dimensional space reconstruction module to generate a flight environment dense depth map, including:

[0028] extracting a sequence of obstacle historical moving trajectory segments from the dense depth map;

[0029] performing time series modeling on the sequence of obstacle historical moving trajectory segments using a gated recurrent unit network to generate an obstacle motion state latent variable representation;

[0030] inputting the obstacle motion state latent variable representation into a conditional random field model to predict an obstacle future moving path probability distribution heat map.

[0031] Preferably, the fusion of the obstacle future moving path probability distribution and the preset UAV flight path planning data includes:

[0032] converting the obstacle future moving path probability distribution heat map into a three-dimensional space occupancy grid model;

[0033] labeling a set of path key waypoints in the preset UAV flight path planning data;

[0034] calculating a waypoint avoidance cost function value based on a spatial overlap region of the three-dimensional space occupancy grid model and the set of path key waypoints.

[0035] Preferably, the generation of the three-dimensional obstacle avoidance heading correction vector includes:

[0036] selecting a set of candidate headings with the lowest cost according to a sorting result of the waypoint avoidance cost function value;

[0037] combining an altitude constraint condition in real-time flight attitude parameters of the UAV to perform vertical direction feasibility verification on the set of candidate headings;

[0038] selecting an optimal three-dimensional obstacle avoidance heading correction vector from the set of verified candidate headings through a Bayesian decision model.

[0039] Preferably, the conversion of the three-dimensional obstacle avoidance heading correction vector into a flight control instruction set includes:

[0040] Decompose the three-dimensional obstacle avoidance heading correction vector into pitch control quantity, roll control quantity and yaw control quantity;

[0041] According to the dynamics model of the unmanned aerial vehicle power system, the pitch control quantity is converted into a rotor speed adjustment instruction;

[0042] The roll control quantity is converted into a rudder deflection angle instruction;

[0043] The yaw control quantity is converted into a tail thruster thrust vector instruction, and a flight control instruction set is formed.

[0044] Preferably, the downward transmission to the unmanned aerial vehicle power system execution module comprises:

[0045] The flight control instruction set is transmitted to the rotor speed controller, the rudder servo controller and the tail thruster controller through the flight control bus;

[0046] The rotor speed controller adjusts the motor drive current according to the rotor speed adjustment instruction;

[0047] The rudder servo controller drives the steering gear actuator according to the rudder deflection angle instruction;

[0048] The tail thruster controller adjusts the thruster nozzle direction and fuel supply according to the thrust vector instruction.

[0049] Compared with the prior art, the beneficial effects of the present application are:

[0050] The left view and right view original data streams are synchronously acquired by binocular vision acquisition terminals, providing accurate stereo vision input for subsequent depth calculation and ensuring the integrity of environmental perception. The parallel convolutional encoding branch of the deep neural network model can efficiently extract stereo vision feature representation in binocular images, reducing feature point loss or mismatch compared with traditional feature matching methods, and improving the stability and robustness of feature extraction.

[0051] The three-dimensional space reconstruction module generates a dense depth map and an initial position coordinate set of obstacles based on the extracted stereo vision features, not only providing three-dimensional structure information of the environment, but also accurately positioning the initial position of the obstacles, laying a foundation for subsequent dynamic analysis. The dynamic obstacle analysis engine calculates the obstacle dynamic threat evaluation index combined with the real-time flight attitude parameters of the unmanned aerial vehicle, combines the motion state of the obstacle with the navigation state of the unmanned aerial vehicle itself, makes the threat evaluation more suitable for the actual flight scene, and avoids the one-sidedness of evaluating only according to the position of the obstacle.

[0052] The spatiotemporal trajectory prediction model calculates the future moving path probability distribution of the obstacle according to the dynamic threat evaluation index, and can predict the movement trend of the obstacle in advance, so as to reserve sufficient reaction time for the unmanned aerial vehicle. The three-dimensional obstacle avoidance heading correction vector is generated by fusing the future moving path probability distribution of the obstacle and the preset flight path planning data of the unmanned aerial vehicle, so that the obstacle avoidance path not only considers the dynamic change of the obstacle, but also meets the preset flight target of the unmanned aerial vehicle, and the rationality and continuity of the obstacle avoidance decision are enhanced.

[0053] The three-dimensional obstacle avoidance heading correction vector is converted into a flight control instruction set and transmitted to an execution module, realizing a closed loop from environment perception to motion control, and ensuring the real-time and accuracy of the obstacle avoidance action. Overall, the method can more comprehensively perceive the environment, more accurately evaluate the threat, and more reasonably plan the path in a complex dynamic environment, improving the reliability and adaptability of the unmanned aerial vehicle autonomous obstacle avoidance, and expanding the application possibility of the unmanned aerial vehicle in various complex scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 A timing diagram of the unmanned aerial vehicle autonomous obstacle avoidance method based on deep learning and binocular vision according to the present application;

[0055] Figure 2 A flowchart for stereo vision feature extraction;

[0056] Figure 3 A flowchart for generating a dense depth map;

[0057] Figure 4 A flowchart for path fusion and evasion cost calculation;

[0058] Figure 5 A flowchart for flight control instruction set conversion. DETAILED DESCRIPTION

[0059] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0060] Please refer to Figure 1 The present application provides an unmanned aerial vehicle autonomous obstacle avoidance method based on deep learning and binocular vision, which comprises:

[0061] The left view raw data stream and the right view raw data stream in the flight environment are synchronously captured by the binocular vision acquisition terminal installed on the unmanned aerial vehicle body.

[0062] A deep neural network model is constructed, which receives the acquired left-view raw data stream and right-view raw data stream synchronously. The model contains a left-view convolutional encoding branch and a right-view convolutional encoding branch arranged in parallel, through which the corresponding data streams are processed respectively to extract and fuse stereo vision feature representation representing the environmental geometric information.

[0063] The output stereo vision feature representation is input into a three-dimensional space reconstruction module. The module processes the stereo vision feature representation to generate a three-dimensional dense depth map of the flight environment, and based on the depth map, an initial position coordinate set of obstacles in the environment is identified and output.

[0064] A dynamic obstacle analysis engine receives the output dense depth map and the initial position coordinate set of obstacles. The engine simultaneously accesses the flight attitude parameters transmitted by the unmanned aerial vehicle flight control system in real time, and comprehensively calculates the dynamic threat evaluation index of each obstacle.

[0065] A space-time trajectory prediction model receives the output obstacle dynamic threat evaluation index. Based on the current and historical obstacle information, the model calculates the possible moving path of the obstacle within a future time window and its corresponding probability distribution.

[0066] The output obstacle future moving path probability distribution is fused with the preset navigation path planning data of the unmanned aerial vehicle task system. Through a path conflict resolution algorithm, a three-dimensional obstacle avoidance heading correction vector is calculated to guide the unmanned aerial vehicle to avoid obstacles.

[0067] A flight control instruction converter receives the generated three-dimensional obstacle avoidance heading correction vector, and analyzes and converts it into a specific flight control instruction set that can be recognized and executed by the unmanned aerial vehicle power system. The instruction set is transmitted to the unmanned aerial vehicle power system execution module through the on-board communication link.

[0068] Embodiment 1: The binocular vision acquisition terminal comprises two independent but strictly synchronized image acquisition units, respectively deployed at the left and right positions of the front end of the unmanned aerial vehicle body. Each unit adopts a global shutter CMOS image sensor with high dynamic range characteristics to adapt to complex lighting environments. The left image sensor is responsible for generating a left view raw data stream, and the right image sensor is responsible for generating a right view raw data stream. The two sensors are connected through a hardware synchronization signal line to ensure that the exposure start time and exposure time length are completely consistent, eliminating the parallax calculation error caused by different time synchronization. The sensor lens selects a fixed focal length wide-angle lens to provide sufficient horizontal and vertical field of view coverage to ensure effective capture of obstacles in front and to the sides. The image sensor output data stream contains standard RGB three-channel color information, each channel has a specific bit depth to retain rich color details. At the same time, each image sensor module is internally integrated with a non-cooled microbolometer array that can independently capture the infrared radiation information of the scene and convert it into an infrared thermal imaging data array that is aligned with the spatial resolution of the visible light image. These infrared data arrays are output synchronously with the RGB data stream to form left view and right view raw data streams containing multispectral information. The hardware interface of the binocular vision acquisition terminal transmits the raw data stream to the subsequent processing module through a high-speed serial bus.

[0069] The real-time flight attitude parameters of the UAV are continuously provided by the on-board multi-sensor fusion system. The core of this system includes a high-precision micro-electro-mechanical system (MEMS) inertial measurement unit (IMU) and a multi-frequency multi-mode global navigation satellite system (GNSS) receiver. The IMU contains a three-axis gyroscope, a three-axis accelerometer, and a three-axis magnetometer, which continuously measure the angular velocity and linear acceleration in the body coordinate system. The GNSS receiver continuously receives signals from multiple satellite constellations and calculates the absolute position of the UAV. The sensor fusion algorithm in the flight control system processes the raw data from the IMU and the GNSS receiver in real-time. This algorithm is based on the Kalman filter framework, which combines the advantages of both sensors: the high-frequency dynamic response characteristics of the IMU and the absolute positioning accuracy and low-frequency stability of the GNSS. The fusion calculation process continuously outputs a set of accurate real-time flight attitude parameters of the UAV. These parameters specifically include: the pitch angle, which describes the rotation state of the UAV body around its X-axis, reflecting the up-down inclination of the nose relative to the horizontal plane; the roll angle, which describes the rotation state of the UAV body around its Y-axis, reflecting the height difference between the left and right sides of the body relative to the horizontal plane; and the yaw angle, which describes the rotation state of the UAV body around its vertical Z-axis, indicating the horizontal orientation of the nose. The altitude parameter is obtained by fusing the barometric altitude data measured by the barometer and the ellipsoidal height data provided by the GNSS, eliminating the effects of temperature drift and barometric pressure changes when relying solely on the barometer. The ground speed vector is obtained by combining the Doppler shift velocity measurement data from the GNSS and the radial velocity data from the on-board miniature Doppler radar. The Doppler radar is particularly helpful in providing velocity estimates when satellite signals are temporarily lost. The fusion algorithm finally outputs a ground speed vector containing three-dimensional velocity components (northward velocity, eastward velocity, and upward velocity) and a combined motion direction, which describes the instantaneous motion state of the UAV relative to the ground. All these flight attitude parameters, pitch angle, roll angle, yaw angle, fused altitude, and ground speed vector containing three-dimensional velocity components and direction, are transmitted at a fixed high frequency (e.g., 100 Hz) to the dynamic obstacle analysis engine through a dedicated high-speed, low-latency data bus. The bus protocol design ensures the integrity of the data and the accuracy of the time stamp, enabling the subsequent modules to obtain attitude information that is strictly aligned in time with the visual data stream.

[0070] In the actual operation of the binocular vision acquisition terminal, the optical characteristics of the two image sensors, including the lens distortion coefficient, focal length, principal point coordinates, and the baseline distance between the sensors, are accurately calibrated before leaving the factory. The calibration parameters are stored in a non-volatile memory and used for online geometric correction in the image processing pipeline to eliminate the influence of lens distortion on the accuracy of stereo matching. The calibration process ensures the epipolar constraint relationship between the left and right views, providing a geometric basis for subsequent stereo vision feature extraction. The exposure time, gain, and other parameters of the image sensor can be dynamically adjusted by the flight control system according to the environmental lighting conditions, and a special automatic exposure control algorithm is executed to obtain image data with sufficient contrast and signal-to-noise ratio under different lighting conditions. In low light environments, the importance of the infrared thermal imaging data point array is more prominent, as it provides scene structure information that is independent of visible light. The transmission of data streams uses an efficient image compression and packaging protocol that optimizes the use of bus bandwidth while ensuring that key feature information is not lost. The physical structure of the binocular vision acquisition terminal takes into account anti-shock and temperature drift compensation design to adapt to the vibration environment and temperature changes during unmanned aerial vehicle flight, maintaining the stability of optical alignment. The attitude parameter fusion system continuously monitors the health status and signal quality indicators of each sensor, and can automatically switch the fusion strategy or provide state estimation in degraded mode when a sensor failure or signal degradation is detected, maintaining reliable perception of the unmanned aerial vehicle's attitude by the flight control system. The timestamps of the attitude parameters and the timestamps of the binocular image data stream are strictly synchronized through the system clock, usually using an accurate hardware time synchronization protocol, allowing the dynamic obstacle analysis engine to accurately associate and calculate the attitude parameters at a specific time with the corresponding visual data acquired at that time.

[0071] Example 2: see Figure 2The deep neural network model adopts a double-branch parallel architecture to process the raw data stream transmitted by the binocular vision acquisition terminal. The left-view convolutional encoding branch is based on a residual network structure design and includes five consecutive processing stages. Each stage is composed of multiple convolutional layers, batch normalization layers, and activation function layers, and includes a cross-layer connection structure. The input left-view raw data stream is first preprocessed, including pixel value normalization and spatial size adjustment. The first stage performs standard convolution operations on the adjusted left-view, outputting an initial feature map. Each subsequent stage performs a feature extraction process with decreasing spatial resolution based on the output of the previous stage. The second stage down-samples the feature map, extracting a feature representation with reduced spatial information and increased channel dimension. The third stage further reduces the spatial resolution of the feature map while increasing the number of channels to capture more abstract image features. The fourth stage maintains similar processing logic and outputs a feature tensor with medium resolution. The fifth stage, as the final processing level, outputs a deep feature representation with the lowest spatial resolution but the most channels. The outputs of the five stages collectively form a left-view multi-level feature tensor sequence, where each level's feature map has a different spatial scale and semantic abstraction level, corresponding to local details, regional structures, and global context information in the image.

[0072] The right-view convolutional encoding branch also includes a basic convolutional structure, but embeds a cross-channel attention mechanism module after each convolutional stage. The module processes as follows: a global average pooling operation is performed on the input feature map to generate a compressed channel description vector. This vector is input into a two-layer fully connected network structure with a nonlinear activation function. The first fully connected layer reduces the channel dimension, and the second layer restores to the original channel number. After processing by the Sigmoid activation function, a weight coefficient vector for each channel is generated. These weight coefficients are multiplied by the original input feature map channel by channel to enhance the response of important channels and suppress the influence of unimportant channels. The weighted feature map is then processed by the subsequent convolutional layers. Similar to the left-view branch, the right-view branch also includes five processing stages, each outputting a feature map enhanced by channel attention, ultimately forming a right-view optimized feature tensor sequence. Each processing stage of the left and right branches maintains consistent spatial resolution, facilitating subsequent cross-view information fusion.

[0073] The cross-view feature matching alignment process adopts a hierarchical processing strategy, and processes the feature maps of corresponding resolutions in the left-view multi-level feature tensor sequence and the right-view optimized feature tensor sequence in turn. For each pair of left and right feature maps with the same spatial resolution, pixel-level correlation calculation is performed. Specifically, at each spatial position of the left feature map, the dot product similarity between its feature vector and the feature vectors of all possible matching positions on the corresponding epipolar line of the right feature map is calculated. This operation is performed in the search range of the entire right-view epipolar line, generating initial correlation volume data. Different levels of feature maps are processed with different disparity search ranges. Deep low-resolution feature maps are processed with a larger disparity range, and shallow high-resolution feature maps are processed with fine disparity adjustment. These initial correlation volume data are input into a cost aggregation network. The network is composed of multiple three-dimensional convolution layers, which perform filtering and smoothing operations on the correlation volume in the spatial and disparity dimensions, aggregate context information, and suppress noise and inconsistencies. The output after cost aggregation is processed by a disparity regression layer to generate an initial disparity estimation map. Subsequently, the low-resolution disparity map is gradually upsampled to high resolution through a cascaded upsampling module, while fusing context information from high-level and low-level feature maps. Finally, all levels of disparity information are fused to form a stereo vision feature representation matrix containing rich geometric and context information. The matrix encodes the three-dimensional structure information of the scene in the form of a dense feature map.

[0074] The dynamic obstacle analysis engine receives the dense depth map and the initial position coordinate set of obstacles output by the three-dimensional space reconstruction module. The engine first analyzes the spatial distribution characteristics of the initial position coordinate set of obstacles. The algorithm calculates the number of other obstacle coordinate points within a predetermined radius neighborhood of each obstacle coordinate point. A Gaussian kernel function is used to weight the point density within the neighborhood, with closer points having higher weights and farther points having lower weights. The statistical distribution of the neighborhood density values of all obstacle points is calculated to identify high-density aggregation areas. The engine also accesses real-time transmitted UAV flight attitude parameters, particularly ground speed vector information containing three-dimensional velocity components and motion direction. Based on the dense depth map, the engine calculates the normal vector direction of each obstacle surface using a surface normal estimation algorithm. This algorithm estimates the local surface direction by least squares plane fitting within each pixel neighborhood of the depth map, and then obtains the surface normal vector. The engine calculates the spatial direction angle between the UAV ground speed vector direction and the obstacle surface normal vector. This angle calculation uses the vector dot product formula, resulting in a cosine value. A preconfigured collision risk level mapping table is stored in the non-volatile memory of the engine. This mapping table is a two-dimensional lookup table, with one dimension being the segmented interval of obstacle spatial distribution density values and the other dimension being the segmented interval of cosine values of the angle between the ground speed vector and the surface normal vector. Each cell stores the corresponding threat level quantization value. The engine locates the corresponding cell region in the mapping table according to the spatial distribution density value and the angle cosine value calculated for the current obstacle point. If the calculated value is exactly at the center of the cell, the threat level value of that cell is directly read; if the calculated value is between cells, the threat level values of adjacent cells are bilinearly interpolated. The engine independently calculates and outputs a scalar value for each obstacle point, which is the dynamic threat evaluation index of the obstacle. When this index is transmitted to the subsequent module, it is accompanied by the position identifier and timestamp information of the corresponding obstacle. The timing synchronization mechanism inside the engine ensures that the input depth map data and attitude parameter data are strictly aligned in time stamp, ensuring that all calculations are based on the perception state at the same time.

[0075] Example 3: see Figure 3The three-dimensional space reconstruction module receives a stereo vision feature representation matrix from the deep neural network model, which contains the encoded scene geometry and texture information. The module first processes the input features through a set of separable convolution layers. These convolution layers contain 1x1 convolution kernels that act to decouple the mixed features. One convolution path is dedicated to extracting disparity-related geometry features, forming a disparity feature channel; the other path focuses on the structured texture information of the image, forming a texture feature channel. The processing of the disparity feature channel uses a spatial gridding algorithm: the continuous disparity value domain is divided into fine discrete grid cells, each representing a specific disparity interval range. The algorithm constructs a probability distribution model in the disparity dimension. For each pixel position on the image plane, the likelihood of belonging to different disparity grid cells is calculated. This calculation uses a local window feature similarity measure combined with spatial constraint relationships of neighboring pixels. After generating the initial disparity probability distribution, a sub-pixel level interpolation technique is used for processing. The bicubic interpolation method is applied to the disparity probability distribution field, smoothing the probability distribution curve and improving the spatial resolution of the depth estimation, resulting in an initial depth probability distribution field. This distribution field reflects the preliminary probability of each point in the scene being at different depth values.

[0076] The processing of the texture feature channel is independent: the direction gradient operator is applied to scan the entire feature map, calculating the intensity change rate of each pixel position in the horizontal and vertical directions. By integrating the gradient components in these two directions, the gradient amplitude of each pixel point is calculated, generating a gradient amplitude map. The gradient amplitude map reveals the location information of object edges and texture-rich areas in the scene. Based on the gradient amplitude map, an adaptive filtering kernel is constructed. The center weight of this filtering kernel is inversely proportional to the gradient amplitude of the current position. This means that in edge areas (high gradient amplitude), the filtering kernel weight is small to protect the edge sharpness; in flat areas (low gradient amplitude), the filtering kernel weight is large to enhance the smoothing effect. Apply this adaptive filtering kernel to the initial depth probability distribution field and perform convolution operations. This operation can be represented mathematically as:

[0077]

[0078] where, represents the depth probability value of depth d at image coordinates (x, y) after optimization; represents the probability value of depth d at coordinates (x+i, y+j) in the initial depth probability distribution field; represents the weight value of the adaptive filtering kernel centered at (x, y) at offset (i, j); this weight value is determined by the gradient amplitude at coordinates (x, y) ; the summation range is within the neighborhood window defined by the filtering kernel size (e.g., The filter process significantly suppresses the noise points in the initial depth probability distribution field caused by texture deficiency or matching errors, while preserving the depth discontinuities near object boundaries. After multiple iterations of optimization, the module outputs a dense depth map with high confidence, where each pixel is associated with a specific depth value.

[0079] The spatio-temporal trajectory prediction module takes the sequence of dense depth maps as input source. The module continuously monitors the connected regions in the depth maps that are identified as obstacles. For each tracked obstacle, the module records its center point 3D coordinates in consecutive time frames. These coordinate points are connected in time order to form a sequence of historical movement trajectory segments of the obstacle. Each trajectory segment contains the position change data of the obstacle within a fixed time window (e.g. past 1 second, corresponding to 10 depth maps). A gated recurrent unit network is employed to model the temporal dynamics of these trajectory segments. The network input is the coordinate increment sequence (i.e. , representing the displacement of the obstacle in 3D space from time t-1 to t). The gated recurrent unit network contains two layers of hidden structure. The first layer of units processes the input displacement sequence, learning the instantaneous velocity variation pattern of the obstacle. The second layer of units receives the output of the first layer, learning the motion trend and pattern features in a longer time span. The gating mechanism (update gate and reset gate) inside the units dynamically regulates the retention and forgetting ratio of historical state information. The network finally outputs a fixed dimension state vector, which serves as the motion state latent variable representation of the obstacle, encoding the current motion characteristics (such as velocity, acceleration, direction trend) of the obstacle and its historical motion pattern.

[0080] The conditional random field model receives the above obstacle motion state latent variable representation as observation evidence. The model constructs a probabilistic graphical model for a sequence of discrete time points in the future (e.g. predicting 2 seconds into the future, with a step size of 0.1 seconds, for a total of 20 time nodes). Each time node corresponds to a possible 3D position state of the obstacle at a certain future time. The model defines two core components: a single node potential function and a transition potential function between adjacent nodes. The single node potential function is computed based on the obstacle motion state latent variable representation, and describes the likelihood of the obstacle being at a certain position at each future time node. Specifically, a fully connected neural network is used to map the motion state latent variable to a probabilistic distribution prior estimate of the position at each future time node. The transition potential function defines the probability of state transition between consecutive time nodes, modeling the continuity and inertia of the obstacle motion. The transition potential function generally prefers state transitions with small changes in position between adjacent time instants. The model employs a belief propagation algorithm for probabilistic inference. This algorithm iteratively propagates messages on the probabilistic graphical model, from the observation nodes (motion state latent variable) to the future time nodes, and between adjacent future nodes. After multiple rounds of message propagation iterations, the probabilistic distribution of the obstacle position state at each future time node gradually converges. Finally, the model integrates the position probability distributions at all future time nodes to generate a four-dimensional probability distribution heat map that stacks the spatial dimensions with the time dimension. In this heat map, the brightness value of each voxel element (containing position coordinates x, y, z) at a certain future time point t represents the probability of the obstacle being at that spatial position at that time. This heat map is the quantitative output of the obstacle future movement path probability distribution.

[0081] Embodiment 4: refer to Figure 4 The obstacle future movement path probability distribution heat map is a four-dimensional data structure, containing spatial three-dimensional coordinates (X, Y, Z) and time dimension T. The conversion module discretizes the heat map into a three-dimensional spatial occupancy grid model: set the spatial voxel resolution to 0.2 meters (i.e. each cubic grid has a side length of 0.2 meters), and the time step to 0.2 seconds. For each future time slice (e.g. t = 0.2s, 0.4s,..., 2.0s), iterate through all spatial grid elements. If the probability value of the corresponding spatial position in the heat map at time t exceeds the threshold value 0.3, mark the grid as "potentially occupied" and store its occupancy probability value. Finally, a set of three-dimensional grid maps arranged in time sequence is formed.

[0082] The UAV pre-set flight path planning data is defined using a sequence of waypoints. Each waypoint contains a three-dimensional coordinate and an estimated arrival timestamp. The module identifies the path turning points and mission critical points to form a set of key waypoints. Table 1 shows some of the key waypoint attributes in an example scenario.

[0083] Table 1: shows some of the key waypoint attributes in an example scenario

[0084] Waypoint ID Coordinate X (m) Coordinate Y (m) Coordinate Z (m) Predicted Time of Arrival (s) WP1 120.5 85.2 50.0 3.2 WP2 135.7 93.8 52.1 4.1 WP3 148.9 110.5 55.3 5.0 WP4 160.2 125.7 58.0 6.0 WP5 172.8 140.3 60.0 7.2

[0085] The waypoint avoidance cost calculation process is as follows: for each point in the set of key waypoints (such as WP3), determine its time influence window (for example, the expected arrival time ± 1 second, that is, t = 4.0-6.0 seconds). Within this time window, extract all corresponding spatial occupancy grid slices. Calculate the Euclidean distance between the waypoint coordinates and all grid centers marked as "potential occupancy". Record the minimum distance value . The cost function is defined as a piecewise function:

[0086] When the distance is less than 1.5 meters, the cost value is 0;

[0087] When the distance is greater than 1.5 meters, the cost value is ; ;

[0088] When the distance is greater than 2.5 meters, the cost value is .

[0089] Therefore, in the example, if the distance of WP3 to the nearest obstacle at t = 5.0 seconds is 1.5 meters, the avoidance cost value is .

[0090] The three-dimensional obstacle avoidance heading correction vector generation process includes four stages: candidate heading sampling: taking the current heading as the reference, uniformly sampling the heading angle combination (yaw angle , pitch angle ) within the range of ± 30 degrees in the horizontal plane and ± 15 degrees in the vertical plane at 5-degree intervals, generating an initial set of candidate headings.

[0091] Trajectory simulation and cost summation: for each candidate heading (such as , ), based on the current speed dynamics model of the UAV, simulate its flight trajectory and predict the actual time through each key waypoint. Calculate the total avoidance cost function value of all waypoints under this trajectory.

[0092] Height constraint verification: check whether the predicted flight height corresponding to the candidate heading is within the safety boundary (such as the minimum safe height of 40 meters and the maximum height limit of 80 meters). Remove the headings that result in a height exceeding the range of [40, 80] meters.

[0093] Bayesian decision screening: construct a decision model input feature vector - including the total cost value of the candidate heading, the angle deviation from the preset path, and the height change rate. Based on the likelihood probability model trained on the historical flight data set, evaluate the feasibility probability of each candidate heading. Select the heading with the highest probability as the final three-dimensional obstacle avoidance heading correction vector. ​​​

[0094] Specific example: During a flight, a moving vehicle obstacle is detected, and its area has a high probability of covering the vicinity of waypoint WP3 within the next 2 seconds. Original heading ( , A straight path results in a WP3 avoidance cost of 10. Among candidate routes:

[0095] Heading A ( , Total cost: 3.5, height: 55 meters, angular deviation: medium.

[0096] Heading B ( , Total cost: 1.2, height: 38 meters (height constraint violated)

[0097] Heading C ( , Total cost: 0.8, height: 53 meters, angular deviation: small.

[0098] The Bayesian model calculates the feasibility probability of heading C to be 0.92 (the highest), therefore, a correction vector is output. , This vector guides the drone to veer to the upper right, avoiding areas with dense obstacles while maintaining a safe altitude.

[0099] After receiving the vector, the flight control command converter decomposes it according to the body coordinate system:

[0100] Yaw correction The tail thruster nozzle is shifted 15° to the left, increasing thrust by 12%.

[0101] Pitch correction The conversion reduces the front rotor speed by 5% and increases the rear rotor speed by 5%.

[0102] The specific execution instructions are generated and transmitted via the control bus. The entire process is completed within 200 milliseconds, enabling dynamic obstacle avoidance path replanning.

[0103] Example 5: See Figure 5, the three-dimensional obstacle avoidance heading correction vector contains three independent component parameters: the pitch control component represents the angular increment of the body rotating around the lateral axis, the roll control component represents the angular increment of the body rotating around the longitudinal axis, and the yaw control component represents the angular increment of the body rotating around the vertical axis. After receiving the vector, the flight control command converter starts a multi-channel parallel processing mechanism. For the pitch control component, the converter calls a pre-installed rotor dynamics model. The model establishes a mapping relationship between the change in the pitch angle and the lift difference between the rotor groups. The model calculates the size of the pitch moment required to maintain the target pitch attitude, combines the current flight speed and air density parameters, and deduces the lift difference value between the left and right symmetrical rotor groups. According to the rotor aerodynamic characteristic database, the lift difference is converted into specific rotor speed adjustment instruction values. The instruction clearly specifies the target speed adjustment value of the left main rotor group and the target speed adjustment value of the right main rotor group, and the two have opposite signs to form differential control. The speed adjustment amount is expressed in absolute speed value or relative percentage.

[0104] The roll control component is input to the control surface aerodynamic effect model processing. The model stores the corresponding relationship between the control surface deflection angle and the roll moment coefficient under different flight states. The model performs real-time interpolation calculation according to the current airspeed and angle of attack parameters to determine the control surface deflection angle required to generate the target roll moment. For unmanned aerial vehicles using aileron control, output the left and right aileron differential deflection instructions; for multi-rotor configurations, output the differential lift adjustment instructions for specific rotor groups. The instruction clearly specifies the target deflection angle of the left control surface and the target deflection angle of the right control surface, which form asymmetric deflection. The deflection angle is expressed in mechanical angle value, accurate to 0.1 degrees.

[0105] The yaw control component is processed by the thruster vector control model. The model contains two parallel calculation units: a thrust size calculation unit and a nozzle direction calculation unit. The thrust size calculation unit calculates the absolute value of the thrust that needs to be adjusted according to the target yaw moment and the current flight height, combined with the thruster thrust-fuel consumption characteristic curve. The nozzle direction calculation unit calculates the nozzle vector angle that generates the target yaw moment based on the spatial geometric relationship of the thruster installation position. The model outputs two parts: the thrust adjustment instruction value and the nozzle deflection instruction value. The thrust adjustment instruction specifies the target thrust value of the thruster or the throttle opening percentage; the nozzle deflection instruction specifies the target deflection direction and angle value. After all control components are converted, the command converter encapsulates the rotor speed adjustment instruction, the control surface deflection angle instruction, and the thruster thrust vector instruction into a complete flight control command set data packet according to the standard protocol.

[0106] The instruction set is transmitted to the execution module through a high-speed flight control bus. The bus adopts a time-triggered communication protocol to ensure the real-time and determinacy of instruction transmission. The rotor speed controller receives the rotor speed adjustment instruction as a bus node. The controller has a built-in speed closed-loop control algorithm: the actual rotor speed is collected in real time through a Hall sensor, compared with the target value to generate a speed error signal. The error signal is input into a proportional-integral controller, and the output is the motor drive current adjustment. The current adjustment signal is amplified through the power drive module, and then drives the three-phase winding of the brushless motor to realize accurate tracking of the rotor speed. The control of the control surface servo controller receives the control surface deflection angle instruction. The controller obtains the actual position of the control surface through a high-precision potentiometer or optical encoder, and the position deviation is formed with the instruction angle. After the position deviation is processed by the servo control law, a pulse width modulation signal is generated to drive the steering gear actuator. The steering gear drives the control surface to rotate to the target angle through a gear transmission mechanism, and the position feedback forms a closed-loop control. The tail thruster controller processes the thrust component and direction component in the thrust vector instruction. The thrust control unit adjusts the propellant supply flow through the fuel metering valve according to the thrust adjustment instruction to realize accurate control of the thrust size. The direction control unit drives the stepper motor or hydraulic actuator according to the nozzle deflection instruction to adjust the vector direction of the thruster nozzle. The nozzle angle is detected in real time by the rotary encoder and fed back to form an angle closed-loop control.

[0107] Each execution controller continuously monitors the execution state during instruction execution. The rotor speed controller feeds back the actual speed, motor current and temperature parameters; the control surface servo controller feeds back the actual angle of the control surface and the load torque of the steering gear; the thruster controller feeds back the actual thrust value, nozzle angle and fuel flow. These state parameters are transmitted back to the flight control instruction converter in real time through the feedback channel. The converter compares the actual execution state with the original instruction to verify whether the execution deviation exceeds the tolerance range, and if so, the instruction compensation mechanism or fault handling program is started. The entire instruction conversion and execution process is completed within strict time constraints to ensure the real-time responsiveness of the obstacle avoidance control. The execution state data is recorded in the flight data recorder for subsequent analysis and maintenance. All controllers have fault detection function, which can automatically switch to the degraded control mode and send an alarm signal in the case of sensor failure, actuator jamming and other abnormal conditions, to maintain the basic controllability of the unmanned aerial vehicle.

[0108] It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting; it is not intended to exclude myriad other embodiments of the present application that other inventors can develop based on the same general inventive concepts embodied by the described embodiments. That is, although the present application is described in terms of particular embodiments and illustrative figures, it should be apparent that the scope of the present application is not limited to these specific embodiments.

[0109] While the embodiments of the application have been shown and described herein, it will be understood by those of ordinary skill in the art that various changes, modifications, alternatives, and variations can be made to the embodiments without departing from the spirit and scope of the application, which is defined by the appended claims and their equivalents.

Claims

1. A method for autonomous obstacle avoidance of a UAV based on deep learning and binocular vision, characterized in that, Includes the following steps: The binocular vision acquisition terminal simultaneously acquires raw data streams of the left and right views in the flight environment; The deep neural network model simultaneously processes the raw data streams of the left and right views, and extracts stereo vision feature representations from the binocular images through parallel convolutional coding branches; The stereo vision feature representation is input into the three-dimensional space reconstruction module to generate a dense depth map of the flight environment and a set of initial position coordinates of obstacles. The dynamic obstacle analysis engine receives the dense depth map and the set of initial position coordinates of the obstacles, and calculates the dynamic threat assessment index of the obstacles by combining the real-time flight attitude parameters of the UAV. Based on the dynamic threat assessment index of obstacles, the probability distribution of the future movement path of obstacles is calculated through a spatiotemporal trajectory prediction model. By integrating the probability distribution of the future movement path of the obstacle with the UAV's preset flight path planning data, a three-dimensional obstacle avoidance heading correction vector is generated. The three-dimensional obstacle avoidance heading correction vector is converted into a flight control command set and transmitted to the UAV power system execution module; The extraction of stereo vision feature representation from binocular images through parallel convolutional coding branches includes: Perform a multi-scale feature pyramid convolution operation on the original data stream of the left view to generate a multi-level feature tensor sequence of the left view; Perform cross-channel attention mechanism feature enhancement processing on the original data stream of the right view to generate an optimized feature tensor sequence for the right view; Cross-view feature matching and alignment are performed between the multi-level feature tensor sequence of the left view and the optimized feature tensor sequence of the right view to form a stereo vision feature representation matrix that fuses binocular parallax. The dynamic obstacle analysis engine receives the dense depth map and the initial position coordinates of the obstacles, and calculates dynamic threat assessment indicators for the obstacles by combining them with the real-time flight attitude parameters of the UAV, including: Analyze the spatial distribution density of the initial position coordinates of obstacles in a dense depth map; The angle between the ground velocity vector direction and the obstacle surface normal vector in the real-time flight attitude parameters of the associated UAV; Based on the spatial distribution density of obstacles and the angle between surface normal vectors, dynamic threat assessment indicators for obstacles are generated in conjunction with a preset collision risk level mapping table.

2. The method of claim 1, wherein the method further comprises: The raw data streams of the left and right views contain RGB color channel information and infrared thermal imaging data, and the real-time flight attitude parameters of the UAV include pitch angle, roll angle, yaw angle, altitude and ground speed vector.

3. The method of claim 1, wherein the method further comprises: The process of generating a dense depth map of the flight environment using the 3D spatial reconstruction module includes: The stereo vision feature representation matrix is ​​decomposed into disparity feature channels and texture feature channels; Sub-pixel level interpolation calculations are performed on the disparity feature channels using a spatial rasterization algorithm to construct an initial depth probability distribution field. The edge gradient information in the fused texture feature channels is used to perform noise filtering optimization on the initial depth probability distribution field, and a dense depth map is output.

4. The method of claim 3, wherein the method further comprises: The stereoscopic vision feature representation is input into the three-dimensional space reconstruction module to generate a dense depth map of the flight environment, including: Extracting sequences of historical movement trajectories of obstacles from dense depth maps; A gated recurrent unit network is used to perform time-series modeling of the historical movement trajectory segments of obstacles, generating a latent variable representation of the obstacle's motion state. The obstacle motion state hidden variable represents an input conditional random field model, and predicts a future moving path probability distribution heat map of the obstacle.

5. The method of claim 4, wherein, The fusion of the future moving path probability distribution of the obstacle and the preset flight path planning data of the unmanned aerial vehicle includes: Converting the future moving path probability distribution heat map of the obstacle into a three-dimensional space occupancy grid model; Labeling a path key waypoint set in the preset flight path planning data of the unmanned aerial vehicle; Based on the spatial overlap area of the three-dimensional space occupancy grid model and the path key waypoint set, a waypoint avoidance cost function value is calculated.

6. The method of claim 5, wherein the method further comprises: The generation of the three-dimensional obstacle avoidance heading correction vector includes: According to the sorting result of the waypoint avoidance cost function value, a set of candidate headings with the lowest cost is selected; Combining the altitude constraint condition in the real-time flight attitude parameters of the unmanned aerial vehicle, the set of candidate headings is verified for vertical direction feasibility; Through a Bayesian decision model, an optimal three-dimensional obstacle avoidance heading correction vector is selected from the set of verified candidate headings.

7. The method of claim 6, wherein the method further comprises: The conversion of the three-dimensional obstacle avoidance heading correction vector into a flight control instruction set includes: Decomposing the three-dimensional obstacle avoidance heading correction vector into a pitch control amount, a roll control amount and a yaw control amount; According to a dynamic model of the power system of the unmanned aerial vehicle, the pitch control amount is converted into a rotor speed adjustment instruction; The roll control amount is converted into a rudder deflection angle instruction; The yaw control amount is converted into a tail thruster thrust vector instruction, and the flight control instruction set is formed by combination.

8. The method of claim 7, wherein, The transmission to the unmanned aerial vehicle power system execution module includes: Through a flight control bus, the flight control instruction set is transmitted to a rotor speed controller, a rudder servo controller and a tail thruster controller; The rotor speed controller adjusts the motor driving current according to the rotor speed adjustment instruction; The rudder servo controller drives the rudder actuator according to the rudder deflection angle instruction; The tail thruster controller adjusts the thruster nozzle direction and fuel supply amount according to the thrust vector instruction.

Images