Unmanned aerial vehicle intelligent obstacle avoidance method and system based on AI vision

Abstract

The invention discloses an unmanned aerial vehicle intelligent obstacle avoidance method and system based on AI vision, and belongs to the technical field of intelligent obstacle avoidance, and the method specifically comprises the steps that an unmanned aerial vehicle obtains angular velocity and linear velocity data in the flight process in real time through an airborne inertial measurement unit; calculating a displacement vector of the image sensor in the exposure time according to the angular velocity and linear velocity data, and constructing a motion blur transfer function; performing deconvolution operation on an original image acquired by an image sensor by applying a motion blur transfer function, and outputting a sharp image; based on the amplitude of the displacement vector, the frame rate controller dynamically adjusts the image capture frequency of the image sensor; the embedded processor deploys a lightweight convolutional neural network model, and the model receives the sharp image and performs obstacle feature extraction and classification; and the obstacle avoidance decision module fuses the processing result of the sharp image and the frame rate control signal to generate an unmanned aerial vehicle course adjustment instruction.

Description

Technical Field

[0001] This invention relates to the field of intelligent obstacle avoidance technology, specifically to an AI vision-based intelligent obstacle avoidance method and system for unmanned aerial vehicles (UAVs). Background Technology

[0002] The rapid development of drone technology has expanded its application scenarios from consumer-grade aerial photography to multiple fields such as industrial inspection, logistics delivery, and emergency rescue. As a core module ensuring drone flight safety, obstacle avoidance systems have undergone continuous technological iteration. Early drone obstacle avoidance relied heavily on traditional sensing methods such as ultrasonic sensors and lidar. While these methods can detect obstacles at close range, they suffer from limitations such as limited sensing range and susceptibility to environmental interference. In recent years, AI vision-based obstacle avoidance technology has gradually become mainstream due to its advantages of wide sensing range and rich information dimensions. Its core involves using onboard image sensors to collect environmental images, employing AI models such as convolutional neural networks to extract features, classify, and locate obstacles in the images, and then having the obstacle avoidance decision module generate heading adjustment commands to achieve autonomous obstacle avoidance.

[0003] During high-speed maneuvering of drones, obstacle avoidance systems based on AI vision face challenges such as obstacle detection delays and misidentifications caused by motion blur and insufficient image frame rates. Motion blur degrades image quality, making it difficult for AI vision models to extract clear visual features, thus reducing the accuracy of obstacle classification and localization. Simultaneously, the demand for high frame rate image processing conflicts with the limited computing resources of the drone's onboard embedded system, further exacerbating the detection process delay. Such delays directly lead to untimely obstacle avoidance responses, increasing the collision risk of drones in dynamic environments and limiting their reliable deployment in high-speed application scenarios. How to solve the motion blur and frame rate adaptation problems while ensuring detection accuracy has become a key bottleneck in the development of AI vision-based intelligent obstacle avoidance technology for drones. Summary of the Invention

[0004] The purpose of this invention is to provide an AI vision-based intelligent obstacle avoidance method and system for drones, thereby solving the problems in the background technology.

[0005] The objective of this invention can be achieved through the following technical solutions: An AI vision-based intelligent obstacle avoidance method for drones includes: S1. The UAV acquires angular velocity and linear velocity data in real time during flight through its onboard inertial measurement unit. S2. Based on the angular velocity and linear velocity data, calculate the displacement vector of the image sensor during the exposure time and construct the motion blur transfer function; S3. Using the motion blur transfer function, perform deconvolution operation on the original image acquired by the image sensor to output a sharpened image; S4. Based on the magnitude of the displacement vector, the frame rate controller dynamically adjusts the image capture frequency of the image sensor; S5: The embedded processor deploys a lightweight convolutional neural network model that receives sharpened images and performs obstacle feature extraction and classification. S6, the obstacle avoidance decision module, integrates the processing results of the sharpened image with the frame rate control signal to generate UAV heading adjustment instructions.

[0006] As a further aspect of the present invention: in step S2, the process of calculating the displacement vector of the image sensor during the exposure time based on the angular velocity and linear velocity data, and constructing the motion blur transfer function, is as follows: By combining angular velocity data with the calibration parameters of the image sensor, the pixel offset in the image plane caused by the rotation of the UAV during the exposure time is calculated; by multiplying the linear velocity data by the exposure time and combining it with the depth estimation parameters, the pixel displacement in the image plane caused by the translation of the UAV during the exposure time is calculated. The pixel offset caused by rotation and the pixel displacement caused by translation are merged to form a composite displacement vector, and the corresponding motion blur transfer function is constructed based on the direction and length of the vector.

[0007] As a further aspect of the present invention: the specific method for calculating the image plane pixel offset caused by the rotation of the drone during the exposure time is as follows: Based on the fixed mounting attitude matrix of the camera relative to the UAV body, the angular velocity data in the body coordinate system is transformed into the camera coordinate system; Using the intrinsic calibration matrix of the image sensor, the angular velocity in the camera coordinate system is converted into the instantaneous motion velocity in the image plane coordinate system; Multiplying the instantaneous motion velocity on the image plane by the exposure time of the image sensor yields the pixel offset of the image plane caused by rotation.

[0008] As a further aspect of the present invention: the specific method for calculating the image plane pixel displacement caused by the drone's translation during the exposure time by multiplying the linear velocity data by the exposure time and combining it with the depth estimation parameters is as follows: Based on the fixed mounting attitude matrix of the camera relative to the UAV body, the linear velocity data in the body coordinate system is transformed to the camera coordinate system; Multiply the linear velocity in the camera coordinate system by the exposure time of the image sensor to obtain the actual translational displacement of the camera in three-dimensional space during the exposure time. Using the focal length parameters of the image sensor and the depth estimation parameters of the scene, the actual translational displacement in three-dimensional space is converted into pixel displacement on the image plane.

[0009] As a further aspect of the present invention: the specific method for constructing the corresponding motion fuzzy transfer function based on the direction and length of the vector is as follows: The pixel offset caused by rotation and the pixel displacement caused by translation are uniformly transformed into the two-dimensional pixel coordinate system of the image plane; In a two-dimensional pixel coordinate system, the two pixel offset vectors are added together to obtain the composite displacement vector. Based on the direction of the synthesized displacement vector, the angle parameter of the fuzzy kernel in the motion fuzzy transfer function is determined; based on the length of the synthesized displacement vector, the length parameter of the fuzzy kernel in the motion fuzzy transfer function is determined, thereby constructing the motion fuzzy transfer function.

[0010] As a further aspect of the present invention: in step S3, the process of applying the motion blur transfer function to perform deconvolution operation on the original image acquired by the image sensor and outputting a sharpened image is as follows: Based on the direction parameter of the motion blur transfer function, the original image is segmented into multiple image regions with consistent motion blur characteristics; For each image region with motion blur characteristics, the corresponding deconvolution kernel function is calculated; using the deconvolution kernel function of each region, frequency domain deconvolution operation is performed on the image data of that region. The deconvolution results of all regions are merged to generate a complete, sharpened image.

[0011] As a further aspect of the present invention: in step S4, the process of dynamically adjusting the image capture frequency of the image sensor based on the magnitude of the displacement vector by the frame rate controller is as follows: The frame rate controller receives displacement vector amplitude data from the displacement vector calculation module in real time, which characterizes the degree of motion of the image sensor during the exposure time. The frame rate controller performs sliding window averaging filtering on the displacement vector amplitude data to eliminate instantaneous fluctuations and output a smooth displacement amplitude. The frame rate controller calculates the rate of change of the smooth displacement amplitude, dynamically updates the internal threshold based on the rate of change, and compares the smooth displacement amplitude with the threshold. The frame rate controller generates an image capture frequency control signal based on the comparison results, which directly instructs the image sensor to adjust its image capture frequency.

[0012] As a further aspect of the present invention: In step S5, the embedded processor deploys a lightweight convolutional neural network model, which receives the sharpened image and performs obstacle feature extraction and classification as follows: The first branch of the convolutional neural network extracts multi-scale spatial features from the sharpened input image, while the second branch performs inter-frame difference analysis on the image to generate motion feature maps. The feature fusion layer of the network performs channel splicing and weighted fusion of multi-scale spatial features and motion feature maps to form a comprehensive feature representation; The network's classifier outputs an obstacle category prediction based on a comprehensive feature representation, and this classifier adaptively adjusts its classification confidence threshold according to the image capture frequency.

[0013] As a further aspect of the present invention: In step S6, the process by which the obstacle avoidance decision module integrates the processing result of the sharpened image with the frame rate control signal to generate the UAV heading adjustment command is as follows: The obstacle avoidance decision module synchronously receives obstacle category and location information from the convolutional neural network model, as well as the current image capture frequency status signal from the frame rate controller; The module's internal preset decision rule base selects the corresponding risk assessment strategy based on the image capture frequency and maps obstacle location information to threat level; The module computing unit performs a weighted fusion of the obstacle threat level and the classification confidence of the convolutional neural network output based on the selected risk assessment strategy to generate a comprehensive threat assessment. Based on the comprehensive threat assessment, the command generation unit queries the preset heading command mapping table and outputs specific heading adjustment angle and direction commands for the UAV.

[0014] An AI vision-based intelligent obstacle avoidance system for drones includes: An inertial measurement module is used by UAVs to acquire angular velocity and linear velocity data in real time during flight via an onboard inertial measurement unit. The motion estimation module is used to calculate the displacement vector of the image sensor during the exposure time based on angular velocity and linear velocity data, and to construct the motion blur transfer function. The image restoration module is used to perform deconvolution operations on the original images acquired by the image sensor using the motion blur transfer function, and output a sharpened image. The frame rate adjustment module is used to dynamically adjust the image capture frequency of the image sensor based on the magnitude of the displacement vector. The obstacle detection module is used to deploy a lightweight convolutional neural network model that receives sharpened images and performs obstacle feature extraction and classification. The obstacle avoidance decision module is used to fuse the processing results of the sharpened image with the frame rate control signal to generate UAV heading adjustment commands.

[0015] The beneficial effects of this invention are: This invention effectively overcomes the interference of motion blur caused by high-speed flight on visual perception by integrating inertial measurement data and image processing. Specifically, based on the real-time angular and linear velocities of the UAV, the comprehensive displacement vector of the image sensor during exposure is accurately calculated, and a precise motion blur degradation model is constructed accordingly. Subsequently, this model is used to perform targeted deconvolution restoration processing on the acquired blurred images. This process can significantly restore the details and edge information that were obscured by blur in the image, providing AI vision models with higher quality and clearer feature input images. This directly improves the efficiency and stability of obstacle visual feature extraction, thereby ensuring the accuracy of obstacle classification and localization at the source and reducing the risk of misidentification caused by image degradation.

[0016] This invention intelligently resolves the contradiction between limited processing resources and real-time information requirements by establishing a closed-loop linkage mechanism between motion state and image acquisition frequency. The technical solution dynamically adjusts the image sensor's capture frame rate based on the displacement vector amplitude, which characterizes the intensity of motion. A lower frame rate is used during stable flight to conserve computing resources, while the frame rate is automatically increased during high-speed maneuvers to obtain more continuous environmental snapshots. This adaptive mechanism allows limited computing resources to be concentrated on processing high-frame-rate image streams during critical periods, avoiding processing bottlenecks and overall latency. Simultaneously, the obstacle avoidance decision module integrates frame rate status signals to assess the timeliness of perceived information, making the final heading adjustment command more aligned with the current dynamic situation. This collectively ensures the system's timely response and reliable decision-making under complex flight conditions, significantly reducing the probability of collisions in dynamic environments. Attached Figure Description

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] Figure 1 This is a flowchart illustrating an AI vision-based intelligent obstacle avoidance method for drones according to the present invention.

[0019] Figure 2 This is a schematic diagram of a module of an AI vision-based intelligent obstacle avoidance system for unmanned aerial vehicles according to the present invention. Detailed Implementation

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1As shown, this invention is an intelligent obstacle avoidance method for drones based on AI vision, comprising the following steps: S1. The UAV acquires angular velocity and linear velocity data in real time during flight through its onboard inertial measurement unit. S2. Based on the angular velocity and linear velocity data, calculate the displacement vector of the image sensor during the exposure time and construct the motion blur transfer function; S3. Using the motion blur transfer function, perform deconvolution operation on the original image acquired by the image sensor to output a sharpened image; S4. Based on the magnitude of the displacement vector, the frame rate controller dynamically adjusts the image capture frequency of the image sensor; S5: The embedded processor deploys a lightweight convolutional neural network model that receives sharpened images and performs obstacle feature extraction and classification. S6, the obstacle avoidance decision module, integrates the processing results of the sharpened image with the frame rate control signal to generate UAV heading adjustment instructions.

[0022] In one embodiment of the present invention, the process in step S1 where the UAV acquires angular velocity and linear velocity data in real time during flight via an onboard inertial measurement unit is as follows: During drone flight, the onboard inertial measurement unit (IMU) continuously operates to provide motion status data. This unit contains gyroscope sensors, fixedly mounted on the drone's fuselage, which sense rotational motion along the X, Y, and Z axes of the drone's coordinate system. When the drone rotates around any axis, the movable element inside the gyroscope sensor displaces due to the Coriolis effect; this displacement is converted into a continuously changing electrical signal, which corresponds precisely to the instantaneous rotational rate of the drone around each axis, thus generating raw angular velocity data reflecting the rotational rates along the three axes. An accelerometer sensor, operating in parallel with the gyroscope sensor, is also mounted along the three axes of the drone's coordinate system. The accelerometer sensor works by detecting the inertial force generated by the drone's variable-speed motion on its internal mass block; when the drone accelerates or decelerates in any axis, the inertial force on the mass block changes, and this force is converted into a corresponding electrical signal by the sensing element. This electrical signal directly reflects the acceleration force experienced by the drone in each axis, excluding gravity, thus generating raw linear acceleration data.

[0023] The inertial measurement unit (IMU) has a dedicated processing circuit built into it, which coordinates and processes the outputs of the aforementioned sensors. This circuit uses a synchronized clock signal to acquire raw angular velocity data from the gyroscope sensor and raw linear acceleration data from the accelerometer sensor at the same time, ensuring that both sets of data have the same timestamp. The acquired raw data is first sent to the signal conditioning module for amplification and preliminary filtering. Subsequently, the core processor of the processing circuit performs calculations on these data; the calculation process includes calibration and compensation of the raw angular velocity data, coordinate transformation and gravity component removal of the raw linear acceleration data, and finally, real-time output of fused angular velocity and linear velocity information that can be used for control.

[0024] In one embodiment of the present invention, step S2, which involves calculating the displacement vector of the image sensor during the exposure time based on the angular velocity and linear velocity data, and constructing the motion blur transfer function, is as follows: By combining angular velocity data with the calibration parameters of the image sensor, the pixel offset in the image plane caused by the drone's rotation during the exposure time is calculated; specifically: During UAV flight, to accurately analyze image blur caused by aircraft rotation, it is necessary to first convert the angular velocity data sensed by the inertial measurement unit (IMU) to the coordinate system of the image sensor. A fixed spatial mounting relationship exists between the UAV's body coordinate system and the camera coordinate system, described by a pre-calibrated three-dimensional rotation matrix. This matrix represents the camera's fixed mounting attitude matrix relative to the UAV body. The processing unit reads the current angular velocity data, which is a three-dimensional vector expressed in the body coordinate system. By performing matrix multiplication on this angular velocity vector and the mounting attitude matrix, it can be transformed from the body coordinate system to the camera coordinate system. The physical principle of this transformation is that matrix multiplication essentially performs a coordinate rotation on the vector, causing the direction and magnitude of the angular velocity vector to be re-expressed in a coordinate system based on the camera's optical axis, thus obtaining the angular velocity in the camera coordinate system.

[0025] After obtaining the angular velocity in the camera coordinate system, it needs to be further mapped to the two-dimensional image plane to calculate the instantaneous motion velocity at the pixel level. This process relies on the intrinsic calibration matrix of the image sensor, which contains key parameters such as focal length, pixel size, and principal point coordinates. The angular velocity in the camera coordinate system is a three-dimensional rotation vector that describes the camera's rotation in space. According to the pinhole imaging model, a three-dimensional point in space will fall onto the two-dimensional image plane of the image sensor through perspective projection. When the camera rotates at a certain angular velocity, the rate of movement of the projected point on the image plane, i.e., the instantaneous motion velocity, can be derived from the cross product of the angular velocity vector and the three-dimensional coordinates of the image point. The processing unit uses the intrinsic calibration matrix to convert this velocity, derived from geometric relationships and expressed in physical units (such as meters per second), into the instantaneous motion velocity of the image plane in pixels per second. This velocity is a two-dimensional vector that characterizes the instantaneous flow direction and speed of image content caused by camera rotation.

[0026] Finally, to obtain the cumulative pixel displacement caused by rotation throughout the exposure period, the instantaneous motion velocity needs to be combined with the exposure time. The exposure time of the image sensor is a known control parameter that determines the length of time the photosensitive element receives light. The processing unit multiplies the instantaneous motion velocity vector of the image plane calculated in the previous step with the exposure time scalar. According to kinematic principles, under the approximate assumption of uniform rotation, the product of velocity and time is the displacement. Therefore, the result of this multiplication directly gives the total pixel position offset of any point on the image due to camera rotation within the exposure time interval. This result is also a two-dimensional vector, whose direction indicates the trajectory of the pixel movement, and whose magnitude represents the distance the pixel moved. This offset will be used to construct the complete motion blur model.

[0027] By multiplying the linear velocity data by the exposure time and combining it with depth estimation parameters, the amount of pixel displacement in the image plane caused by the drone's translation during the exposure time is calculated; the specific method is as follows: To accurately assess the blurring effect caused by the UAV's translational motion on imaging, the linear velocity measured by the inertial measurement unit (IMU) must first be analyzed within the image sensor's own coordinate system. The UAV's body coordinate system and the camera coordinate system have a fixed relative installation relationship in physical space. This relationship is fully described by a pre-determined and stored three-dimensional rotation matrix, namely the fixed installation attitude matrix of the camera relative to the UAV body. The processing unit obtains the current three-dimensional linear velocity vector expressed in the body coordinate system, which describes the instantaneous movement velocity of the UAV along its own forward / backward, left / right, and up / down directions. By performing matrix multiplication on this linear velocity vector and the installation attitude matrix, it can be transformed to the camera coordinate system. The principle is that the rotation matrix defines the coordinate axis rotation transformation relationship from the body coordinate system to the camera coordinate system; the matrix multiplication performs the same rotation transformation on the velocity vector, thus obtaining a new velocity vector whose direction and magnitude are both expressed with reference to the camera's optical axis and the plane perpendicular to it—that is, the linear velocity in the camera coordinate system.

[0028] After obtaining the linear velocity in the camera coordinate system, it is necessary to calculate the camera's positional change in real three-dimensional space during the specific exposure time of the image sensor. The exposure time of the image sensor is a known and controllable parameter that defines the duration for which the photosensitive element continuously collects light signals. The processing unit directly multiplies the linear velocity vector in the camera coordinate system with the exposure time scalar. Physically, if we assume that the UAV's translational motion is uniform during the short exposure time, then according to the basic kinematics, the product of velocity and time equals the displacement generated during that time period. Therefore, the result of this calculation is a three-dimensional displacement vector, which precisely represents the actual physical distance the camera's optical center moves along each coordinate axis in three-dimensional space between the start and end of the exposure time.

[0029] Finally, the physical displacements in the aforementioned three-dimensional space need to be converted into changes in pixel position observed on the two-dimensional image plane. This conversion relies on the internal geometric parameters of the image sensor and the depth information of the scene. The focal length parameter of the image sensor is a key intrinsic parameter that determines the perspective scale of the image. The depth estimation parameters of the scene are provided by an onboard depth sensing module (such as binocular vision or LiDAR), representing the approximate distance of each point in the scene relative to the optical center of the camera. The conversion process is based on the principle of similar triangles in perspective projection. Specifically, the three-dimensional displacement vector is decomposed into two components: one component is parallel to the image plane (i.e., perpendicular to the optical axis), and the other component is along the optical axis (i.e., the depth direction). The displacement component parallel to the image plane directly causes the image content to slip on the plane, and the corresponding pixel displacement can be calculated by multiplying the physical displacement component by the focal length and then dividing by the depth value at that point. The displacement component along the optical axis mainly affects the scale of the image (i.e., whether objects appear larger or smaller), and its contribution to the change in pixel position is usually negligible when the exposure time is short and the displacement is much smaller than the depth value. In this way, the actual translational displacement in three-dimensional space is quantitatively converted into the displacement of two-dimensional pixels on the image plane.

[0030] The pixel offsets caused by rotation and the pixel displacements caused by translation are merged to form a composite displacement vector, and a corresponding motion blur transfer function is constructed based on the direction and length of this vector; the specific method is as follows: After calculating the pixel offsets caused by rotation and translation, these two physical effects need to be unified into a common reference system for synthesis. Although both the pixel offset caused by rotation and the pixel displacement caused by translation represent pixel motion, they may originate from different intermediate calculation coordinate systems. The two-dimensional pixel coordinate system of the image plane is a unified reference system with the top-left corner of the image as the origin and pixels as the unit. The processing unit uses the intrinsic parameter calibration matrix of the image sensor to confirm the rotation offset, which is already expressed as a vector in the image plane; at the same time, it also normalizes the translation displacement calculated through three-dimensional projection to this unified two-dimensional pixel coordinate system using the same perspective projection relationship. The principle of this step is that only by mapping all motion effects to the same two-dimensional metric space can accurate vector synthesis be performed, thereby comprehensively reflecting the comprehensive motion trajectory of any point on the image during exposure.

[0031] Once the two offsets are in the same coordinate system, a comprehensive calculation can be performed. The processing unit reads the converted rotation pixel offset vector and translation pixel displacement vector from memory. Both vectors contain components in both horizontal and vertical directions. Subsequently, the processing unit performs vector addition, adding the horizontal components of the two vectors to obtain the horizontal component of the composite vector, and adding the vertical components of the two vectors to obtain the vertical component of the composite vector. The mathematical and physical principle behind this is that vector addition follows the parallelogram law, which represents that when a pixel is simultaneously driven by rotation and translation, its final motion effect is a linear superposition of these two independent motion effects. Through this operation, a new two-dimensional vector is obtained, namely the composite displacement vector, which fully describes the overall direction and total distance of motion of any pixel in the image from the starting point to the ending point within the exposure time.

[0032] Finally, a motion blur transfer function for image restoration is constructed based on the synthesized displacement vector. Mathematically, this function is often represented as a blur kernel. The processing unit first calculates the direction of the synthesized displacement vector, i.e., the angle between the vector and the positive direction of the horizontal axis of the image plane. According to geometric principles, this angle directly determines the tilt direction of the motion blur trail formed on the image. Therefore, this angle value is set as the angle parameter of the blur kernel in the motion blur transfer function. Next, the processing unit calculates the length of the synthesized displacement vector, i.e., the magnitude of the vector. This magnitude represents the total pixel displacement experienced by the pixel along the motion direction during exposure, and it determines the length of the motion trail. Therefore, this magnitude value is set as the length parameter of the blur kernel. Based on the determined angle and length parameters, the processing unit draws a line segment of length equal to the set value along the angle direction on a discrete pixel grid, and assigns non-zero weights to the grid points covered by the line segment, thereby generating a discrete, linear point spread function template, thus completing the specific construction of the motion blur transfer function.

[0033] In one embodiment of the present invention, step S3, in which the process of applying a motion blur transfer function to perform a deconvolution operation on the original image acquired by the image sensor and outputting a sharpened image, is as follows: After obtaining the transfer function characterizing the global motion blur effect, it is not directly applied to the entire image for restoration. Because the scene in front of the drone has varying depths during flight, and the motion of different parts relative to the drone may not be entirely consistent, the actual blur level in different regions of the image differs slightly from the model described by the main transfer function. Therefore, the processing unit first performs intelligent segmentation of the original image based on the direction parameter contained in the motion blur transfer function. This direction parameter indicates the dominant direction of motion blur. The processing unit analyzes the pixel intensity gradient along an axis perpendicular to this direction; locations where significant gradient transitions typically correspond to the boundaries of objects with different depths or motion characteristics in the scene. Based on these analyses, the original image is divided into multiple continuous strips or blocks, ensuring that each segmented region has similar depth features and relative motion states, thus maintaining consistent motion blur characteristics within the region and laying the foundation for subsequent accurate restoration.

[0034] For each segmented image region, the processing unit needs to calculate a dedicated deconvolution kernel function. First, the system fine-tunes the global motion blur transfer function based on the region's average depth information and the UAV's motion data, generating a point spread function estimate that better reflects the actual blur situation in that region. Then, based on this estimated point spread function, a deconvolution kernel is constructed using inverse filtering principles. Considering that direct inverse filtering amplifies noise, regularization methods, such as Wiener filtering, are typically used in practice to strike a balance between deblurring and noise suppression, thereby calculating a stable deconvolution kernel function suitable for the image data in that region. Next, the processing unit performs frequency domain deconvolution operations on the image data in that region. The principle is that, according to the convolution theorem, spatial domain convolution is equivalent to frequency domain multiplication. Therefore, the processing unit first converts the image data of the region to the frequency domain using a Fast Fourier Transform, and also converts the corresponding deconvolution kernel function to the frequency domain. Then, in the frequency domain, it performs a complex division operation between the image spectrum and the deconvolution kernel spectrum (in the Wiener filtering framework, this is a specific weighted division) to obtain the frequency domain representation of the restored image. Finally, through an inverse Fast Fourier Transform, the result is converted back to the spatial domain to obtain the preliminary sharpened image data of the region.

[0035] After completing the deconvolution operation for all independent regions, the processing unit needs to integrate these local results into a complete image. Since there are boundaries between segmented regions, direct stitching may result in discontinuities in grayscale or texture at the boundaries, forming seams. Therefore, the processing unit employs a weighted fusion strategy during the merging process. For pixels located within a region, the result of the deconvolution operation for that region is directly used; for pixels near the region boundary, the processing results of both their own region and adjacent regions are considered. Based on the distance from the pixel to the center of the two regions or other smoothness constraints, different weights are assigned to the results from different regions, resulting in linear mixing. The principle of this fusion operation is to make the transition between processing results from different regions smooth and natural through a gradual weight transition, avoiding the introduction of new visual defects. After fusion calculations for all pixels, a globally consistent, detailed, and complete sharpened image is finally generated.

[0036] In one embodiment of the present invention, in step S4, the process of dynamically adjusting the image capture frequency of the image sensor based on the magnitude of the displacement vector by the frame rate controller is as follows: The frame rate controller receives displacement vector amplitude data in real time from the displacement vector calculation module. This displacement vector amplitude data is a scalar value calculated by the displacement vector calculation module based on data from the inertial measurement unit and image sensor parameters. It quantifies the total pixel displacement length caused by the image sensor's movement on the image plane due to the drone's motion within a single exposure cycle. This data directly characterizes the intensity of the image sensor's motion per unit time. The frame rate controller continuously acquires this data stream through a dedicated data interface, serving as the core input for evaluating the current motion state and deciding on frame rate adjustments.

[0037] After obtaining the raw amplitude data, the frame rate controller performs a sliding window averaging filter on it. This process aims to eliminate potential instantaneous spikes or high-frequency noise fluctuations in the data. These fluctuations may be caused by instantaneous sensor errors or slight high-frequency jitter of the drone and do not reflect the true overall motion trend. The frame rate controller internally maintains a memory queue containing amplitude data from the most recent sampling times, i.e., the sliding window. Whenever new amplitude data arrives, it is added to the end of the queue, while historical data at the front of the queue is discarded. Then, the arithmetic mean of all data in the current queue is calculated, and this average is output as the smoothed displacement amplitude. The principle is that the arithmetic mean can effectively smooth random fluctuations, making the output value more representative of the average intensity of motion over a period of time.

[0038] Subsequently, the frame rate controller makes adaptive decisions based on the smooth displacement amplitude. The controller first calculates the rate of change of the current smooth displacement amplitude relative to the previous calculation cycle, reflecting whether the intensity of motion is increasing or decreasing. Based on the magnitude and sign of this rate of change, the controller dynamically adjusts an internal threshold for judgment; for example, when the rate of change is positive and large, indicating that motion is rapidly intensifying, the controller appropriately lowers the threshold, allowing the system to respond to the trend of increasing motion earlier. Next, the controller compares the current smooth displacement amplitude with this dynamically updated threshold to determine the current motion level, such as "low," "medium," or "high." This step introduces a hysteresis and adaptive mechanism to avoid frequent frame rate switching near the motion intensity threshold due to a fixed threshold, while enabling the system to adapt to different flight dynamics.

[0039] Finally, the frame rate controller generates the final control command based on the comparison results. Based on the motion level determined by comparing the smooth displacement amplitude with the dynamic threshold, the controller selects the corresponding target image capture frequency from a preset mapping table. This mapping table defines the frequency values ​​to be used to balance image quality and processing load at different motion levels. After selecting the target frequency, the controller generates a digital control signal containing specific frequency parameters and sends it to the image sensor's drive circuit via the communication bus. Upon receiving this signal, the image sensor drive circuit analyzes the frequency parameters and adjusts its internal clock and timing generator settings accordingly, thereby changing the interval between the image sensor's next exposure capture and ultimately achieving real-time, dynamic adjustment of the image capture frequency.

[0040] In one embodiment of the present invention, in step S5, the embedded processor deploys a lightweight convolutional neural network model, which receives a sharpened image and performs obstacle feature extraction and classification as follows: The first branch of the convolutional neural network performs multi-scale spatial feature extraction on the sharpened input image. This branch consists of multiple cascaded convolutional layers, some of which use a large stride to reduce the spatial size of the feature maps and expand the receptive field, thereby capturing the overall contour features of larger obstacles in the image. Simultaneously, the network structure incorporates bypass connections to fuse the feature maps output from shallow convolutions—which are spatially larger but contain rich details—with the feature maps output from deeper layers—which are semantically more coarse. This approach leverages the different granularities of information that different depths of the convolutional neural network focus on, enabling the model to simultaneously perceive various spatial information, from fine textures to large structures, effectively handling obstacles of varying sizes and distances.

[0041] Meanwhile, the network's second branch performs inter-frame difference analysis on the sharpened image to generate a motion feature map. This branch receives the sharpened image of the current frame and the previous frame image cached by the system. It first converts the two images to grayscale to simplify calculations, then calculates the absolute difference in their grayscale values ​​pixel by pixel. The principle behind this difference is that for a static background, the pixel values ​​change very little between consecutive frames, while moving objects will produce areas of significant pixel value difference. Next, a thresholding step is used to set the portion of the difference result below the threshold to zero, while the portion above the threshold is retained, thus obtaining a binarized initial motion region mask. Finally, a set of lightweight convolutional layers is used to refine and feature-encode the mask, outputting a motion feature map that matches the size of the feature map from the first branch, where highlighted areas indicate the locations of potential, moving obstacles.

[0042] The feature fusion layer of the network is responsible for integrating the two types of features mentioned above. This layer first concatenates the multi-scale spatial feature map generated by the first branch and the motion feature map generated by the second branch along the channel dimension, forming a comprehensive feature tensor with more channels. Subsequently, the fusion layer does not simply treat all channels equally, but processes them through a weighted module based on a channel attention mechanism. This module first performs global average pooling on the comprehensive feature tensor to obtain global statistical information for each channel; then, it analyzes this information through a small fully connected network and generates a weight coefficient between zero and one for each channel, the principle of which is to evaluate the importance of each channel's features to the current obstacle recognition task. Finally, these weight coefficients are multiplied by the corresponding original feature channels to achieve weighted fusion, thereby highlighting important features and suppressing unimportant features, forming an enhanced comprehensive feature representation.

[0043] The network's classifier makes the final prediction based on the aforementioned comprehensive feature representation. The classifier typically consists of fully connected layers that map the comprehensive feature representation into a vector, where each element corresponds to a score for a predefined obstacle category. Simultaneously, the classifier adaptively adjusts the confidence threshold required for classification based on the real-time image capture frequency from the image sensor. The principle is that when the capture frequency is low, the time interval between consecutive frames increases, and the displacement of moving objects may be more significant. However, the temporal continuity assumption relied upon by inter-frame difference analysis is weakened, and noise in the motion feature map may increase. Therefore, the system appropriately increases the classification confidence threshold, requiring the model to have a higher degree of certainty to classify an obstacle, thus reducing false positives. Conversely, when the capture frequency is high, the data is more consistent and reliable, and the threshold can be appropriately lowered to improve detection sensitivity. Finally, the classifier compares the scores of each category with the adjusted threshold and outputs the final obstacle category prediction result.

[0044] In one embodiment of the present invention, in step S6, the process by which the obstacle avoidance decision module fuses the processing result of the sharpened image with the frame rate control signal to generate the UAV heading adjustment command is as follows: The obstacle avoidance decision-making module synchronously receives two streams of information from the forward vision processing pipeline via an internal data bus. The first stream comes from the convolutional neural network model, transmitted in the form of structured data packets. These packets contain the identified obstacle category labels, their pixel coordinate bounding boxes in the image, and the model's output classification confidence score. The second stream comes from the frame rate controller, representing the current image capture frequency status signal. This digitally encoded signal indicates whether the image sensor is operating at a high, normal, or low frequency. The principle behind this synchronous reception of both types of information is that the static properties of the obstacle and the system's dynamic perception update frequency together constitute the decision-making context. Both must be considered simultaneously to make a flight decision adapted to the current situation.

[0045] Upon receiving the aforementioned information, the module's internally pre-defined decision rule base begins operation. This rule base contains multiple risk assessment strategies, each associated with a specific image capture frequency. The principle is that different frame rates mean varying speeds of environmental perception updates and information timeliness. At high frame rates, information updates are rapid, allowing the system to more actively handle dynamic obstacles; at low frame rates, information updates are slow, requiring a more conservative strategy to address potential information lag. The decision rule base automatically selects the corresponding activation strategy based on the current frame rate signal. Simultaneously, the module uses the obstacle's location information, particularly the offset distance and angle between its bounding box center and the image center (i.e., the drone's flight path), as input. Through the geometric mapping relationships defined in the selected strategy, it calculates the obstacle's initial threat level, such as far, medium, near, or low, medium, high.

[0046] Next, the module's computation unit further refines the initial threat level, generating a comprehensive threat assessment. The computation unit reads the classification confidence score output by the convolutional neural network model for each obstacle. This score reflects the model's confidence in its own identification results. Based on the preset rules in the selected risk assessment strategy, the computation unit uses the classification confidence score as a correction factor. The principle is that an obstacle identified by the model with high confidence is more likely to actually exist, and the decision should give it more weight; conversely, a low-confidence identification result may stem from false detection or ambiguity, and its threat should be carefully assessed. The computation unit uses a weighted formula to fuse the initial threat level with the classification confidence score; for example, when the confidence score is higher than a certain benchmark, the threat level is increased, and when it is lower than the benchmark, the threat level is decreased, ultimately outputting a comprehensive threat assessment value calibrated with confidence.

[0047] Finally, the command generation unit executes specific command mappings based on the comprehensive threat assessment value. This unit internally stores a preset heading command mapping table, which defines the suggested avoidance actions corresponding to different ranges of comprehensive threat assessment values. These actions are typically quantified as the angle and direction of heading adjustments, such as a specific degree of left or right yaw. The command generation unit queries this mapping table, finds the entry matching the current comprehensive threat assessment value, and parses out the target heading angle change contained within it. Subsequently, the unit encapsulates this change into a standard flight control command package and sends it to the UAV's flight controller via the flight control system's communication interface. Upon receiving the command, the flight controller coordinates it with other flight commands, ultimately driving the actuators to change the UAV's heading and complete the obstacle avoidance maneuver.

[0048] Please see Figure 2 As shown, the present invention also provides an AI vision-based intelligent obstacle avoidance system for drones, comprising: An inertial measurement module is used by UAVs to acquire angular velocity and linear velocity data in real time during flight via an onboard inertial measurement unit. The motion estimation module is used to calculate the displacement vector of the image sensor during the exposure time based on angular velocity and linear velocity data, and to construct the motion blur transfer function. The image restoration module is used to perform deconvolution operations on the original images acquired by the image sensor using the motion blur transfer function, and output a sharpened image. The frame rate adjustment module is used to dynamically adjust the image capture frequency of the image sensor based on the magnitude of the displacement vector. The obstacle detection module is used to deploy a lightweight convolutional neural network model that receives sharpened images and performs obstacle feature extraction and classification. The obstacle avoidance decision module is used to fuse the processing results of the sharpened image with the frame rate control signal to generate UAV heading adjustment commands.

[0049] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A method for intelligent obstacle avoidance of drones based on AI vision, characterized in that, Includes the following steps: S1. The UAV acquires angular velocity and linear velocity data in real time during flight through its onboard inertial measurement unit. S2. Based on the angular velocity and linear velocity data, calculate the displacement vector of the image sensor during the exposure time and construct the motion blur transfer function; S3. Using the motion blur transfer function, perform deconvolution operation on the original image acquired by the image sensor to output a sharpened image; S4. Based on the magnitude of the displacement vector, the frame rate controller dynamically adjusts the image capture frequency of the image sensor; S5: The embedded processor deploys a lightweight convolutional neural network model that receives sharpened images and performs obstacle feature extraction and classification. S6, the obstacle avoidance decision module, integrates the processing results of the sharpened image with the frame rate control signal to generate UAV heading adjustment instructions.

2. The AI ​​vision-based intelligent obstacle avoidance method for unmanned aerial vehicles according to claim 1, characterized in that, In step S2, the process of calculating the displacement vector of the image sensor during the exposure time based on the angular velocity and linear velocity data, and constructing the motion blur transfer function, is as follows: By combining angular velocity data with the calibration parameters of the image sensor, the pixel offset in the image plane caused by the rotation of the UAV during the exposure time is calculated; by multiplying the linear velocity data by the exposure time and combining it with the depth estimation parameters, the pixel displacement in the image plane caused by the translation of the UAV during the exposure time is calculated. The pixel offset caused by rotation and the pixel displacement caused by translation are merged to form a composite displacement vector, and the corresponding motion blur transfer function is constructed based on the direction and length of the vector.

3. The AI ​​vision-based intelligent obstacle avoidance method for unmanned aerial vehicles according to claim 2, characterized in that, The specific method for calculating the image plane pixel offset caused by the drone's rotation during the exposure time is as follows: Based on the fixed installation attitude matrix of the camera relative to the UAV body, the angular velocity data in the body coordinate system is transformed into the camera coordinate system; Using the intrinsic calibration matrix of the image sensor, the angular velocity in the camera coordinate system is converted into the instantaneous motion velocity in the image plane coordinate system; Multiplying the instantaneous motion velocity on the image plane by the exposure time of the image sensor yields the pixel offset of the image plane caused by rotation.

4. The AI ​​vision-based intelligent obstacle avoidance method for unmanned aerial vehicles according to claim 2, characterized in that, The specific method for calculating the image plane pixel displacement caused by the drone's translation during the exposure time by multiplying the linear velocity data by the exposure time and combining it with the depth estimation parameters is as follows: Based on the fixed mounting attitude matrix of the camera relative to the UAV body, the linear velocity data in the body coordinate system is transformed to the camera coordinate system; Multiply the linear velocity in the camera coordinate system by the exposure time of the image sensor to obtain the actual translational displacement of the camera in three-dimensional space during the exposure time. Using the focal length parameters of the image sensor and the depth estimation parameters of the scene, the actual translational displacement in three-dimensional space is converted into pixel displacement on the image plane.

5. The AI ​​vision-based intelligent obstacle avoidance method for unmanned aerial vehicles according to claim 2, characterized in that, The specific method for constructing the corresponding motion fuzzy transfer function based on the direction and length of the vector is as follows: The pixel offset caused by rotation and the pixel displacement caused by translation are uniformly transformed into the two-dimensional pixel coordinate system of the image plane; In a two-dimensional pixel coordinate system, the two pixel offset vectors are added together to obtain the composite displacement vector. Based on the direction of the synthesized displacement vector, the angle parameter of the fuzzy kernel in the motion fuzzy transfer function is determined; based on the length of the synthesized displacement vector, the length parameter of the fuzzy kernel in the motion fuzzy transfer function is determined, thereby constructing the motion fuzzy transfer function.

6. The AI ​​vision-based intelligent obstacle avoidance method for unmanned aerial vehicles according to claim 1, characterized in that, In step S3, the process of applying the motion blur transfer function to perform deconvolution on the original image acquired by the image sensor and outputting a sharpened image is as follows: Based on the direction parameter of the motion blur transfer function, the original image is segmented into multiple image regions with consistent motion blur characteristics; For each image region with motion blur characteristics, the corresponding deconvolution kernel function is calculated; using the deconvolution kernel function of each region, frequency domain deconvolution operation is performed on the image data of that region. The deconvolution results of all regions are merged to generate a complete, sharpened image.

7. The AI ​​vision-based intelligent obstacle avoidance method for unmanned aerial vehicles according to claim 1, characterized in that, In step S4, the process of dynamically adjusting the image capture frequency of the image sensor based on the magnitude of the displacement vector by the frame rate controller is as follows: The frame rate controller receives displacement vector amplitude data from the displacement vector calculation module in real time, which characterizes the degree of motion of the image sensor during the exposure time. The frame rate controller performs sliding window averaging filtering on the displacement vector amplitude data to eliminate instantaneous fluctuations and output a smooth displacement amplitude. The frame rate controller calculates the rate of change of the smooth displacement amplitude, dynamically updates the internal threshold based on the rate of change, and compares the smooth displacement amplitude with the threshold. The frame rate controller generates an image capture frequency control signal based on the comparison results, which directly instructs the image sensor to adjust its image capture frequency.

8. The AI ​​vision-based intelligent obstacle avoidance method for unmanned aerial vehicles according to claim 1, characterized in that, In step S5, the embedded processor deploys a lightweight convolutional neural network model. The process by which this model receives the sharpened image and performs obstacle feature extraction and classification is as follows: The first branch of the convolutional neural network extracts multi-scale spatial features from the sharpened input image, while the second branch performs inter-frame difference analysis on the image to generate motion feature maps. The feature fusion layer of the network performs channel splicing and weighted fusion of multi-scale spatial features and motion feature maps to form a comprehensive feature representation; The network's classifier outputs an obstacle category prediction based on a comprehensive feature representation, and this classifier adaptively adjusts its classification confidence threshold according to the image capture frequency.

9. The AI ​​vision-based intelligent obstacle avoidance method for unmanned aerial vehicles according to claim 1, characterized in that, In step S6, the process by which the obstacle avoidance decision module integrates the processing results of the sharpened image with the frame rate control signal to generate the UAV heading adjustment command is as follows: The obstacle avoidance decision module synchronously receives obstacle category and location information from the convolutional neural network model, as well as the current image capture frequency status signal from the frame rate controller; The module's internal preset decision rule base selects the corresponding risk assessment strategy based on the image capture frequency and maps obstacle location information to threat level; The module computing unit performs a weighted fusion of the obstacle threat level and the classification confidence of the convolutional neural network output based on the selected risk assessment strategy to generate a comprehensive threat assessment. Based on the comprehensive threat assessment, the command generation unit queries the preset heading command mapping table and outputs specific heading adjustment angle and direction commands for the UAV.

10. An AI vision-based intelligent obstacle avoidance system for unmanned aerial vehicles (UAVs), implemented in any one of claims 1-9, characterized in that, include: An inertial measurement module is used by UAVs to acquire angular velocity and linear velocity data in real time during flight via an onboard inertial measurement unit. The motion estimation module is used to calculate the displacement vector of the image sensor during the exposure time based on angular velocity and linear velocity data, and to construct the motion blur transfer function. The image restoration module is used to perform deconvolution operations on the original images acquired by the image sensor using the motion blur transfer function, and output a sharpened image. The frame rate adjustment module is used to dynamically adjust the image capture frequency of the image sensor based on the magnitude of the displacement vector. The obstacle detection module is used to deploy a lightweight convolutional neural network model that receives sharpened images and performs obstacle feature extraction and classification. The obstacle avoidance decision module is used to fuse the processing results of the sharpened image with the frame rate control signal to generate UAV heading adjustment commands.

Images