Unmanned aerial vehicle-based obstacle three-dimensional model reconstruction method and system

Abstract

The application relates to the technical field of image processing, in particular to a method and system for reconstructing a three-dimensional model of an obstacle based on a UAV (unmanned aerial vehicle). The method comprises the following steps: acquiring continuous multiple frames of gray images collected by the UAV, determining the average displacement distance and the inter-frame displacement angle of feature points in the gray images; dividing the gray images into multiple regions, and screening false feature points according to the average displacement distance and the inter-frame displacement angle of the feature points in the regions, removing the false feature points, and obtaining true feature points; clustering the true feature points according to the inter-frame displacement angle and the average displacement distance of the true feature points, and obtaining multiple clustering clusters; determining local optimal fuzzy kernels of the clustering clusters, removing the fuzziness of local regions corresponding to the clustering clusters by using the local optimal fuzzy kernels, obtaining clear images, and constructing a visual three-dimensional model of the obstacle in the clear images based on the clear images. In this way, the application improves the recognition accuracy of the obstacle.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a method and system for reconstructing three-dimensional obstacle models based on unmanned aerial vehicles (UAVs). Background Technology

[0002] In fields such as operation and maintenance inspection, drones have gradually replaced traditional manual foot and vehicle inspections, becoming core operational equipment. They can be equipped with high-definition cameras and other data acquisition devices, autonomously flying along preset routes to efficiently complete data collection tasks, significantly improving inspection efficiency and safety. However, drones often face complex operating environments, making obstacle avoidance and 3D reconstruction crucial functions for ensuring flight safety and achieving precise operations, indispensable in scenarios such as surveying, construction, and disaster relief.

[0003] In some scenarios, the industry mainstream adopts a sensor combination solution of "vision + LiDAR" or "millimeter-wave radar + vision" to build an obstacle recognition and 3D modeling system. Both solutions leverage the advantages of multi-sensor data fusion to achieve complementary perception of environmental information, providing environmental support for drone operations. However, when drones move at high speeds, the image acquisition unit is easily affected by relative motion, resulting in significant motion blur, leading to loss of image details and reduced feature recognition accuracy. This blur directly interferes with the subsequent feature extraction and matching accuracy of obstacles, resulting in a low obstacle recognition accuracy. Summary of the Invention

[0004] To address the technical problem of low obstacle recognition accuracy, the present invention aims to provide a method for reconstructing three-dimensional obstacle models based on unmanned aerial vehicles (UAVs).

[0005] To solve the above technical problems, the specific technical solution adopted is as follows:

[0006] This invention provides a method for reconstructing a 3D obstacle model based on an unmanned aerial vehicle (UAV), comprising: acquiring multiple consecutive frames of grayscale images collected by the UAV; determining the average displacement distance and inter-frame displacement angle of feature points in the grayscale images based on their coordinates; dividing the grayscale images into multiple regions; filtering false feature points based on the average displacement distance and inter-frame displacement angle of each feature point in the region; removing false feature points to obtain true feature points; clustering the true feature points based on their inter-frame displacement angle and average displacement distance to obtain multiple clusters; determining the local optimal blur kernel of each cluster based on the average displacement distance and inter-frame displacement angle of the true feature points in each cluster; deblurring the local regions corresponding to each cluster using the local optimal blur kernel to obtain a clear image; and constructing a visualized 3D model of the obstacles in the clear image based on the clear image.

[0007] Optionally, determining the average displacement distance and inter-frame displacement angle of feature points in grayscale images based on their coordinates in the grayscale images includes: determining the inter-frame displacement distance and inter-frame displacement angle of feature points based on their coordinates in adjacent grayscale images; and determining the average displacement distance of feature points in each grayscale image based on their inter-frame displacement distance.

[0008] Optionally, filtering false feature points based on the average displacement distance and inter-frame displacement angle of each feature point in the region includes: determining the average displacement reference of the region based on the average displacement distance and the number of feature points; filtering false feature points based on the average displacement reference and average displacement distance, and removing false feature points from each feature point to obtain the remaining feature points; filtering false feature points from the remaining feature points based on the inter-frame displacement angle of the remaining feature points, and removing false feature points from the remaining feature points to obtain the true feature points.

[0009] Optionally, filtering false feature points based on the average displacement benchmark and the average displacement distance includes: calculating the absolute value of a first difference between the average displacement benchmark and the average displacement distance; and identifying feature points whose absolute value of the first difference is greater than a first threshold as false feature points.

[0010] Optionally, filtering false feature points from the remaining feature points based on their inter-frame displacement angles includes: when the UAV is in straight-line flight, determining the absolute value of the second difference between the inter-frame displacement angles of adjacent remaining feature points; if the absolute value of the second difference between the remaining feature points is greater than a second threshold, identifying the remaining feature points as false feature points; when the UAV is in side-flying flight, determining the average value of the inter-frame displacement angles of the remaining feature points as the displacement angle of the remaining feature points; determining the average displacement angle based on the displacement angles of each remaining feature point; determining the theoretical displacement angle of the remaining feature points based on the coordinates of the remaining feature points and the coordinates of the center point of the grayscale image; determining the UAV's flight attitude identifier based on its flight state; determining the adaptive theoretical angle of the feature points based on the average displacement angle, the theoretical displacement angle, and the flight attitude identifier; and identifying feature points whose absolute value of the third difference between the displacement angle of the remaining feature points and the adaptive theoretical angle is less than a third threshold as false feature points.

[0011] Optionally, clustering each true feature point based on its inter-frame displacement angle and average displacement distance yields multiple clusters, including: determining the displacement angle of the true feature point based on its inter-frame displacement angle; converting the displacement angle into a two-dimensional vector and normalizing the average displacement distance of the true feature point to obtain a normalized displacement distance; determining the feature vector of the true feature point based on the two-dimensional vector and the normalized displacement distance; and clustering each true feature point based on its feature vector and the UAV's flight attitude identifier to obtain multiple clusters.

[0012] Optionally, clustering each true feature point based on the feature vector of the true feature point and the flight attitude identifier of the UAV to obtain multiple clusters includes: determining the target number of clusters based on the flight attitude identifier of the UAV; and clustering each true feature point based on the feature vector of the true feature point to obtain the target number of clusters.

[0013] Optionally, determining the local optimal fuzzy kernel of a cluster based on the average displacement distance and inter-frame displacement angle of the true feature points in each cluster includes: determining the local average displacement of the cluster based on the average displacement distance and the number of true feature points in each cluster; determining the average value of the inter-frame displacement angle of the true feature points as the displacement angle of the true feature points; determining the local displacement angle of the cluster based on the displacement angle of the true feature points in each cluster; and determining the local optimal fuzzy kernel based on the local average displacement and local displacement angle of the cluster.

[0014] Optionally, determining the locally optimal fuzzy kernel based on the local average displacement and local displacement angle of the cluster includes: determining the edge length of the fuzzy kernel of the cluster based on the local average displacement; determining the center coordinates of the fuzzy kernel of the cluster based on the edge length of the fuzzy kernel; and determining the location of the trailing image of the fuzzy kernel within the fuzzy kernel based on the local displacement angle and local average displacement. The first offset in direction, and in The second offset in the direction; using the center coordinates of the blur kernel, the first offset, and the second offset, determine the coordinates of the path pixels along the direction from the center of the blur kernel to the first and second offsets; determine the zero matrix according to the side length of the blur kernel and assign the first value to the position of the path pixel coordinates in the zero matrix, and take the second value for the other positions; normalize the zero matrix according to the number of path pixels to obtain the local optimal blur kernel, where the first value is greater than the second value.

[0015] This invention offers the following advantages: By calculating the average displacement distance and inter-frame displacement angle of feature points in multiple consecutive frames of grayscale images from a UAV, a quantitative representation of the blurred state is achieved. Furthermore, after dividing the grayscale image into multiple regions, false feature points are screened and eliminated based on the average displacement distance and inter-frame displacement angle of the aforementioned feature points. This effectively avoids noise and background interference, ensuring the purity of true feature points and addressing the detail loss problem caused by motion blur at its source, laying a reliable foundation for subsequent deblurring processes. Furthermore, a locally optimal blur kernel is determined based on the average displacement distance and inter-frame displacement angle of feature points in each cluster. Targeted deblurring is then applied to the corresponding regions, breaking the limitations of traditional single-kernel deblurring. While deblurring, details such as obstacle contours and textures are preserved, resulting in a high-quality, clear image. The obtained clear image provides high-quality data for constructing a visual 3D model of obstacles, ensuring the accuracy of model size and shape. Thus, the clear image retains details such as obstacle contours and textures, avoiding interference with the subsequent feature extraction and matching accuracy of obstacles, and improving the accuracy of obstacle recognition. Attached Figure Description

[0016] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a method for reconstructing a three-dimensional obstacle model based on an unmanned aerial vehicle (UAV) according to an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of a three-dimensional obstacle model reconstruction system based on an unmanned aerial vehicle (UAV) according to an embodiment of the present invention. Detailed Implementation

[0019] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, details the specific implementation, structure, features, and effects of a method and system for reconstructing a three-dimensional obstacle model based on an unmanned aerial vehicle (UAV) according to the present invention. In the following description, different "one embodiment" or "another embodiment" do not necessarily refer to the same embodiment. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0020] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0022] The scenario targeted by this invention is that images captured by drones during high-speed movement are often accompanied by motion blur, which seriously affects the accuracy of obstacle recognition, leading to false detection or loss of obstacles and reducing the accuracy of modeling.

[0023] The following describes in detail, with reference to the accompanying drawings, a specific scheme of the obstacle 3D model reconstruction method based on UAV provided by the present invention.

[0024] Example 1:

[0025] Please see Figure 1 The diagram illustrates a flowchart of a method for reconstructing a three-dimensional obstacle model based on a drone, according to an embodiment of the present invention, including:

[0026] Step S101: Acquire multiple consecutive frames of grayscale images captured by the UAV, and determine the average displacement distance and inter-frame displacement angle of the feature points in the grayscale images based on the coordinates of the feature points in the grayscale images.

[0027] Specifically, this invention utilizes a drone to acquire multiple consecutive frames of images ahead of its flight path in real time during flight. During image acquisition, motion blur occurs due to the movement of objects, creating trailing shadows. Motion blur is essentially the trailing shadow created by moving objects in the image. To restore the effects of motion blur, a blur kernel needs to be calculated based on the length and direction of the trailing shadow. Feature points (corner points) are areas in an image with the most dramatic texture changes (such as road sign corners, building edges, etc.). They possess characteristics of "high stability and high recognizability," are not easily affected by changes in lighting or noise, and are ideal targets for tracking motion trajectories. Therefore, this invention can use the Harris corner detection algorithm to find feature points in the image.

[0028] Furthermore, consecutive multiple frames of images are the basis for obtaining the inter-frame displacement of feature points. Five consecutive frames of images can effectively offset the random errors caused by airflow jitter in a single frame, improving the stability of displacement calculation. Therefore, this embodiment of the invention uses five consecutive frames of images as an example for illustration. Of course, the number of consecutive multiple frames of images can also be other, and this embodiment of the invention is not limited thereto. By acquiring consecutive multiple frames of images and using the SIFT feature point matching algorithm, the inter-frame displacement data of the same feature point (i.e., the data of the blur trail) can be calculated, and then the key parameters for blur kernel calculation can be derived.

[0029] Based on the above analysis, in this embodiment of the invention, during the flight of the UAV, a camera is used to collect multiple consecutive frames of images in front of the flight path in real time, and the images are processed into grayscale to obtain grayscale images.

[0030] Furthermore, in this embodiment of the invention, the Sobel operator is used to calculate the gradient of the grayscale image, and then the Harris corner detection algorithm is used to detect feature points in the grayscale image based on the gradient.

[0031] Furthermore, for multiple consecutive grayscale images, the SIFT feature point matching algorithm can be used to match the same feature points in different grayscale images. By calculating the coordinate changes of these matched feature points between different frames, the inter-frame displacement distance and inter-frame displacement angle can be obtained. As an optional embodiment of the present invention, determining the average displacement distance and inter-frame displacement angle of feature points in a grayscale image based on their coordinates in the grayscale image includes: determining the inter-frame displacement distance and inter-frame displacement angle of feature points based on the coordinates of feature points in each grayscale image in adjacent grayscale images; and determining the average displacement distance of feature points in each grayscale image based on the inter-frame displacement distance of feature points.

[0032] Specifically, in this embodiment of the invention, five consecutive grayscale images are taken. ( The earliest grayscale image, The first grayscale image after the earliest frame. This represents the second grayscale image after the earliest frame. This represents the third grayscale image after the earliest frame. (The grayscale image of the current frame), for feature points The coordinates in the above 5 consecutive grayscale images are: , This represents the x-coordinate in the j-th grayscale image above. Let the vertical coordinate of the j-th frame grayscale image be represented. Calculate the displacement vector between the four groups of adjacent grayscale images: , Let represent the displacement vector between the grayscale images of the k-th adjacent frames. This represents the x-coordinate of the feature point in the (k+1)th frame of the grayscale image. This represents the ordinate of the feature point in the (k+1)th frame of the grayscale image. This represents the x-coordinate of the feature point in the k-th grayscale image. This represents the ordinate of the feature point in the k-th frame of the grayscale image.

[0033] Furthermore, in this embodiment of the invention, the inter-frame displacement distance of feature points is calculated using the following formula:

[0034] ;

[0035] In the above formula, This represents the inter-frame displacement distance of a feature point in the grayscale image of the kth adjacent frame. This represents the x-coordinate of the feature point in the (k+1)th frame of the grayscale image. This represents the ordinate of the feature point in the (k+1)th frame of the grayscale image. This represents the x-coordinate of the feature point in the k-th grayscale image. This represents the ordinate of the feature point in the k-th frame of the grayscale image.

[0036] Furthermore, in this embodiment of the invention, the inter-frame displacement angle of the feature points is calculated using the following formula:

[0037] ;

[0038] In the above formula, This represents the inter-frame displacement angle of the feature point in the grayscale image of the kth adjacent frame. This represents the x-coordinate of the feature point in the (k+1)th frame of the grayscale image. This represents the ordinate of the feature point in the (k+1)th frame of the grayscale image. This represents the x-coordinate of the feature point in the k-th grayscale image. This represents the ordinate of the feature point in the k-th frame of the grayscale image. This represents the two-parameter arctangent function, which calculates the angle value of the feature point in the grayscale image of the kth adjacent frame based on the coordinates of the feature point.

[0039] Furthermore, in this embodiment of the invention, the average displacement distance of feature points in each frame of grayscale image is calculated using the following formula:

[0040] ;

[0041] In the above formula, It represents the average displacement distance of the i-th feature point in multiple consecutive grayscale images. This represents the inter-frame displacement distance of a feature point in the grayscale image of the kth adjacent frame.

[0042] Step S102: Divide the grayscale image into multiple regions, and filter out false feature points based on the average displacement distance and inter-frame displacement angle of each feature point in the region. Remove the false feature points to obtain the true feature points.

[0043] Specifically, grayscale images contain noise, glare, and other false feature points. Directly calculating the blur kernel using the original data will distort the deblurring effect. Therefore, these interfering points need to be removed to avoid affecting the calculation results. During drone flight, the camera is focused on a linearly extending road, buildings, and surrounding environment. The motion displacement of all real physical features follows the rule that the closer the distance, the faster the relative speed and the greater the inter-frame displacement, which differs in near, medium, and far views. Furthermore, the drone's flight attitude greatly affects the displacement of feature points; when flying sideways, the orientation is consistent, while when flying straight, they radiate along the depth of field, requiring separate discussion. Removing these non-compliant "false feature points" is crucial to obtaining the most realistic blur data and thus finding the locally optimal blur kernel.

[0044] Furthermore, as an optional embodiment of the present invention, filtering false feature points based on the average displacement distance and inter-frame displacement angle of each feature point in the region includes: determining the average displacement reference of the region based on the average displacement distance of each feature point in the region and the number of feature points; filtering false feature points based on the average displacement reference and the average displacement distance, and removing the false feature points from each feature point to obtain the remaining feature points; filtering false feature points from the remaining feature points based on the inter-frame displacement angle of the remaining feature points, and removing the false feature points from the remaining feature points to obtain the true feature points.

[0045] Specifically, this invention, based on the principle of near-large and far-small, divides a grayscale image into three parts: foreground, midground, and background. It calculates the displacement changes of feature points across multiple consecutive grayscale frames and eliminates false feature points that do not conform to the rules. When a drone flies, the difference in distance between the lens and obstacles directly leads to differences in the displacement of feature points in different areas of the image. This is a fundamental principle of optical imaging and a key to distinguishing real feature points (such as signs, building edges, etc.) from false feature points (such as noise, road reflections). Therefore, this invention divides the grayscale image into three regions based on the near-large and far-small characteristics: the foreground region, the midground region, and the background region. Feature points that do not conform to the displacement distance rules are eliminated according to the characteristics of each region. The foreground region refers to the area containing foreground objects, such as guardrails or spilled debris within 10 meters of the roadside. When the drone flies, foreground objects move relatively quickly in the grayscale image, resulting in a large displacement distance (long trailing shadow) between consecutive grayscale frames. The midground region refers to the transition area between the foreground and background. The distant view area refers to the area containing distant objects, such as trees or distant mountains 100 meters away. The relative speed of distant objects and the drone is slow, and the displacement distance between consecutive grayscale images is small (short ghosting).

[0046] Furthermore, the displacement of false feature points (such as sensor noise and lens flare) is completely random and does not follow the displacement rule of near-large and far-small, so it can be accurately eliminated through this rule.

[0047] Furthermore, the imaging process of a camera projects three-dimensional space (a stereoscopic scene) onto a two-dimensional image. When viewed horizontally forward, the "proximity" and "distance" of three-dimensional space extend along the lens's central axis (directly in front). The rule of perspective projection is that a plane perpendicular to the lens's central axis in three-dimensional space occupies a more peripheral area in the image the closer it is to the lens; conversely, the farther away it is, the more central it appears. Therefore, for a resolution of [missing information - likely a resolution value], [missing information - likely a resolution value], [missing information - likely a resolution value]. The image (L is the length of the acquired grayscale image, W is the width of the acquired grayscale image), with the center point of the original grayscale image as the reference, is taken respectively. and Two concentric rectangular windows divide the image into a distant region (the center of the grayscale image). The area within the window), the mid-range area (center of the grayscale image). and The circular area between the two and the foreground area (center of the grayscale image) (The area outside the image edge).

[0048] Furthermore, the displacement of the true characteristic points within each region should fluctuate around a reference value (the distances within the same region are basically consistent). Therefore, it is necessary to calculate the average displacement reference for each region. In this embodiment of the invention, the average displacement reference for each region is calculated using the following formula:

[0049] ;

[0050] In the above formula, This represents the average displacement reference for the p-th region. It represents the average displacement distance of the i-th feature point in multiple consecutive grayscale images. This represents the number of feature points in the p-th region.

[0051] Furthermore, as an optional embodiment of the present invention, screening false feature points based on the average displacement benchmark and the average displacement distance includes: calculating the absolute value of a first difference between the average displacement benchmark and the average displacement distance; and determining feature points whose absolute value of the first difference is greater than a first threshold as false feature points.

[0052] Specifically, the first threshold in this embodiment can be selected according to actual conditions, and this embodiment does not limit it. In this embodiment, the first threshold is determined based on the average displacement distance of each feature point, thereby setting corresponding thresholds specifically for different regions and improving the accuracy of filtering false feature points in each region. For example, the first threshold is set as follows: The embodiments of this invention use the following formula to filter false feature points:

[0053] ;

[0054] In the above formula, This represents the average displacement reference for the p-th region. This represents the average displacement distance of the i-th feature point across multiple consecutive grayscale images. Thus, using the above formula, feature points that deviate significantly from the average displacement benchmark within the region are marked as "false feature points" and removed from the data.

[0055] Furthermore, during drone flight, the inter-frame displacement angle of feature points in the acquired grayscale image is determined by the "position change of feature points within the lens's field of view in two adjacent frames," which is essentially the rotation angle of the line of sight. When the drone flies at a constant speed, the relative position change between the lens and the feature points is continuous and uniform, and the angular velocity of the line of sight rotation is constant. This is reflected in the image as the feature points moving at essentially the same angle in each frame, without significant fluctuations. However, the angles of false feature points will change irregularly. Some "false feature points" can be eliminated by checking the stability of the inter-frame displacement angle. In addition, there are generally two flight attitudes during drone flight. One is the side-flying attitude: the drone translates along the horizontal direction (x-axis) of the image, producing a pure horizontal linear trail. The trail direction is consistent with the side-flying direction, and there is no center-edge blur difference in the entire image. It is only affected by "near objects appear larger and far objects appear smaller" (the trail is longer for near objects and shorter for distant objects), and the displacement direction of all feature points in the image is the same. Secondly, there's the straight-flight attitude: the drone moves along the depth of field (perpendicular to the image plane). Due to depth parallax, the relative movement distance in the image center is small (light blur), while the relative movement distance in the edge areas is large (heavy blur), exhibiting a radial pattern from "center to edge." The displacement direction of feature points across the entire image covers a 360° range, requiring the calculation of local angles by segmenting the regions. If the attitude is not differentiated and the standard of "global angle consistency" is used to judge feature points in the straight-flight scene, many true points with radial angles will be misclassified as false points, and vice versa. Therefore, it is necessary to combine the two flight attitudes to set an angle judgment standard.

[0056] Furthermore, as an optional embodiment of the present invention, filtering false feature points from the remaining feature points based on the inter-frame displacement angles of the remaining feature points includes: when the UAV is in a straight-line flight state, determining the absolute value of the second difference between the inter-frame displacement angles of adjacent remaining feature points; if the absolute value of the second difference between the remaining feature points is greater than a second threshold, determining the remaining feature points as false feature points; when the UAV is in a side-flying flight state, determining the average value of the inter-frame displacement angles of the remaining feature points as the displacement angle of the remaining feature points; determining the average displacement angle based on the displacement angles of each remaining feature point; determining the theoretical displacement angle of the remaining feature points based on the coordinates of the remaining feature points and the coordinates of the center point of the grayscale image; determining the flight attitude identifier of the UAV based on the flight state of the UAV; determining the adaptive theoretical angle of the feature points based on the average displacement angle, the theoretical displacement angle, and the flight attitude identifier; and designating feature points whose absolute value of the third difference between the displacement angle of the remaining feature points and the adaptive theoretical angle is less than a third threshold as false feature points.

[0057] Specifically, drones generally fly at a constant speed in a straight line, so the inter-frame displacement angle of the same feature point should remain unchanged. For feature point i, the absolute value of the second difference between the inter-frame displacement angles of adjacent groups can be calculated using the following formula:

[0058] ;

[0059] In the above formula, This represents the absolute value of the second difference in the inter-frame displacement angle of feature point i in adjacent groups. This represents the inter-frame displacement angle of the feature point in the (k+1)th group. This represents the inter-frame displacement angle of the feature point in the k-th group.

[0060] Furthermore, the second threshold can be determined according to the actual situation. In this embodiment of the invention, the second threshold is set to... ,when This indicates that the displacement angle of the feature point fluctuates too much, so the point is marked as a "false feature point" and removed. This indicates that the displacement angle is stable, and the average value of the inter-frame displacement angle (eliminating single-frame fluctuations) is taken as the displacement angle of this feature point. The following formula is used for calculation:

[0061] ;

[0062] In the above formula, This represents the displacement angle of feature point i. This represents the inter-frame displacement angle of the feature point in the k-th group. It is worth noting that, based on the 5 consecutive grayscale images, adjacent consecutive grayscale images are sequentially designated as group 0, group 1, group 2, and group 3 in the order from front to back.

[0063] Furthermore, when flying sideways, the UAV moves horizontally to the left or right. The displacement angles of all feature points on the map are consistent and the directions are uniform. Therefore, the average displacement angle can be directly calculated using the following formula:

[0064] ;

[0065] In the above formula, This represents the average displacement angle. This represents the displacement angle of feature point i. This indicates the number of remaining feature points.

[0066] Furthermore, during direct flight, due to the influence of depth parallax, the displacement direction of feature points is radial with the image center as the center. Since the angle of the blur trail radiates 360 degrees in the image, the theoretical angle of the feature point can be calculated based on the difference between its coordinates in the image and the image center. Let the x-coordinate of the image center point c be... and ordinate for (L is the length of the grayscale image, W is the width of the grayscale image), then for feature points , This represents the x-coordinate of feature point i. Let i represent the ordinate of feature point i, which can be used to calculate the theoretical displacement angle using the following formula:

[0067] ;

[0068] In the above formula, This represents the theoretical displacement angle. This represents the two-parameter arctangent function.

[0069] Furthermore, to achieve adaptive adjustment, a flight attitude identifier needs to be introduced. In this embodiment of the invention, the flight attitude identifier is denoted as att, which can be represented by the following formula:

[0070] ;

[0071] in, When =1, it indicates that the drone is in a straight flight state (lateral translation). When =0, it indicates that the drone is in a side-flying state (depth of field direction).

[0072] Furthermore, in this embodiment of the invention, the adaptive theoretical angle is determined using the following formula:

[0073] ;

[0074] In the above formula, This represents the perspective of adaptive theory. This represents the theoretical displacement angle. This represents the average displacement angle.

[0075] Furthermore, the third threshold can be selected based on actual circumstances; in this embodiment of the invention, it is set to [value missing]. For feature points Its displacement angle as a true feature point Must meet: If not satisfied If the feature point is false, it will be marked as a false feature point and removed.

[0076] Step S103: Cluster each true feature point according to the inter-frame displacement angle and average displacement distance of the true feature points to obtain multiple clusters.

[0077] Specifically, in real-world scenarios, drones are required to process collected data in real time. The K-means algorithm has low computational complexity, making it suitable for embedded computing power. Furthermore, K-means can automatically group features based on "feature vector similarity," eliminating the need for manual grouping rules and adapting to feature point distributions in different scenarios (e.g., clear near-to-mid-to-far grouping during side-flying, and multiple angle groupings during straight-flying). For feature points within the same cluster, whose attributes are similar, a local fuzzy kernel can be calculated based on the feature point data within the cluster.

[0078] Furthermore, as an optional embodiment of the present invention, clustering each true feature point according to the inter-frame displacement angle and average displacement distance of the true feature points to obtain multiple clusters includes: determining the displacement angle of the true feature points according to the inter-frame displacement angle; converting the displacement angle into a two-dimensional vector and normalizing the average displacement distance of the true feature points to obtain a normalized displacement distance; determining the feature vector of the true feature points according to the two-dimensional vector and the normalized displacement distance; and clustering each true feature point according to the feature vector of the true feature points and the flight attitude identifier of the UAV to obtain multiple clusters.

[0079] Specifically, for true feature points, the embodiments of the present invention First, its displacement angle Converting to a two-dimensional vector (eliminating periodicity) is specifically represented by the following formula:

[0080] ;

[0081] Specifically, This represents the displacement angle of the true feature point. The angle vector representing the x-direction. This represents the angle vector in the y-direction. The cosine value of the displacement angle of the true feature point. The sine value represents the displacement angle of the true feature point.

[0082] Furthermore, for true feature points average displacement distance To perform normalization and avoid excessively high weights, the following formula is used for calculation:

[0083] ;

[0084] In the above formula, Representing true feature points The normalized displacement distance. Representing true feature points The average displacement distance. This represents the minimum average displacement distance among all feature points. This represents the maximum value among the average displacement distances of all feature points.

[0085] Furthermore, the true feature points are determined using the following formula. Feature vectors:

[0086] ;

[0087] In the above formula, Representing true feature points eigenvectors. Representing true feature points The normalized displacement distance. The angle vector representing the x-direction. This represents the angle vector in the y-direction.

[0088] Furthermore, as an optional embodiment of the present invention, clustering each true feature point based on the feature vector of the true feature point and the flight attitude identifier of the UAV to obtain multiple clusters includes: determining the target number of clusters based on the flight attitude identifier of the UAV; and clustering each true feature point based on the feature vector of the true feature point to obtain the target number of clusters.

[0089] Specifically, in this embodiment of the invention, the K-means clustering algorithm is used to cluster feature point data based on feature vectors. The target number K for clustering is selected using the following formula:

[0090] ;

[0091] In the above formula, This represents the target number of clusters for clustering feature point data. This indicates the flight attitude indicator (1 for straight flight, 0 for side flight).

[0092] When the drone flies sideways, the displacement angle of the feature points is the same. Considering only the difference in displacement distance between near, middle and far views, it is divided into three clusters, so K=3. When the drone flies straight, the grayscale image is divided into regions of 10 degrees each, and then the regions are divided into three clusters according to near, middle and far views, so K=3×36=108.

[0093] Step S104: Based on the average displacement distance and inter-frame displacement angle of the true feature points in each cluster, determine the local optimal blur kernel of the cluster, use each local optimal blur kernel to deblur the local region corresponding to each cluster to obtain a clear image, and construct a visual 3D model of the obstacles in the clear image based on the clear image.

[0094] Specifically, in this embodiment of the invention, the local average displacement is calculated for each cluster partitioned according to the K-means algorithm. And local displacement angle. In one optional embodiment of the present invention, determining the local optimal fuzzy kernel of a cluster based on the average displacement distance and inter-frame displacement angle of the true feature points in each cluster includes: determining the local average displacement of the cluster based on the average displacement distance and the number of true feature points in each cluster; determining the average value of the inter-frame displacement angles of the true feature points as the displacement angle of the true feature points; determining the local displacement angle of the cluster based on the displacement angles of the true feature points in each cluster; and determining the local optimal fuzzy kernel based on the local average displacement and local displacement angle of the cluster.

[0095] Specifically, the embodiments of the present invention use the following formula to calculate the local average displacement of clusters:

[0096] ;

[0097] In the above formula, This represents the local average displacement of a cluster. Representing true feature points The average displacement distance. This represents the number of true feature points in a cluster.

[0098] Furthermore, in this embodiment of the invention, the local displacement angle of the cluster is calculated using the following formula:

[0099] ;

[0100] In the above formula, This represents the local displacement angle of the cluster. Representing true feature points The displacement angle. This indicates the number of true feature points in a cluster.

[0101] Furthermore, as an optional embodiment of the present invention, determining the locally optimal fuzzy kernel based on the local average displacement and local displacement angle of the cluster includes: determining the edge length of the fuzzy kernel of the cluster based on the local average displacement; determining the center coordinates of the fuzzy kernel of the cluster based on the edge length of the fuzzy kernel; and determining the location of the trailing image of the fuzzy kernel within the fuzzy kernel based on the local displacement angle and local average displacement. The first offset in direction, and in The second offset in the direction; using the center coordinates of the blur kernel, the first offset, and the second offset, determine the coordinates of the path pixels along the direction from the center of the blur kernel to the first and second offsets; determine the zero matrix according to the side length of the blur kernel and assign the first value to the position of the path pixel coordinates in the zero matrix, and take the second value for the other positions; normalize the zero matrix according to the number of path pixels to obtain the local optimal blur kernel.

[0102] Specifically, in this embodiment of the invention, the first value is 1, and the second value is 0. This embodiment first determines the size of the blur kernel based on the local average displacement D to adapt to the blur length. The side length S of the blur kernel can be expressed by the following formula:

[0103] ;

[0104] In the above formula, This represents the edge length of the fuzzy kernel. This represents the local average displacement. .

[0105] Furthermore, in this embodiment of the invention, the center coordinates of the fuzzy kernel of the cluster are determined using the following formula:

[0106] = ;

[0107] In the above formula, The x-coordinate represents the center coordinates of the fuzzy kernel of the cluster. The ordinate represents the center coordinates of the fuzzy kernel of the cluster. This represents the edge length of the fuzzy kernel.

[0108] Furthermore, embodiments of the present invention calculate the blur trail within the blur kernel. , The total offset in direction is as follows:

[0109] ;

[0110] ;

[0111] In the above formula, The trailing shadow of the blur kernel is within the blur kernel. The first offset in direction. The trailing shadow of the blur kernel is within the blur kernel. The second offset in direction. This represents the local average displacement. The cosine value represents the local displacement angle of the cluster. This represents the sine value of the local displacement angle of the cluster.

[0112] Furthermore, in this embodiment of the invention, pixel path points are generated specifically starting from the kernel center and following... Take in uniform direction The number of pixels (rounded up to ensure coverage of the ghosting), the first 1 pixel The coordinates are represented by the following formula:

[0113] ;

[0114] ;

[0115] In the above formula, This represents the x-coordinate of the j-th pixel. This represents the ordinate of the j-th pixel. The trailing shadow of the blur kernel is within the blur kernel. The first offset in direction. The trailing shadow of the blur kernel is within the blur kernel. The second offset in direction. The x-coordinate represents the center coordinates of the fuzzy kernel of the cluster. The ordinate represents the center coordinates of the fuzzy kernel of the cluster. This indicates the number of pixels taken.

[0116] Furthermore, embodiments of the present invention will Coordinates rounded to integers .

[0117] Furthermore, the initialization of this embodiment of the invention... The zero matrix In the zero matrix Assigning values ​​at the coordinates of the pixels in the middle path The remaining positions remain at 0.

[0118] Furthermore, the present invention provides a statistical zero matrix. The number of pixels in the middle path is denoted as Then for the zero matrix Normalization is performed, specifically by changing the value of each path pixel in the zero matrix to... Then the local optimal fuzzy kernel is obtained. .

[0119] Furthermore, by obtaining the locally optimal blur kernel in the local region through the above embodiments of the present invention, the blurred image can be deconvolved using the locally optimal blur kernel to achieve the effect of deblurring. However, direct deconvolution has serious drawbacks, such as noise amplification and loss of details. Wiener filtering is needed to find the optimal balance between "restoring a clear image" and "suppressing noise" to meet the real-time and anti-interference requirements of UAV operation.

[0120] Furthermore, embodiments of the present invention target local regions of grayscale images. and local fuzzy kernel Perform Fast Fourier Transform on each region to obtain the frequency domain representation of the local region. Frequency domain representation of local fuzzy kernel Then, using the Wiener filtering formula below, substitute... and :

[0121] ;

[0122] In the above formula, A frequency domain representation of a sharp image. yes The complex conjugate, yes The square of the modulus. For signal-to-noise ratio, take an empirical value. . Frequency domain representation of a local region. The frequency domain representation of the local fuzzy kernel.

[0123] Furthermore, the embodiments of the present invention obtain the above formula as follows: Perform a two-dimensional inverse FFT transform to obtain a clear image.

[0124] Furthermore, for the clear image after deblurring, the SfM algorithm can be used to extract and match image feature points in the clear image to calculate the camera pose; then, based on multi-view stereo matching, a dense point cloud is generated, and the point cloud is denoised, simplified, and hole-filled; then, a triangular mesh model is constructed from the point cloud, which maps the image texture to form a visualized 3D model of obstacles in the clear image.

[0125] The technical solution provided by this invention obtains the optimal blur kernel for a local region by analyzing feature points in multiple consecutive frames of grayscale images from a UAV. This local blur kernel can then be used to deconvolve the blurred image to achieve deblurring. However, direct deconvolution has serious drawbacks (such as noise amplification and detail loss). Wiener filtering is needed to find the optimal balance between "restoring a clear image" and "suppressing noise" to meet the real-time performance and anti-interference requirements of UAV operation. The average displacement distance and inter-frame displacement angle are calculated to quantitatively characterize the blurred state. Furthermore, after dividing the grayscale image into multiple regions, false feature points are filtered and eliminated based on the average displacement distance and inter-frame displacement angle of the aforementioned feature points. This effectively avoids noise and background interference, ensures the purity of true feature points, and solves the detail loss problem caused by motion blur at its source, laying a reliable foundation for subsequent deblurring processes. Furthermore, based on the average displacement distance and inter-frame displacement angle of the feature points of each cluster, a locally optimal blur kernel is determined. This kernel is then used to specifically deblur the corresponding regions, breaking the limitations of traditional single-kernel approaches for the entire image. While deblurring, it preserves details such as obstacle outlines and textures, resulting in high-quality, clear images. These clear images provide high-quality data for constructing visual 3D models of obstacles, ensuring the accuracy of model size and shape. Thus, the clear images retain details such as obstacle outlines and textures, avoiding interference with subsequent feature extraction and matching accuracy, and improving obstacle recognition accuracy.

[0126] Example 2:

[0127] Corresponding to the UAV-based obstacle 3D model reconstruction method provided in the above embodiments, based on the same technical concept, this invention also provides a UAV-based obstacle 3D model reconstruction system, which is used to execute the above UAV-based obstacle 3D model reconstruction method. Figure 2 A schematic diagram of another obstacle 3D model reconstruction system based on a UAV provided in one embodiment of the present invention is shown below. Figure 2 As shown. The obstacle 3D model reconstruction system based on UAVs can vary significantly due to differences in configuration or performance. It may include one or more processors 201 and memory 202. The memory 202 stores computer programs that can run on the processor 201. The processor 201 executes the programs stored in the memory 202 to achieve the above... Figure 1 The various steps in the method embodiment are described. The memory 202 can be temporary or persistent storage. The application stored in the memory 202 may include one or more modules (not shown in the figures), each module may include a series of computer-executable instructions for the UAV-based obstacle 3D model reconstruction system.

[0128] Furthermore, the processor 201 can be configured to communicate with the memory 202 and execute a series of computer-executable instructions stored in the memory 202 on the UAV-based obstacle 3D model reconstruction system. The UAV-based obstacle 3D model reconstruction system may also include one or more power supplies 203, one or more wired or wireless network interfaces 204, one or more input / output interfaces 205, and one or more keyboards 206.

[0129] Specifically, in this embodiment, the UAV-based obstacle 3D model reconstruction system includes a processor, a communication interface, a memory, and a communication bus; wherein, the processor, communication interface, and memory communicate with each other via the bus; the memory stores computer programs; and the processor executes the programs stored in the memory to achieve the above... Figure 1 The various steps in the method embodiments are the same as those in the above method embodiments, and have the same beneficial effects. To avoid repetition, the embodiments of the present invention will not be described again here.

[0130] It should be noted that the obstacle 3D model reconstruction system based on UAV provided in this embodiment of the invention and the obstacle 3D model reconstruction method based on UAV provided in this embodiment of the invention are based on the same application concept. Therefore, the specific implementation of this embodiment can refer to the implementation of the aforementioned obstacle 3D model reconstruction method based on UAV, and has the same or similar beneficial effects. Repeated parts will not be repeated.

[0131] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. The processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0132] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0133] This invention also proposes a computer-readable storage medium storing one or more programs, which, when executed by a UAV-based obstacle 3D model reconstruction system including multiple applications, cause the UAV-based obstacle 3D model reconstruction system to perform... Figure 1 The methods disclosed in the embodiments shown achieve the functions and beneficial effects of the methods in the preceding method embodiments, and will not be repeated here.

[0134] The computer-readable storage media include read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for reconstructing a 3D obstacle model based on an unmanned aerial vehicle (UAV), characterized in that, include: Acquire consecutive multi-frame grayscale images captured by the UAV, and determine the average displacement distance and inter-frame displacement angle of the feature points in the grayscale images based on the coordinates of the feature points in the grayscale images. The grayscale image is divided into multiple regions, and false feature points are filtered out based on the average displacement distance of each feature point in the region and the inter-frame displacement angle. The false feature points are then removed to obtain the true feature points. Based on the inter-frame displacement angle and the average displacement distance of the true feature points, each true feature point is clustered to obtain multiple clusters; Based on the average displacement distance and inter-frame displacement angle of the true feature points in each cluster, the local optimal blur kernel of the cluster is determined. The local regions corresponding to each cluster are deblurred using the local optimal blur kernel to obtain a clear image. Based on the clear image, a visual 3D model of the obstacles in the clear image is constructed.

2. The method for reconstructing a three-dimensional obstacle model based on an unmanned aerial vehicle (UAV) according to claim 1, characterized in that, The step of determining the average displacement distance and inter-frame displacement angle of the feature points in the grayscale image based on their coordinates in the grayscale image includes: Based on the coordinates of the feature points in each frame's grayscale image in the grayscale images of adjacent frames, determine the inter-frame displacement distance and inter-frame displacement angle of the feature points; The average displacement distance of the feature point in each frame of the grayscale image is determined based on the inter-frame displacement distance of the feature point.

3. The method for reconstructing a three-dimensional obstacle model based on an unmanned aerial vehicle (UAV) according to claim 1, characterized in that, The step of filtering false feature points based on the average displacement distance of each feature point in the region and the inter-frame displacement angle includes: The average displacement reference of the region is determined based on the average displacement distance of each feature point in the region and the number of feature points. Based on the average displacement benchmark and the average displacement distance, false feature points are screened out and removed from each of the feature points to obtain the remaining feature points; Based on the inter-frame displacement angle of the remaining feature points, false feature points are filtered out from the remaining feature points, and the false feature points are removed from the remaining feature points to obtain the true feature points.

4. The method for reconstructing a three-dimensional obstacle model based on an unmanned aerial vehicle (UAV) according to claim 3, characterized in that, The process of filtering false feature points based on the average displacement benchmark and the average displacement distance includes: Calculate the absolute value of the first difference between the average displacement reference and the average displacement distance; Feature points whose absolute value of the first difference is greater than the first threshold are identified as false feature points.

5. The method for reconstructing a three-dimensional obstacle model based on an unmanned aerial vehicle (UAV) according to claim 3, characterized in that, The step of filtering false feature points from the remaining feature points based on the inter-frame displacement angle of the remaining feature points includes: When the UAV is in a straight flight state, determine the absolute value of the second difference of the inter-frame displacement angle between the remaining feature points; If the absolute value of the second difference of the remaining feature points is greater than the second threshold, the remaining feature points are determined to be false feature points. When the UAV is in a side-flying state, the average value of the inter-frame displacement angle of the remaining feature points is determined to be the displacement angle of the remaining feature points. The average displacement angle is determined based on the displacement angle of each of the remaining feature points. Based on the coordinates of the remaining feature points and the coordinates of the center point of the grayscale image, determine the theoretical displacement angle of the remaining feature points; The flight attitude identifier of the UAV is determined based on its flight status; Based on the average displacement angle, the theoretical displacement angle, and the flight attitude identifier, the adaptive theoretical angle of the feature point is determined; Feature points whose absolute value of the third difference between the displacement angle of the remaining feature points and the adaptive theoretical angle is less than the third threshold are designated as false feature points.

6. The method for reconstructing a three-dimensional obstacle model based on an unmanned aerial vehicle (UAV) according to claim 1, characterized in that, The step of clustering the true feature points based on their inter-frame displacement angle and average displacement distance to obtain multiple clusters includes: The displacement angle of the true feature point is determined based on the inter-frame displacement angle of the true feature point. The displacement angle is converted into a two-dimensional vector, and the average displacement distance of the true feature point is normalized to obtain the normalized displacement distance. The feature vector of the true feature point is determined based on the two-dimensional vector and the normalized displacement distance. Based on the feature vectors of the true feature points and the flight attitude identifier of the UAV, the true feature points are clustered to obtain multiple clusters.

7. The method for reconstructing a three-dimensional obstacle model based on an unmanned aerial vehicle (UAV) according to claim 6, characterized in that, The clustering of the true feature points based on their feature vectors and the UAV's flight attitude identifier yields multiple clusters, including: The target number of clusters is determined based on the flight attitude identifiers of the UAV; Based on the feature vectors of the true feature points, each true feature point is clustered to obtain the target number of clusters.

8. The method for reconstructing a three-dimensional obstacle model based on an unmanned aerial vehicle (UAV) according to claim 1, characterized in that, The step of determining the local optimal blur kernel of each cluster based on the average displacement distance and inter-frame displacement angle of the true feature points in each cluster includes: The local average displacement of each cluster is determined based on the average displacement distance of the true feature points in each cluster and the number of the true feature points. The average value of the inter-frame displacement angle of the true feature point is determined to be the displacement angle of the true feature point. The local displacement angle of each cluster is determined based on the displacement angle of the true feature points in each cluster. Based on the local average displacement of the cluster and the local displacement angle, a local optimal fuzzy kernel is determined.

9. The method for reconstructing a three-dimensional obstacle model based on an unmanned aerial vehicle (UAV) according to claim 8, characterized in that, The determination of the local optimal fuzzy kernel based on the local average displacement of the cluster and the local displacement angle includes: The fuzzy kernel edge length of the cluster is determined based on the local average displacement. The center coordinates of the fuzzy kernel of the cluster are determined based on the side length of the fuzzy kernel; Based on the local displacement angle and the local average displacement, the trailing shadow of the blur kernel is determined within the blur kernel. The first offset in direction, and in The second offset in direction; Using the center coordinates of the blur kernel, the first offset, and the second offset, determine the coordinates of the path pixels along the direction from the center of the blur kernel to the first offset and the second offset; A zero matrix is ​​determined based on the side length of the fuzzy kernel, and a first value is assigned to the position of the path pixel in the zero matrix, while the other positions are assigned a second value, wherein the first value is greater than the second value. The zero matrix is ​​normalized based on the number of path pixels to obtain the local optimal blur kernel.

10. A three-dimensional obstacle model reconstruction system based on unmanned aerial vehicles (UAVs), characterized in that, include: Processor and memory; wherein the memory is used to store computer programs that can run on the processor; A processor is used to execute a program stored in memory to implement the steps of the obstacle 3D model reconstruction method based on UAV as described in any one of claims 1-9.

Images