Unmanned aerial vehicle positioning method and system based on machine vision

Abstract

The invention relates to the technical field of image processing, in particular to an unmanned aerial vehicle positioning method and system based on machine vision, and the method comprises the steps: obtaining a current frame image and a reference image in a real-time video stream of an unmanned aerial vehicle, generating an initial matching pair set, and calculating the structural consistency of each matching pair, the method comprises the steps of adaptively determining a screening threshold value of a current frame image based on structural consistency, determining a screening matching pair set by utilizing the screening threshold value, evaluating a positioning contribution weight of each matching pair of the screening matching pair set, executing weighted pose calculation based on the positioning contribution weights, and obtaining an instantaneous pose of an unmanned aerial vehicle in the current frame image. And inputting the instantaneous pose as an observation value into a time sequence filtering model, performing time sequence fusion in combination with a motion model of the unmanned aerial vehicle, and outputting the final pose estimation of the unmanned aerial vehicle in the current frame image so as to complete the accurate positioning of the unmanned aerial vehicle. The method improves the accuracy of unmanned aerial vehicle positioning.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a machine vision-based method and system for unmanned aerial vehicle (UAV) localization. Background Technology

[0002] Drones are commonly used in surveying, inspection and other fields. Their autonomous flight relies on high-precision real-time positioning capabilities. Drone positioning mainly depends on GNSS (Global Navigation Satellite System). In GNSS-denied environments such as between urban buildings, under bridges or indoors, drones must rely on technologies such as visual odometry or visual SLAM (simultaneous localization and mapping) to perform autonomous positioning by fusing data from onboard cameras and IMU (Inertial Measurement Unit).

[0003] In mainstream visual positioning technologies, a core step is to calculate pose by matching feature points of the current frame image with those of a reference image. Since the IMU inevitably accumulates drift when calculating pose by integrating its own motion, it is necessary to periodically introduce external absolute measurements for correction. The instantaneous pose calculated based on a single frame image plays the role of external absolute measurement. In the fusion framework, a Kalman filter is usually used as a temporal filtering model to fuse the high-frequency motion prediction values ​​acquired by the IMU with the low-frequency instantaneous pose calculated by vision. The core purpose is to use the instantaneous pose to continuously correct the accumulated drift of the IMU in order to output an accurate and continuous positioning trajectory during the flight of the UAV.

[0004] However, in traditional Kalman filters, the observation noise covariance matrix, which describes the uncertainty of visual measurements, is usually set to a fixed, offline-calibrated empirical value. This method of using a fixed observation noise covariance matrix cannot adapt to the drastic dynamic changes in visual measurement quality, leading to the failure of the correction process in different scenarios. Specifically, when the UAV is in an ideal scene with rich texture and well-defined features, the actual accuracy of the visual positioning result is very high. A fixed, conservative noise covariance matrix will cause the filter to have insufficient trust in high-quality visual data, resulting in an incomplete correction process. The cumulative drift of the IMU cannot be corrected most effectively, thus limiting the final positioning accuracy. Conversely, when the UAV flies over multiple areas with similar textures or experiences violent movement that causes image blurring, the actual accuracy of the visual positioning result will drop sharply. In this case, a fixed, optimistic noise covariance matrix will cause the filter to retain some trust in these low-quality data full of errors, leading to errors in the correction of the cumulative drift of the IMU.

[0005] In summary, this fixed noise covariance matrix can easily inject harmful measurement noise directly into the state estimation, causing jumps or drifts in the positioning trajectory and resulting in inaccurate UAV positioning. Summary of the Invention

[0006] To address the problem of inaccurate positioning caused by the inability of traditional UAV vision-based positioning fusion frameworks to adapt to dynamic changes in visual measurement quality due to the use of fixed observation noise covariance, this invention proposes a machine vision-based UAV positioning method and system.

[0007] In a first aspect, the present invention provides a machine vision-based method for UAV localization, comprising: Acquire a current frame image and a preset reference image from the real-time video stream of the drone. By detecting the first feature point set of the current frame image and the second feature point set of the reference image, establish the matching relationship between the first feature point set and the second feature point set, and generate an initial matching pair set. For each matching pair in the initial matching pair set, the structural consistency of the matching pair is calculated based on the geometric configuration of the local neighborhood feature points of the two feature points contained in the matching pair, so as to obtain the structural consistency set corresponding to the initial matching pair set. Based on the statistical characteristics of the structural consistency contained in the structural consistency set, the filtering threshold of the current frame image is adaptively determined. The initial set of matching pairs is filtered using a screening threshold to obtain a set of filtered matching pairs. The location contribution weight of each matching pair is determined based on the extent to which the structural consistency of each matching pair in the set exceeds the screening threshold, and the extent and the location contribution weight are positively correlated. Weighted pose calculation is performed based on the localization contribution weights of all matching pairs to obtain the instantaneous pose of the UAV in the current frame image. This instantaneous pose is then used as an observation and input into a temporal filtering model. The model is then combined with the UAV's motion model for temporal fusion to output the final pose estimate of the UAV in the current frame image, thereby completing the accurate localization of the UAV.

[0008] This technical solution acquires a snapshot of the drone's current view and a preset reference image. Through preliminary feature detection and matching, an initial set of matching pairs is established. By calculating the geometric similarity of each matching pair to its neighborhood environment in its respective image, a structural consistency is assigned to each pair. By analyzing the statistical features of all structural consistency data, a selection threshold best suited to the current scene complexity is adaptively determined. After filtering the initial set of matching pairs using this threshold, a refined localization contribution weight allocation is further performed. This approach assumes that not all data passing the selection are equally important; instead, a non-linear localization contribution is assigned to each matching pair based on the extent to which its structural consistency exceeds the selection threshold. The weighting ensures that matching pairs with better structural consistency and higher accuracy play a dominant role in the final localization. Based on these matching pairs with localization contribution weights, a weighted pose calculation is performed to obtain a high-quality instantaneous pose. Further considering that the instantaneous pose of a single frame image may still have instantaneous interference, a temporal consistency check is introduced. This high-quality instantaneous pose is used as an external observation and fed into a temporal filtering model that incorporates the UAV's own motion inertia. By comparing the external observation with the prediction based on historical trajectories, the model smooths out any abrupt changes that do not conform to physical laws, and finally outputs a final pose estimate that is both spatially accurate and temporally continuous, thus completing the accurate localization of the UAV.

[0009] Preferably, establishing a matching relationship between the first feature point set and the second feature point set to generate an initial matching pair set includes: detecting the current frame image and the reference image respectively using a feature point detection algorithm to obtain a first feature point set consisting of all feature points of the current frame image and a second feature point set consisting of all feature points of the reference image; processing the first feature point set and the second feature point set respectively through semantic segmentation to obtain a first semantic label for each feature point in the first feature point set and a second semantic label for each feature point in the second feature point set; and generating an initial matching pair set by matching only feature points with the same semantic label using a feature point matching algorithm based on the first semantic label and the second semantic label.

[0010] Preferably, the structural consistency of the matching pair is determined based on the following method: each matching pair contains two feature points located in the current frame image and the reference image, respectively; local neighborhood feature points are divided for each feature point using a nearest neighbor detection algorithm; the feature point is pointed to each local neighborhood feature point to construct a neighborhood vector; the horizontal rightward direction of the feature point is used as the reference direction; all neighborhood vectors and their corresponding local neighborhood feature points are sorted in ascending order of the angle between each neighborhood vector and the reference direction; for each feature point, the angle difference between the two neighborhood vectors at the same index is calculated; the structural consistency of the matching pair is evaluated based on the angle difference and the distance between the two feature points and their respective local neighborhood feature points.

[0011] Preferably, the structural consistency of the matching pairs satisfies the following relationship:

[0012] in, It is the first Structural consistency of each matching pair This refers to the number of local neighborhood feature points for each of the two feature points in the matching pair. The index of the local neighborhood feature point. For the first The matching pair contains the first two feature points. The angular difference of each neighborhood vector For the first The first feature point contained in the matching pair is related to its second feature point. The distance between local neighborhood feature points For the first The matching pair contains the second feature point and its second feature point. The distance between local neighborhood feature points For all The average value, and Both are the tolerance variances for angle differences and distance deviations, which are preset. It is a natural exponential function.

[0013] This technical solution uses a negative exponential function combined with two independent penalty terms to accurately evaluate the consistency of the local geometric configuration of each matching pair. The first penalty term takes into account the invariance of the UAV's own rotation and selects to analyze the consistency of the relative angular arrangement between the neighborhood vectors. The second penalty term is based on the distance between the feature point and its local neighborhood feature points, dynamically establishing a distance reference benchmark, which can accurately identify erroneous feature point matching caused by non-uniform geometric distortion in the local area of ​​the feature point.

[0014] Preferably, the filtering threshold of the current frame image is adaptively determined based on the statistical characteristics of the structural consistency contained in the structural consistency set, including: determining the information entropy of the current frame image through the probability distribution of each structural consistency in the structural consistency set, and using the high quantile and the medium quantile of the structural consistency set as two statistical anchor points; and performing linear interpolation between the two statistical anchor points based on the empirical cumulative distribution function value of the information entropy of the current frame image within a preset historical time window to generate the filtering threshold of the current frame image.

[0015] Preferably, the linear interpolation between two statistical anchor points to generate the filtering threshold for the current frame image is based on the following formula:

[0016] in, An adaptive filtering threshold for the current frame image. and These are two statistical anchor points, namely the high quantile and the median quantile of the structural consistency set. The high quantile and the median quantile were determined using the percentile method. This is the empirical cumulative distribution function value of the information entropy of the current frame image within a preset historical time window.

[0017] This technical solution constructs a threshold decision model driven by real-time data. It performs linear interpolation between two dynamically calculated statistical anchor points, and the resulting screening threshold is dynamically adaptive based on the quality of the current frame image. The weight of the linear interpolation is determined by the relative ranking of the complexity of the current frame image in its recent historical performance. This enables the threshold decision model to remain sensitive to occasional complexity in generally simple environments, ensuring that the screening threshold can automatically converge to an optimal balance point that guarantees both accuracy and robustness in any scenario.

[0018] Preferably, the empirical cumulative distribution function value of the information entropy of the current frame image within a preset historical time window is determined as follows: a historical time window is formed from the start of the UAV flight to the corresponding time of the current frame image; the information entropy of each frame image within the historical time window is obtained to form a historical information entropy sequence; the number of items in the historical information entropy sequence that are less than or equal to the information entropy of the current frame image is counted, and then divided by the length of the historical information entropy sequence to obtain the empirical cumulative distribution function value of the information entropy of the current frame image in the historical information entropy sequence.

[0019] Preferably, the method for determining the location contribution weight of a matching pair based on the extent to which the structural consistency of each matching pair in the set of matched pairs exceeds the screening threshold is as follows: for each matching pair, the extent to which the structural consistency of the matching pair exceeds the adaptive screening threshold is normalized, the normalization result is subjected to a positively correlated nonlinear mapping, and the obtained mapping value is used as the location contribution weight of the matching pair.

[0020] This technical solution constructs a nonlinear weight allocation mechanism, the core of which is to associate the weights with the relative superiority of the matching quality. Through normalization, the input is decoupled into a standardized interval independent of the threshold. Based on the nonlinear mapping, a strategic weight adjustment is introduced. For matching pairs with lower normalization values, the marginal gain of the weights is maximized, thereby effectively distinguishing and amplifying the influence of medium-quality matching pairs. For matching pairs with higher normalization values, the increase of the weights tends to be gradual, avoiding oversensitivity to small differences between high-quality matching pairs. This design allows the pose calculation process to more robustly rely on a core set of reliable matching pairs, while suppressing the uncertainty of those matching pairs whose quality is barely acceptable.

[0021] Preferably, before performing temporal fusion with the UAV's motion model, the following operations are performed: calculating the reliability index of the observations by combining the information entropy of the current frame image and the average positioning contribution weight of all matching pairs in the selected matching pair set, wherein the reliability index is negatively correlated with the information entropy and positively correlated with the average positioning contribution weight; using a Kalman filter as the temporal filtering model, determining a noise benchmark proportional to the prediction uncertainty covariance of the Kalman filter itself, and calculating a dynamic scaling factor based on the reliability index of the observations, wherein the magnitude of the dynamic scaling factor is negatively correlated with the reliability index of the observations; and dynamically adjusting the observation noise covariance in the Kalman filter by applying the dynamic scaling factor to the noise benchmark to achieve adaptive fusion between the observations and the predictions of the motion model.

[0022] This technical solution constructs a dual adaptive closed-loop control system, which achieves this by deeply coupling the adjustment of observation noise with the external observation quality and the internal state of the filter. It proportionally correlates the noise benchmark with the prediction uncertainty covariance of the Kalman filter itself, so that the noise expectation is dynamically correlated with the current internal uncertainty state of the filter. The observation noise covariance is adjusted by a dynamic scaling factor related to the observation quality. When the external observation quality is high, the confidence level is increased to effectively correct drift. When the external observation quality is poor, the confidence level is reduced to suppress the harmful measurement that affects the correction and ensure robustness, thereby achieving optimal state estimation in any scenario.

[0023] Secondly, the present invention also provides a machine vision-based drone positioning system, the drone positioning system including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of any of the drone positioning methods described above.

[0024] The present invention has the following effects: This invention introduces a real-time visual measurement quality diagnosis and adaptive confidence adjustment mechanism into the traditional visual fusion framework. First, it deeply refines and evaluates the initial matching data through semantic priors and local geometric structure verification. Then, based on the evaluation results, it not only adaptively generates screening thresholds driven by data to obtain the localization contribution weight of each matching pair, but also performs weighted pose estimation to evaluate the reliability index of the UAV's pose estimation in the current frame image. Finally, it dynamically couples the reliability index with the prediction uncertainty of the Kalman filter itself to adjust the observation noise covariance in real time. This solves the problem of IMU drift correction failure caused by using a fixed noise covariance, enabling the Kalman filter to fully utilize high-quality visual data to improve accuracy while avoiding interference from low-quality data, thereby improving the accuracy of UAV localization. Attached Figure Description

[0025] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0026] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0027] Reference Figure 1 A machine vision-based drone localization method includes the following steps: S1: Obtain the current frame and reference image and extract the initial matching pair.

[0028] To efficiently and distinctly extract structured feature information from the visual sensor input of the UAV and perform preliminary association matching to provide raw data input for subsequent refined quality assessment and pose calculation, this embodiment first acquires any frame image from the real-time video stream from the camera on the UAV as a current frame image, and obtains the corresponding reference image from a pre-built map or keyframe library. Then, feature point detection is performed on the current frame image and the reference image. In order to ensure the number of feature points while taking into account the real-time requirements of the airborne computing platform, this embodiment uses a fast corner detection algorithm to detect the current frame image and the reference image. The fast corner detection algorithm quickly identifies corner points by comparing the grayscale difference between a pixel and its neighboring pixels. Each corner point is treated as a feature point, thus generating the feature point set of the current frame image and the feature point set of the reference image, which are respectively denoted as the first feature point set and the second feature point set.

[0029] The current frame image and the reference image together form the basis of pose calculation. The current frame image refers to a two-dimensional image that is acquired in real time from the UAV's camera and represents its current viewpoint. The feature points in this image only contain two-dimensional pixel coordinate information. The reference image, on the other hand, comes from a pre-built map or keyframe library with three-dimensional spatial information. Its core value lies in the fact that each feature point in the image corresponds precisely to a known three-dimensional coordinate in the world coordinate system. By matching the two-dimensional feature points of the current frame image with the corresponding three-dimensional feature points in the reference image, a series of positional correspondences are established, enabling the system to calculate the instantaneous pose of the UAV based on these correspondences.

[0030] To suppress mismatches caused by scene semantic inconsistency at the source, this embodiment introduces semantic segmentation as a strong prior constraint. Specifically, the current frame image and the reference image are input in parallel into a pre-trained lightweight semantic segmentation network, such as the MobileUNet semantic segmentation model. This semantic segmentation model assigns a predefined category label, such as building, road, vegetation, etc., to each pixel of the two images. Since a feature point is essentially a pixel, the semantic label of each feature point has already been obtained.

[0031] For each feature point with a semantic label, its BRIEF feature descriptor is calculated, and initial matching is performed. The matching process follows a strict same-class matching principle, specifically: For any feature point in the first feature point set, feature points with the same semantic label as that feature point are selected from the second feature point set to form a set of feature points to be matched. The Hamming distance between the BRIEF descriptor of that feature point and the BRIEF descriptors of each feature point in the set of feature points to be matched is calculated. Feature points that match that feature point are determined according to the minimum Hamming distance, thus establishing a matching relationship between the first and second feature point sets. Based on this matching relationship, each feature point in the first feature point set and each matching feature point in the second feature point set form a matching pair. Finally, all feature points in the first feature point set are traversed and the matching process is repeated to form an initial set of matching pairs. This semantic prior-based matching strategy can eliminate a large number of logically unreasonable potential matching feature points, improving the purity of the initial matching set.

[0032] S2: Evaluate the consistency of the local neighborhood structure of each matching pair.

[0033] This step aims to address the problem that initial matching pairs lack geometric constraints based solely on feature descriptor similarity. To this end, this invention proposes a mechanism for evaluating the similarity of the local neighborhood geometric configurations of matching pairs, generating a structural consistency for each matching pair that reflects its geometric reliability.

[0034] Specifically: First, for the first matching pair in the initial set... A matching pair is formed by a feature point located in the current frame and a feature point located in the reference image, denoted as the first feature point and the second feature point, respectively. The nearest neighbor detection algorithm is used to find five local neighbor feature points for the first feature point and five local neighbor feature points for the second feature point. In this embodiment, five local neighbor feature points are selected, achieving a good balance between ensuring geometric stability and computational efficiency. Setting the number of local neighbor feature points to five is based on the consideration that too few (e.g., less than three) would lead to overly simple local geometric configurations, easily causing ambiguity and sensitivity to noise; while too many would increase the computational burden and may introduce irrelevant distant feature points, affecting real-time performance. The number of local neighbor feature points can be adjusted according to image resolution and feature point density. In low-resolution or sparse feature scenarios, it can be appropriately reduced to three to ensure real-time performance; in high-resolution or dense feature scenarios, it can be increased to seven or eight to improve the stability of the structural description.

[0035] Subsequently, the first feature point is linked to its five local neighboring feature points, forming five neighborhood vectors for the first feature point. Using the horizontal-to-right direction of the first feature point as the reference direction, the angle between each neighborhood vector and the reference direction is obtained by dividing the angle counter-clockwise. At this point, one neighborhood feature point corresponds to one neighborhood vector, and one neighborhood vector corresponds to one angle. Following the same processing method as the first feature point, the five neighborhood vectors and their angles for the second feature point are obtained.

[0036] The five neighborhood vectors of each feature point are sorted in ascending order of the included angle, numbered from 1 to 5. The structural consistency of the matching pair is evaluated based on the difference in the included angle between two neighborhood vectors at the same number and the distance between the two feature points and their respective local neighborhood feature points. A correct matching pair should contain two feature points whose local geometric structures should maintain a high degree of similarity between the two images. The better the feature point matching, the better the result, and vice versa.

[0037] Specifically, it follows the following relationship:

[0038] in, It is the first Structural consistency of each matching pair This refers to the number of local neighborhood feature points for each of the two feature points in the matching pair. The index of the local neighborhood feature point. For the first The matching pair contains the first two feature points. The angular difference of each neighborhood vector For the first The first feature point contained in the matching pair is related to its second feature point. The distance between local neighborhood feature points For the first The matching pair contains the second feature point and its second feature point. The distance between local neighborhood feature points For all The average value, and Both are the tolerance variances for angle differences and distance deviations, which are preset. It is a natural exponential function.

[0039] In this relation: It is the mean term, for The consistency evaluation results of each neighborhood feature point are averaged to eliminate outlier interference from individual neighborhood feature points. The Gaussian decay results of each neighborhood feature point are averaged, which is equivalent to further smoothing the extreme Gaussian decay values ​​of individual points, avoiding outliers from dominating the final result. This allows the structural consistency to retain the advantage of Gaussian decay in accurately penalizing differences, while reducing the impact of random errors through the mean, resulting in a more stable consistency evaluation result. An angle penalty term is used to transform the angular differences of neighborhood vectors into a penalty term that conforms to Gaussian decay, thus addressing the matching bias caused by UAV rotation. This directly quantifies the squared error of the angle difference deviating from the ideal value of 0, and is the core of constructing the Gaussian penalty. This squared term ensures that the penalty value is non-negative, and the smaller the angle difference, the closer the penalty is to 0, which perfectly conforms to the characteristic of the Gaussian distribution to penalize around the ideal value. Dividing by This is a typical variance normalization operation in a Gaussian distribution. Corresponding to the variance of the Gaussian distribution, the larger the variance, the smaller the penalty term for the same angle difference, the smoother the Gaussian decay, and the higher the tolerance for angle differences. Conversely, the smaller the variance, the larger the penalty term, the steeper the Gaussian decay, and the more sensitive to angle differences. By following the path from angle difference to Gaussianization penalty to exponential decay, the angle consistency score conforms to the Gaussian distribution: small angle differences only cause slight decay, retaining reasonable matches; large angle differences cause drastic decay, eliminating mismatches, and improving resistance to rotation interference. The distance penalty term transforms the difference in distance ratio between two feature points and their neighboring feature points into a Gaussian penalty term, addressing image scaling interference caused by the distance of the drone. This, along with the angle term, forms a two-dimensional Gaussian constraint. , Equivalent to the mean of a Gaussian distribution, the squared term ensures that the difference from the mean is non-negative, and conforms to the core logic of the Gaussian distribution: "the farther from the mean, the lower the probability density." The closer the distance proportion is to the mean, the smaller the penalty. This corresponds to the center of the Gaussian distribution, then divided by... , It represents the tolerance variance for the degree of distance deviation, corresponding to the variance of a Gaussian distribution, and controls the decay rate of the distance ratio difference. The larger the value, the smoother the Gaussian decay, making it suitable for scenes with sparse textures and large scaling fluctuations. The smaller the value, the steeper the attenuation, making it suitable for scenes with dense textures and stable scaling. Gaussianization ensures that the distance ratio consistency score also follows a Gaussian law. Even with slight scaling of the drone, as long as the distance ratio remains close to the mean, it maintains a low penalty and high attenuation retention value, avoiding misjudging high-quality matching pairs due to scaling. The exponential mapping term transforms the total Gaussian penalty term for angle and distance into a consistency term within the 0-1 range. This is a direct manifestation of Gaussian decay. The total penalty term equals the sum of the angle Gaussian penalty term and the distance ratio penalty term, essentially representing the superposition of the two-dimensional deviations. When the total penalty term equals 0, there is no difference between the two dimensions. Due to the negative sign before the total penalty term, the exponential mapping term reaches its maximum value of 1, resulting in the best consistency. As the total penalty term increases, the exponential mapping term monotonically decreases along a Gaussian curve and approaches 0. The greater the deviation, the lower the consistency score. Through Gaussian decay, a smooth mapping is achieved where smaller differences lead to better consistency.

[0040] in, and One method for determining this is as follows: Since both are variances of the normal distribution that the geometric differences of the correct matching pair follow in the current scene, for a correct matching pair, its angular difference... Deviation of distance ratio It should approximately follow a zero-mean normal distribution. and Essentially, the variance of this normal distribution reflects the dispersion of the geometric differences between correct matching pairs. The more stable the scene and the lower the noise, the smaller the variance; conversely, the more complex the scene and the higher the noise, the larger the variance. Therefore, the structural consistency of all matching pairs in the initial matching pair set is sorted in ascending order, and the quartile method is used to construct a reliable matching pair set from matching pairs with structural consistency greater than the third quartile. The angular difference of each matching pair in the reliable matching pair set is then obtained. Difference between distance ratios Calculate all matching pairs unbiased variance as All matching pairs unbiased variance as If there are 100 matching pairs, then a total of 500 will be obtained. and 500 , these 500 unbiased variance as , these 500 unbiased variance as .

[0041] In summary, the essence of this relationship is a three-layer structure of two-dimensional Gaussian penalty, exponential decay mapping, and mean smoothing, ultimately outputting a structural consistency score in the 0-1 interval. The core carrier of Gaussian decay is the exponential term, which fully follows the characteristic of the Gaussian function that "the greater the deviation from the benchmark value, the more significant the decay." The smaller the total penalty term of the exponential term, the smaller the difference in geometric configuration, and the closer the value after Gaussian decay is to 1, indicating high consistency. Conversely, the larger the total penalty term, the greater the difference in geometric configuration, and the closer the value after Gaussian decay is to 0, indicating low consistency. This is the key to accurately distinguishing between high-quality and low-quality matching pairs.

[0042] A higher structural consistency (closer to 1) indicates a highly isomorphic local neighborhood geometry between the two feature points, signifying a high-quality, high-confidence match. Conversely, a lower structural consistency (closer to 0) suggests significant differences in local geometry, potentially indicating a mismatch due to texture similarity. Following this method, by obtaining the structural consistency of each match in the initial set, a structural consistency set corresponding to the initial set of match pairs is obtained.

[0043] S3: Generate screening thresholds based on statistical features of structural consistency.

[0044] To better adapt to scene changes in images, a fully data-driven dynamic threshold generation mechanism is proposed. This mechanism adaptively determines the filtering threshold for the current frame image based on the statistical characteristics of structural consistency contained in the structural consistency set. Essentially, this filtering threshold directly correlates the reliability of pose calculation with the physical world features observed by the UAV. The raw data for UAV positioning originates from scene images captured by its camera; therefore, the richness of the scene's features directly determines the quality of the visual information available for positioning. In scenes with rich textures and complex structures (such as urban building complexes), the UAV can acquire a large number of high-quality, highly discriminative feature points, resulting in high information entropy. In this case, the filtering threshold is appropriately relaxed to include a sufficient number of effective matching pairs for robust statistical calculation, ensuring the robustness of positioning. Conversely, in scenes with sparse textures and degraded features (such as open water surfaces or large walls), high-quality feature points are scarce, resulting in low information entropy. The filtering threshold is tightened, retaining only the highest-confidence matching pairs with the strictest criteria. The core purpose is to prevent fuzzy or incorrect matching caused by scene degradation from contaminating the final pose calculation.

[0045] Specifically, it includes: First, real-time statistical analysis is performed on the structural consistency set, and the high quantiles of the score set are obtained using the percentile method. (90th percentile) and median The median was chosen as one of the two statistical anchors. The 90th percentile was selected to establish the statistical anchor in the top interval of the data distribution with the best quality, ensuring the strictness of the screening benchmark. Practice has shown that the 90th percentile can effectively remove outliers in most scenarios while retaining enough high-quality matching pairs.

[0046] Simultaneously, the information entropy of the structural consistency set is calculated and used as the information entropy of the current frame image. A historical time window is formed by considering the first frame of the UAV's real-time video stream up to the frame preceding the current frame. Following the method used to acquire the information entropy of the current frame image, the information entropy of each frame within the historical time window is acquired, forming a historical information entropy sequence. The number of elements in the historical information entropy sequence that are less than or equal to the information entropy of the current frame image is counted, and then divided by the length of the historical information entropy sequence to obtain the empirical cumulative distribution function value of the current frame image's information entropy in the historical information entropy sequence. Using all historical data can provide the most stable statistical baseline for calculating the empirical cumulative distribution function, avoiding fluctuations caused by short-term dramatic changes in the scenario.

[0047] Finally, a linear interpolation is performed between the two statistical anchor points using the empirical cumulative distribution function value to generate the filtering threshold for the current frame image. :

[0048] in, The filtering threshold for the current frame image. and These are two statistical anchor points, namely the high quantile and the median quantile of the structural consistency set. The high quantile and the median quantile were determined using the percentile method. This represents the empirical cumulative distribution function value of the information entropy of the current frame image within the historical time window.

[0049] In this relationship, the lower the information entropy of the current frame image, the sparser the scene texture. The smaller it is, the closer it is to 0, making Approaching high quantile More stringent selection criteria are adopted; conversely, the higher the information entropy of the current frame image, the richer the scene texture and the more complex the details. The larger it is, the closer it is to 1, making Approaching the median site A more inclusive standard is adopted to ensure that there are enough matching pairs for pose calculation.

[0050] S4: Filter the initial set of matching pairs based on the filtering threshold and calculate the location contribution weight.

[0051] This step aims to purify the initial set of matching pairs using a screening threshold and to evaluate the contribution weight of the selected matching pairs in the localization process.

[0052] First, use a filtering threshold. Filter all initial matching pairs and retain only those pairs with a structural consistency greater than 1. The matching pairs form the filter matching pair set, using As the index of the matching pair in the set of filtering matching pairs, obtain the structural consistency of each matching pair in the set of filtering matching pairs. Normalize the magnitude of the structural consistency of the matching pair that exceeds the adaptive filtering threshold to obtain a normalized value. Apply a positively correlated nonlinear mapping to the normalized value and use the obtained mapping value as the localization contribution weight of the matching pair.

[0053] Specifically, the nonlinear mapping is performed through the following relation:

[0054] in, It is the first in the set of matching pairs to be filtered. The positioning contribution weight of each matching pair. It is a sine function. It is the first in the set of matching pairs to be filtered. Structural consistency of each matching pair As the filtering threshold, It is a parameter to prevent the denominator from being zero, and is usually a very small positive number. .

[0055] In this relation, This is the normalization term; the numerator is the magnitude of exceeding the screening threshold, and the denominator is the maximum possible magnitude of exceeding the threshold. hour, , The sine function in It has a monotonically increasing property within the interval, when Slightly larger When the screening criteria are just met, the value of the sine function is small, approaching 0. The positioning contribution weight of such a matching pair is very small. The closer the sine function is to 1, the larger its value, and the greater its contribution to the localization of this matching pair, with a maximum value of 1. Simultaneously, the slope of the sine function... The upper part gradually decreases, as As the value increases, the slope of the sine function decreases from 1 to 0, meaning the growth rate of the positioning contribution weight slows down. Specifically, when the magnitude exceeding the screening threshold is small, In When the weight of the positioning contribution increases rapidly and exceeds the screening threshold by a large margin, In Weight growth slowed, even near The weight will not increase dramatically.

[0056] This nonlinear adjustment based on the sine function ensures that matching pairs with structural consistency just above the screening threshold receive smaller weights, while matching pairs with scores far above the screening threshold receive larger weights. However, the growth rate of the weights slows down as the scores approach the upper limit. This design avoids the excessive influence of individual extremely high structural consistency matching pairs on the solution results, thereby improving the robustness of pose solution.

[0057] Furthermore, this invention is not limited to using a sine function. Any nonlinear function that exhibits monotonically increasing and decreasing marginal gain characteristics in the interval [0,1], such as the Sigmoid function or a partial tangent function, can be used to implement the nonlinear weight allocation operation of this invention.

[0058] S5: Perform weighted pose calculation based on the localization contribution weights of all matching pairs to obtain the instantaneous pose of the UAV in the current frame image.

[0059] In existing pose calculation methods based on PnP (Perspective n-Point) or ICP (Iterative Closest Point), all matching feature points are usually considered to be of equal confidence. That is, when constructing the cost function, the reprojection error terms of each matching point have the same weight. Based on this, this invention introduces the localization contribution weight obtained from structural consistency evaluation and embeds this weight as a weighting coefficient of the error term into the traditional pose calculation formula. This achieves weighted amplification of high-confidence matching points and suppression of low-confidence matching points. This approach does not change the pose calculation framework, but only optimizes the weight allocation of each term in the cost function. Therefore, it can be seamlessly integrated into the pose solver.

[0060] Specifically, the essence of traditional pose calculation is to solve the current pose of the UAV by analyzing a series of matching pairs. A matching pair consists of a 3D point with a known position in the world coordinate system and a 2D point observed in the current camera image. Corresponding to the current scene, the observed 2D point is the feature point in the current frame image of the matching pair, denoted as the first feature point of the matching pair. The 3D point with the known position is the feature point in the reference image of the matching pair, denoted as the second feature point of the matching pair. The goal of pose calculation is to calculate the optimal camera pose by minimizing the error between the theoretical projected position of the second feature point of all matching pairs and the actual observed position of the first feature point, i.e., the reprojection error. Its mathematical form is usually expressed as:

[0061] in, The sequence number of the matching pair. The total number of matching pairs. For the first The coordinates of the first feature point of each matching pair, i.e., the two-dimensional coordinates of the feature point located in the current frame image. For the first The coordinates of the second feature point of each matching pair, i.e., the three-dimensional coordinates of the feature point located in the reference image. It is a translation vector. The sign for calculating the distance between coordinates. and These are the camera's rotation matrix and translation vector, which together constitute the drone's current pose and are the unknowns that this formula needs to solve for. Yes By rotation Peaceful relocation After transformation, Completely transform to the camera coordinate system. This involves transforming the coordinates to the camera coordinate system using the camera's intrinsic parameter matrix. Perform perspective projection transformation onto the camera's virtual imaging plane, when When followed by parentheses, it indicates a projection function. When it appears in trigonometric functions or mathematical operations, it represents pi (π). It uses a projection function to transform the coordinates on the virtual imaging plane into the final two-dimensional pixel coordinates, thus obtaining... The theoretical projection position in the current pose. This represents the distance between the actual observed position of the first feature point and the theoretical projected position of the second feature point. The larger the distance, the larger the error. The square of the distance is then calculated to obtain the reprojection error of the matching pair. The reprojection errors of all matching pairs are summed up and used as the matching cost function. The smaller the sum, the better the matching result conforms to the geometric constraints. Therefore, by minimizing the matching cost function, we obtain... and This allows us to obtain the instantaneous pose of the drone in the current frame image.

[0062] Instantaneous pose refers to the complete state description of a drone in three-dimensional space at a specific instant, like a snapshot in a photograph. It includes the drone's position and orientation, specifically comprising two parts: position information, describing the drone's exact location in three-dimensional space, usually represented by a three-dimensional coordinate system relative to a fixed world coordinate system (e.g., the takeoff point or the origin of a map); and attitude information, describing the drone's orientation or rotation state. There are various mathematical representations, the most common being Euler angles, which are the most intuitive and include three independent rotation angles: roll (left-right roll along the axis of travel); pitch (up-down pitch of the nose); and yaw (left-right yaw of the nose).

[0063] Traditional pose estimation treats all matching points as equivalent contributions, and the error of each matching pair is treated equally. In this invention, the positioning contribution weights are directly used as weighting coefficients for the error term and inserted into the objective function, corresponding to the following formula positions:

[0064] in, To filter the first matching pair in the set The localization contribution weight of each matching pair, and the meaning of the remaining characters are consistent with the meaning in the mathematical form of traditional pose calculation.

[0065] In summary, this invention does not change the mathematical framework of pose calculation itself, but introduces a matching confidence weight into the matching cost function of traditional pose calculation. This enhances the quality of matching and suppresses potential mismatches, thereby improving the stability and accuracy of pose calculation without changing the algorithm structure.

[0066] S6: Evaluate the visual observation reliability index of the instantaneous pose of the UAV in the current frame image, and combine the motion model to dynamically update the observation noise covariance to output the final positioning trajectory of the UAV.

[0067] The instantaneous pose of the UAV in the current frame image is used as the observation value and input into a temporal filtering model. The model is then combined with the motion model of the UAV to perform temporal fusion, so as to output the final pose estimate of the UAV in the current frame image and complete the accurate localization of the UAV.

[0068] Specifically, the high-quality instantaneous visual pose is fused with the motion model provided by the UAV's own IMU (Inertial Measurement Unit) in a time sequence to output a smooth and accurate positioning trajectory.

[0069] The observation noise covariance matrix in a Kalman filter is explained as follows: This matrix is ​​a core parameter in the Kalman filter framework. Its function is to quantitatively describe the uncertainty or noise level of the external sensor measurements themselves. It represents the degree of distrust the filter has in the visual pose calculation results. A small observation noise covariance matrix means that the filter is told to have a high degree of trust in the current visual measurement; conversely, a large observation noise covariance matrix indicates that the filter should reduce its trust in the visual measurement and rely more on its own motion prediction. In traditional methods, the observation noise covariance matrix is ​​usually a fixed value set empirically.

[0070] This step achieves dynamic updating of the observation noise covariance at the current moment through the following process, and outputs the final positioning trajectory of the UAV: The first step is to calculate the reliability index of the visual observation. This index is constructed based on the following: a reliable visual observation requires not only high internal matching quality but also that the scene itself provides sufficiently rich and stable information. Combining the two dimensions of scene richness and matching quality, the credibility of the current visual observation is evaluated (from 0 to 1, the closer to 1, the more reliable). Specifically, this is based on the following relationship:

[0071] in, It is the drone in the A visual observation reliability index for instantaneous pose in frame images. The grayscale entropy of the current frame image reflects the richness of the scene's texture. The higher the value, the more complex and detailed the scene, providing a more reliable environmental basis for visual positioning. s is the number of grayscale histogram bins when calculating information entropy (the default value in engineering is 256, corresponding to an 8-bit grayscale image). It is the theoretical maximum information entropy. hour, Its function is to Normalize to the interval between 0 and 1. It is the normalized information entropy, reflecting whether the current frame image can provide reliable visual information. The closer the value is to 1, the richer the texture of the current frame image and the more reliable the environmental basis for visual observation. It is the mean of the location contribution weights of all matching pairs in the set of reliable matching pairs. It reflects how good the quality of the matching pair itself is. The closer it is to 1, the higher the overall quality of the matching pair and the better the data quality of visual observation. This means that if the texture of the current frame image is richer, and the feature point matching quality between the current frame image and the reference image is higher, then... The larger the value, and the closer it is to 1, the more reliable the current visual observation is, and the more trustworthy it is for correcting IMU drift; conversely, the sparser the texture of the current frame image, and the worse the feature point matching quality between the current frame image and the reference image, the lower the value. The closer it is to 0, the less reliable the visual observation is, and the less trustworthy it is for correcting IMU drift. Its impact on filtering and fusion should be reduced.

[0072] The second step is based on the drone's... The visual observation reliability index of instantaneous pose in frame image and the prediction uncertainty of Kalman filter itself are used to dynamically update the observation noise covariance matrix at the current moment, realizing the logic that the more reliable the vision, the higher the trust level.

[0073] Specifically:

[0074] in, It is the drone in the The observation noise covariance matrix at the corresponding time point of the frame image. It is the drone in the A visual observation reliability index for instantaneous pose in frame images. This is the observation matrix, which maps the UAV's system state (such as position, attitude, velocity, etc.) to the visual observation space. For example, if the system state is 6-dimensional (3-dimensional position + 3-dimensional attitude), and the visual observation also outputs a 6-dimensional pose, then... It can be set as a 6×6 identity matrix. If the dimensions of visual observation and system state are different, Conversion rules need to be designed based on the sensor model (such as camera projection relationships). It is the prior prediction covariance matrix, generated from the UAV's motion model (such as IMU integration), representing the covariance based on... The uncertainty in predicting the state at time t is the state at time t. The larger the value, the less reliable the prediction results of the motion model are (e.g., when the IMU has cumulative drift). It will gradually increase. It will be by The uncertainty of the motion model prediction in the state space, as described, is determined by the observation matrix. and its transpose matrix Projected onto the observation space, because This describes the uncertainty in the system's state space, while visual observation is a measurement of the observation space; therefore, through... To achieve the mapping from state to observation, and through transpose matrix This approach achieves an inverse mapping from observation to state, projecting the uncertainty of the state space onto the observation space. It ensures consistency in dimension and physical meaning with the observation noise covariance, satisfying the mathematical requirements of Kalman filtering. Here, the state is the filter's internal comprehensive judgment of all UAV conditions (such as speed), the observation is partial external evidence (instantaneous pose) provided by the vision system, and the observation matrix... It's a converter that connects these two, allowing them to be compared.

[0075] In this formula, This is a visual distrust ratio factor, which quantifies the degree of distrust towards the current visual perception. The higher, The smaller the value, the lower the distrust level, indicating that the filter trusts the visual observation results more. (Noise covariance matrix) The smaller. Conversely, The lower, The larger the value, the higher the distrust level, indicating that the filter distrusts the visual observation results more. (Noise covariance matrix) The larger. The larger the prior prediction covariance of a motion model (such as an IMU), the less reliable the motion model's predictions are (e.g., IMUs exhibit cumulative drift). This can be verified through the observation matrix. and its transpose The operation projects the uncertainty of the motion model in the state space onto the measurement space of visual observation, ensuring consistency with the dimension and physical meaning of the observation noise covariance.

[0076] This formula will use a... The negatively correlated distrust scaling factor is multiplied by the projection of the filter's own prediction uncertainty into the observation space to obtain a dynamically updated observation noise covariance matrix at the current moment. This allows the Kalman filter to intelligently balance visual accuracy and motion model robustness. When the vision is good, visual drift is corrected; when the vision is poor, the motion model is used to maintain continuity, ultimately outputting a smoother, more accurate, and robust UAV positioning trajectory.

[0077] The third step involves using the updated observation noise covariance matrix to complete state fusion and generate the positioning trajectory. This yields the dynamic... Then, the Kalman filter performs its standard update steps, first calculating the Kalman gain. Its size is The gain is inversely proportional to the visual measurement value and the IMU prediction value. Then, the gain is used to weight and fuse the visual measurement value and the IMU prediction value to obtain the optimal posterior state estimate at the current moment, that is, the corrected UAV position, attitude and velocity.

[0078] This process repeats continuously in every frame as the drone flies, depending on the reliability of visual observation. High time, Therefore, it decreases, resulting in a decrease in Kalman gain. Increasing the filter's reliability allows it to place more trust in visual results, effectively correcting the accumulated drift of the IMU. When low, Therefore, it increases, resulting in a Kalman gain. By reducing the filter's reliance on short-term IMU predictions, the interference of poor visual measurements on state estimation is effectively suppressed.

[0079] Ultimately, the filter generates an optimally fused posterior state estimate point in each frame, which is the most accurate UAV state at the current moment. The position information of this series of continuously output state estimate points together constitutes the complete, smooth and high-precision UAV positioning trajectory generated by this invention. Through the coordinated work of these steps, the accuracy and reliability of UAV autonomous positioning in various complex and dynamically changing scenarios are improved.

[0080] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A machine vision-based UAV localization method, characterized in that, include: Acquire a current frame image and a preset reference image from the real-time video stream of the drone. By detecting the first feature point set of the current frame image and the second feature point set of the reference image, establish the matching relationship between the first feature point set and the second feature point set, and generate an initial matching pair set. For each matching pair in the initial matching pair set, the structural consistency of the matching pair is calculated based on the geometric configuration of the local neighborhood feature points of the two feature points contained in the matching pair, so as to obtain the structural consistency set corresponding to the initial matching pair set. Based on the statistical characteristics of the structural consistency contained in the structural consistency set, the filtering threshold of the current frame image is adaptively determined. The initial set of matching pairs is filtered using a screening threshold to obtain a set of filtered matching pairs. The location contribution weight of each matching pair is determined based on the extent to which the structural consistency of each matching pair in the set exceeds the screening threshold, and the extent and the location contribution weight are positively correlated. Weighted pose calculation is performed based on the localization contribution weights of all matching pairs to obtain the instantaneous pose of the UAV in the current frame image. This instantaneous pose is then used as an observation and input into a temporal filtering model. The model is then combined with the UAV's motion model for temporal fusion to output the final pose estimate of the UAV in the current frame image, thereby completing the accurate localization of the UAV.

2. The UAV positioning method according to claim 1, characterized in that, Establish the matching relationship between the first feature point set and the second feature point set, and generate an initial set of matching pairs, including: The current frame image and the reference image are detected by a feature point detection algorithm to obtain a first feature point set consisting of all feature points in the current frame image and a second feature point set consisting of all feature points in the reference image. The first feature point set and the second feature point set are processed by semantic segmentation to obtain a first semantic label for each feature point in the first feature point set and a second semantic label for each feature point in the second feature point set. Based on the first semantic label and the second semantic label, the feature point matching algorithm is used to match only feature points with the same semantic label to generate an initial set of matching pairs.

3. The UAV positioning method according to claim 1, characterized in that, The structural consistency of the matching pairs is determined as follows: each matching pair contains two feature points located in the current frame image and a reference image, respectively; local neighborhood feature points are divided for each feature point using a nearest neighbor detection algorithm, and each feature point is pointed to a local neighborhood feature point to construct a neighborhood vector. Taking the horizontal rightward direction of the feature point as the reference direction, all neighborhood vectors and their corresponding local neighborhood feature points are sorted in ascending order of the angle between each neighborhood vector and the reference direction; for each feature point, the angle difference between the two neighborhood vectors at the same index is calculated; the structural consistency of the matching pair is evaluated based on the angle difference and the distance between each feature point and its respective local neighborhood feature point.

4. The UAV positioning method according to claim 3, characterized in that, The structural consistency of the matched pairs satisfies the following relationship: ;in, It is the first Structural consistency of each matching pair This refers to the number of local neighborhood feature points for each of the two feature points in the matching pair. The index of the local neighborhood feature point. For the first The matching pair contains the first two feature points. The angular difference of each neighborhood vector For the first The first feature point contained in the matching pair is related to its second feature point. The distance between local neighborhood feature points For the first The matching pair contains the second feature point and its second feature point. The distance between local neighborhood feature points For all The average value, and Both are the tolerance variances for angle differences and distance deviations, which are preset. It is a natural exponential function.

5. The UAV positioning method according to claim 1, characterized in that, Based on the statistical characteristics of structural consistency contained in the structural consistency set, the filtering threshold for the current frame image is adaptively determined, including: The information entropy of the current frame image is determined by the probability distribution of each structural consistency in the structural consistency set. The high quantile and medium quantile of the structural consistency set are used as two statistical anchor points. Based on the empirical cumulative distribution function value of the information entropy of the current frame image within a preset historical time window, linear interpolation is performed between the two statistical anchor points to generate the screening threshold of the current frame image.

6. The UAV positioning method according to claim 5, characterized in that, Linear interpolation between two statistical anchor points is used to generate the filtering threshold for the current frame image based on the following formula: ； in, An adaptive filtering threshold for the current frame image. and These are two statistical anchor points, namely the high quantile and the median quantile of the structural consistency set. The high quantile and the median quantile were determined using the percentile method. This is the empirical cumulative distribution function value of the information entropy of the current frame image within a preset historical time window.

7. The UAV positioning method according to claim 5, characterized in that, The empirical cumulative distribution function value of the information entropy of the current frame image within a preset historical time window is determined based on the following method: A historical time window is formed from the moment the drone begins flight to the corresponding moment of the current frame image. The information entropy of each frame image within the historical time window is obtained to form a historical information entropy sequence. The empirical cumulative distribution function value of the information entropy of the current frame image in the historical information entropy sequence is obtained by counting the number of information entropy values ​​in the historical information entropy sequence that are less than or equal to the information entropy of the current frame image, and then dividing by the length of the historical information entropy sequence.

8. The UAV positioning method according to claim 7, characterized in that, The method for determining the location contribution weight of each matching pair based on the extent to which the structural consistency of each matching pair in the set exceeds the screening threshold is as follows: For each matching pair, the magnitude by which the structural consistency of the matching pair exceeds the adaptive screening threshold is normalized. The normalization result is then subjected to a positively correlated nonlinear mapping, and the resulting mapping value is used as the localization contribution weight of the matching pair.

9. The UAV positioning method according to claim 1, characterized in that, Before performing time-series fusion with the drone's motion model, the following operations are also performed: The reliability index of the observation is calculated by combining the information entropy of the current frame image and the average localization contribution weight of all matching pairs in the filtered matching pair set. The magnitude of the reliability index is negatively correlated with the information entropy and positively correlated with the average localization contribution weight. A Kalman filter is used as the time-series filtering model. A noise benchmark proportional to the prediction uncertainty covariance of the Kalman filter itself is determined. A dynamic scaling factor is calculated based on the reliability index of the observations. The magnitude of the dynamic scaling factor is negatively correlated with the reliability index of the observations. By applying the dynamic scaling factor to the noise benchmark, the observation noise covariance in the Kalman filter is dynamically adjusted to achieve adaptive fusion between the observations and the predictions of the motion model.

10. A machine vision-based unmanned aerial vehicle (UAV) positioning system, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the UAV positioning method as described in any one of claims 1-9.

Images