A method and system for positioning a UAV based on machine vision

Abstract

The application relates to the technical field of image processing, in particular to a UAV positioning method and system based on machine vision, which comprises the following steps: acquiring a current frame image and a reference image in a real-time video stream of a UAV, generating an initial matching pair set, calculating the structural consistency of each matching pair, adaptively determining a screening threshold of the current frame image based on the structural consistency, determining a screening matching pair set by using the screening threshold, evaluating the positioning contribution weight of each matching pair in the screening matching pair set, performing weighted pose solving based on the positioning contribution weight, obtaining the instantaneous pose of the UAV in the current frame image, inputting the instantaneous pose into a time sequence filtering model as an observation value, combining a motion model of the UAV to perform time sequence fusion, and outputting the final pose estimation of the UAV in the current frame image to complete accurate positioning of the UAV. The method improves the accuracy of UAV positioning.

Description

TECHNICAL FIELD

[0001] The application relates to the field of image processing, and particularly relates to a method and system for positioning unmanned aerial vehicles based on machine vision. BACKGROUND

[0002] Unmanned aerial vehicles are usually applied in the fields of surveying and mapping, inspection and the like, and the core of autonomous flight thereof depends on high-precision real-time positioning capability. The unmanned aerial vehicle positioning mainly depends on a global navigation satellite system (GNSS), and in GNSS-denied environments such as among city buildings, under bridges or indoors, the unmanned aerial vehicle must rely on visual odometry or visual simultaneous localization and mapping (SLAM) technology to perform autonomous positioning by fusing data of a machine-mounted camera and an inertial measurement unit (IMU).

[0003] In mainstream visual positioning technology, a core link is to calculate a pose by matching feature points of a current frame image and a reference image. Since IMU calculates a pose by integration of self motion, inevitable cumulative drift is generated, and thus an external absolute measurement value needs to be periodically introduced to correct the cumulative drift. The instantaneous pose calculated based on a single frame image plays a role of the external absolute measurement, and in a fusion framework, a Kalman filter is usually adopted as a time series filtering model to fuse high-frequency motion prediction values obtained by the IMU and low-frequency instantaneous poses calculated by vision. The core purpose is to use the instantaneous pose to continuously correct the cumulative drift of the IMU to output an accurate and continuous positioning trajectory in the process of flight of the unmanned aerial vehicle.

[0004] However, in a traditional Kalman filter, an observation noise covariance matrix describing visual measurement uncertainty is usually set as a fixed, offline-calibrated empirical value. This method of using a fixed observation noise covariance matrix cannot adapt to dramatic dynamic changes of visual measurement quality, thereby causing the correction process to fail in different scenarios. Specifically, when the unmanned aerial vehicle is in an ideal scenario with rich texture and clear features, the actual accuracy of the visual positioning result is very high. The fixed, conservative noise covariance matrix causes the filter to have insufficient trust in the high-quality visual data, so that the correction process is not executed thoroughly, the cumulative drift of the IMU cannot be corrected most effectively, and thus the final positioning accuracy is limited. Conversely, when the unmanned aerial vehicle flies over multiple texture-similar areas or experiences severe motion to cause image blur, the actual accuracy of the visual positioning result will drop sharply. At this time, the fixed, optimistic noise covariance matrix causes the filter to still have certain trust in the low-quality data full of errors, thereby causing errors in correction of the cumulative drift of the IMU.

[0005] In summary, the fixed noise covariance matrix is easy to directly inject harmful measurement noise into state estimation, causing the positioning trajectory to jump or drift, and leading to inaccurate positioning of the unmanned aerial vehicle. SUMMARY

[0006] To solve the problem that the traditional unmanned aerial vehicle (UAV) visual positioning fusion framework cannot adapt to the dynamic changes of visual measurement quality due to the use of fixed observation noise covariance, causing inaccurate positioning, the present application provides a UAV positioning method and system based on machine vision.

[0007] In the first aspect, the present application provides a UAV positioning method based on machine vision, comprising:

[0008] Obtaining a current frame image in a real-time video stream of the UAV and a preset reference image, establishing a matching relationship between a first feature point set of the current frame image and a second feature point set of the reference image by detecting the first feature point set and the second feature point set, and generating an initial matching pair set;

[0009] For each matching pair of the initial matching pair set, the structural consistency of the matching pair is calculated based on the geometric configuration of the local neighborhood feature points of the two feature points contained in the matching pair, to obtain a structural consistency set corresponding to the initial matching pair set, and the screening threshold of the current frame image is adaptively determined according to the statistical characteristics of the structural consistency contained in the structural consistency set;

[0010] The initial matching pair set is screened using the screening threshold to obtain a screened matching pair set, the positioning contribution weight of each matching pair of the screened matching pair set is determined according to the amplitude by which the structural consistency of the matching pair exceeds the screening threshold, and the amplitude and the positioning contribution weight are in a positive correlation;

[0011] Performing weighted pose solving based on the positioning contribution weights of all matching pairs to obtain the instantaneous pose of the UAV in the current frame image; inputting the instantaneous pose as an observation value into a time series filtering model, combining the motion model of the UAV for time series fusion to output the final pose estimation of the UAV in the current frame image, to complete the accurate positioning of the UAV.

[0012] The technical solution obtains the instantaneous snapshot of the current visual angle of the unmanned aerial vehicle and the preset reference image, and establishes an initial matching pair set through preliminary feature detection and matching. A structure consistency is given to each matching pair by calculating the geometric configuration similarity of the neighborhood environment of each matching pair in the respective images. A screening threshold most suitable for the complexity of the current scene is adaptively determined by analyzing the statistical characteristics of all structure consistencies. After screening of the initial matching pair set is completed using the screening threshold, fine positioning contribution weight distribution is further performed. It is believed that not all screened data is equally important, but a nonlinear positioning contribution weight is assigned to each matching pair according to the magnitude by which the structure consistency of the matching pair exceeds the screening threshold, so as to ensure that the matching pairs with better and more accurate and reliable structure consistencies have a dominant role in the final positioning. Weighted pose solving is performed based on the matching pairs with the positioning contribution weight, to obtain a high-quality instantaneous pose. Further considering that the instantaneous pose of a single image may still have instantaneous interference, time sequence consistency checking is introduced. The high-quality instantaneous pose is taken as an external observation value and is input into a time sequence filtering model combined with the motion inertia of the unmanned aerial vehicle. The model compares the external observation value with a predicted value based on the historical trajectory, smooths out any sudden changes that do not conform to the physical law, and finally outputs a final pose estimate that is accurate in space and continuous in time, thereby completing the accurate positioning of the unmanned aerial vehicle.

[0013] Preferably, the matching relationship between the first feature point set and the second feature point set is established, and an initial matching pair set is generated, including: detecting the current frame image and the reference image respectively by a feature point detection algorithm to obtain a first feature point set composed of all feature points of the current frame image and a second feature point set composed of all feature points of the reference image; processing the first feature point set and the second feature point set respectively by semantic segmentation to obtain a first semantic label of each feature point in the first feature point set and a second semantic label of each feature point in the second feature point set; and matching only feature points with the same semantic label by a feature point matching algorithm according to the first semantic label and the second semantic label to generate the initial matching pair set.

[0014] Preferably, the structure consistency of the matching pair is determined based on the following manner: each matching pair contains two feature points respectively located in the current frame image and the reference image; the local neighborhood feature points of each feature point are divided by a nearest neighbor detection algorithm, the feature point is directed to each local neighborhood feature point to construct a neighborhood vector, and in the reference direction of the horizontal right direction of the feature point, all neighborhood vectors and corresponding local neighborhood feature points are sorted in order of the included angle between each neighborhood vector and the reference direction from small to large; for the neighborhood vectors and local neighborhood feature points sorted by the two feature points respectively, the included angle difference of the two neighborhood vectors at the same serial number is calculated; and the structure consistency of the matching pair is evaluated according to the included angle difference and the distance between the two feature points and the respective local neighborhood feature points.

[0015] Preferably, the structure consistency of the matching pair satisfies the following relationship:

[0016]

[0017] wherein, is the structure consistency of the i th matching pair, is the structure consistency of the i th matching pair, is the number of local neighborhood feature points of the two feature points contained in the matching pair respectively, is the serial number of the local neighborhood feature point, is the angle difference of the j th neighborhood vector of the two feature points contained in the i th matching pair, is the distance between the first feature point and the j th local neighborhood feature point of the first feature point contained in the i th matching pair, is the distance between the second feature point and the j th local neighborhood feature point of the second feature point contained in the i th matching pair, is the average value of all i, and are the tolerance variance of the preset angle difference and the tolerance variance of the distance deviation degree, is a natural exponential function. This technical solution combines two independent penalty terms through a negative exponential function, accurately evaluates the local geometric configuration consistency of each matching pair, the first penalty term considers the invariance of the rotation of the unmanned aerial vehicle, selects the consistency of the relative angle arrangement between the neighborhood vectors, and the second penalty term is based on the distance between the feature point and its local neighborhood feature point, dynamically establishes a reference benchmark of the distance, and can accurately identify the false feature point matching caused by the non-uniform geometric distortion in the local area of the feature point.

[0018] ​​​​​​​

[0019] Preferably, the adaptive determination of the screening threshold of the current frame image according to the statistical characteristics of the structural consistency contained in the structural consistency set comprises: determining the information entropy of the current frame image through the probability distribution of each structural consistency in the structural consistency set, taking the high quantile and the medium quantile of the structural consistency set as two statistical anchor points; performing linear interpolation between the two statistical anchor points to generate the screening threshold of the current frame image based on the empirical cumulative distribution function value of the information entropy of the current frame image within a preset historical time window.

[0020] Preferably, the linear interpolation between the two statistical anchor points to generate the screening threshold of the current frame image is performed based on the following formula:

[0021]

[0022] wherein, is the adaptive screening threshold of the current frame image, and are two statistical anchor points, respectively the high quantile and the medium quantile of the structural consistency set, the high quantile and the medium quantile being determined by the percentile method, is the empirical cumulative distribution function value of the information entropy of the current frame image within a preset historical time window.

[0023] This technical solution constructs a threshold decision model driven by real-time data, performs linear interpolation between two dynamically calculated statistical anchor points, and the obtained screening threshold is dynamically adaptive according to the quality of the current frame image. The weight of linear interpolation is determined by the relative ranking of the complexity of the current frame image in its recent history, so that the threshold decision model can be sensitive to occasional complexity in a generally simple environment, and ensures that the screening threshold can automatically converge to an optimal balance point that can guarantee both accuracy and robustness in any scenario.

[0024] Preferably, the empirical cumulative distribution function value of the information entropy of the current frame image within a preset historical time window is determined based on the following manner: a historical time window is constituted from the moment when the unmanned aerial vehicle starts flying to the corresponding moment of the current frame image, the information entropy of each frame image within the historical time window is obtained to constitute a historical information entropy sequence; the number of historical information entropies in the historical information entropy sequence that is less than or equal to the information entropy of the current frame image is counted, and then divided by the length of the historical information entropy sequence to obtain the empirical cumulative distribution function value of the information entropy of the current frame image in the historical information entropy sequence.

[0025] Preferably, the method for determining the positioning contribution weight of each matching pair in the matching pair set according to the amplitude by which the structural consistency of the matching pair exceeds the screening threshold is: for each matching pair, normalizing the amplitude by which the structural consistency of the matching pair exceeds the adaptive screening threshold, performing positive correlation nonlinear mapping on the normalized result, and taking the mapping value obtained as the positioning contribution weight of the matching pair.

[0026] The technical solution constructs a nonlinear weight distribution mechanism, the core of which is to associate the weight with the relative superiority of the matching quality, decouples the input into a standardized interval irrelevant to the threshold through normalization processing, introduces a strategic weight adjustment based on nonlinear mapping, the marginal gain of the weight is maximum for the matching pair with a lower normalized value, thereby effectively distinguishing and amplifying the influence of the matching pair with a medium quality, and the growth of the weight tends to be flat for the matching pair with a higher normalized value, thereby avoiding excessive sensitivity to the slight difference between the matching pairs with high quality, which makes the pose solution process more robustly rely on a core set composed of reliable matching pairs, while suppressing the uncertainty of the matching pairs with only a barely qualified quality.

[0027] Preferably, before the time series fusion in combination with the motion model of the unmanned aerial vehicle, the following operations are further performed, including: calculating a reliability index of the observation value in combination with the information entropy of the current frame image and the average positioning contribution weight of all matching pairs in the matching pair set, the size of the reliability index is negatively correlated with the information entropy and positively correlated with the average positioning contribution weight; adopting a Kalman filter as a time series filtering model, determining a noise reference proportional to the prediction uncertainty covariance of the Kalman filter itself, calculating a dynamic scaling factor according to the reliability index of the observation value, the size of the dynamic scaling factor is in a negative correlation with the reliability index of the observation value; and dynamically adjusting the observation noise covariance in the Kalman filter by applying the dynamic scaling factor to the noise reference, to realize adaptive fusion between the observation value and the predicted value of the motion model.

[0028] The technical solution constructs a dual adaptive closed-loop control system, which is realized by deeply coupling the adjustment of the observation noise with the external observation quality and the internal state of the filter, proportionally associates the noise reference with the prediction uncertainty covariance of the Kalman filter itself, dynamically associates the noise expectation with the internal uncertainty state of the filter at this moment, adjusts the observation noise covariance through a dynamic scaling factor related to the observation quality, increases the trust degree when the external observation quality is high to effectively correct the drift, and reduces the trust degree when the external observation quality is poor to suppress the harmful measurement to ensure robustness, thereby achieving optimal state estimation in any scenario.

[0029] In a second aspect, the present application also provides a UAV positioning system based on machine vision, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any one of the UAV positioning methods.

[0030] The present application has the following effects:

[0031] The present application introduces a real-time visual measurement quality diagnosis and adaptive trust adjustment mechanism into a traditional visual fusion framework, first purifies and evaluates the initial matching data through semantic priori and local geometric structure verification, then based on the evaluation results, adaptively generates a filtering threshold from data driving to obtain the positioning contribution weight of each matching pair, performs weighted pose estimation, evaluates the reliability index of the UAV pose estimation in the current frame image, and finally dynamically couples the reliability index with the prediction uncertainty of the Kalman filter itself to adjust the observation noise covariance in real time, solves the problem of invalidation of the IMU drift correction process caused by the use of fixed noise covariance, makes the Kalman filter be able to take full advantage of high-quality visual data to improve the precision while avoiding the interference of low-quality data, and further improves the accuracy of the UAV positioning. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a flowchart of the present application. DETAILED DESCRIPTION

[0033] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application.

[0034] Referring to Figure 1 A UAV positioning method based on machine vision, specifically comprising the following steps:

[0035] S1: acquiring a current frame and a reference image and extracting initial matching pairs.

[0036] In order to efficiently and discriminatively extract structured feature information from the visual sensor input of the unmanned aerial vehicle, and preliminarily match the correlation, and provide original data input for subsequent fine quality evaluation and pose solving, first, an arbitrary frame image in the real-time video stream obtained from the camera carried by the unmanned aerial vehicle is taken as a current frame image, and a reference image corresponding to the current frame image is obtained from a pre-constructed map or key frame library, then feature point detection is performed on the current frame image and the reference image, in order to ensure the number of feature points while taking into account the real-time requirement of the airborne computing platform, the current frame image and the reference image are detected by using a fast corner detection algorithm, the fast corner detection algorithm quickly identifies a corner point by comparing the gray difference between a pixel point and its neighborhood pixels, each corner point is taken as a feature point, and thus a feature point set of the current frame image and a feature point set of the reference image are generated, which are denoted as a first feature point set and a second feature point set respectively.

[0037] The current frame image and the reference image jointly constitute the basis for pose solving, the current frame image refers to a two-dimensional image representing the current view angle of the camera of the unmanned aerial vehicle, and the feature points in the image only contain two-dimensional pixel coordinate information, while the reference image is derived from a pre-constructed map or key frame library with three-dimensional spatial information, and the core value lies in that each feature point in the image is accurately corresponding to a known three-dimensional coordinate in the world coordinate system, by matching the two-dimensional feature points of the current frame image with the corresponding three-dimensional feature points in the reference image, a series of corresponding relationships of positions are established, so that the system can solve the instantaneous pose of the unmanned aerial vehicle based on the corresponding relationships.

[0038] In order to suppress the mismatch caused by scene semantic inconsistency from the source, the embodiment introduces semantic segmentation as a strong prior constraint, specifically, the current frame image and the reference image are input into a pre-trained lightweight semantic segmentation network in parallel, for example, a MobileUNet semantic segmentation model, the semantic segmentation model will assign a predefined class label, such as building, road, vegetation, etc., to each pixel point of the two images, since a feature point is essentially a pixel point, the semantic label of each feature point is obtained at this time.

[0039] The BRIEF feature descriptor is calculated for each feature point with a semantic label, and initial matching is performed, the matching process follows the strict same-class matching principle, specifically:

[0040] For any feature point in the first feature point set, feature points with the same semantic label as that feature point are selected from the second feature point set to form a set of feature points to be matched. The Hamming distance between the BRIEF descriptor of that feature point and the BRIEF descriptors of each feature point in the set of feature points to be matched is calculated. Feature points that match that feature point are determined according to the minimum Hamming distance, thus establishing a matching relationship between the first and second feature point sets. Based on this matching relationship, each feature point in the first feature point set and each matching feature point in the second feature point set form a matching pair. Finally, all feature points in the first feature point set are traversed and the matching process is repeated to form an initial set of matching pairs. This semantic prior-based matching strategy can eliminate a large number of logically unreasonable potential matching feature points, improving the purity of the initial matching set.

[0041] S2: Evaluate the consistency of the local neighborhood structure of each matching pair.

[0042] This step aims to address the problem that initial matching pairs lack geometric constraints based solely on feature descriptor similarity. To this end, this invention proposes a mechanism for evaluating the similarity of the local neighborhood geometric configurations of matching pairs, generating a structural consistency for each matching pair that reflects its geometric reliability.

[0043] Specifically:

[0044] First, for the first matching pair in the initial set... A matching pair is formed by a feature point located in the current frame and a feature point located in the reference image, denoted as the first feature point and the second feature point, respectively. The nearest neighbor detection algorithm is used to find five local neighbor feature points for the first feature point and five local neighbor feature points for the second feature point. In this embodiment, five local neighbor feature points are selected, achieving a good balance between ensuring geometric stability and computational efficiency. Setting the number of local neighbor feature points to five is based on the consideration that too few (e.g., less than three) would lead to overly simple local geometric configurations, easily causing ambiguity and sensitivity to noise; while too many would increase the computational burden and may introduce irrelevant distant feature points, affecting real-time performance. The number of local neighbor feature points can be adjusted according to image resolution and feature point density. In low-resolution or sparse feature scenarios, it can be appropriately reduced to three to ensure real-time performance; in high-resolution or dense feature scenarios, it can be increased to seven or eight to improve the stability of the structural description.

[0045] Subsequently, the first feature point is linked to its five local neighboring feature points, forming five neighborhood vectors for the first feature point. Using the horizontal-to-right direction of the first feature point as the reference direction, the angle between each neighborhood vector and the reference direction is obtained by dividing the angle counter-clockwise. At this point, one neighborhood feature point corresponds to one neighborhood vector, and one neighborhood vector corresponds to one angle. Following the same processing method as the first feature point, the five neighborhood vectors and their angles for the second feature point are obtained.

[0046] The five neighborhood vectors of each feature point are sorted in ascending order of the included angle, numbered from 1 to 5. The structural consistency of the matching pair is evaluated based on the difference in the included angle between two neighborhood vectors at the same number and the distance between the two feature points and their respective local neighborhood feature points. A correct matching pair should contain two feature points whose local geometric structures should maintain a high degree of similarity between the two images. The better the feature point matching, the better the result, and vice versa.

[0047] Specifically, it follows the following relationship:

[0048]

[0049] in, It is the first Structural consistency of each matching pair This refers to the number of local neighborhood feature points for each of the two feature points in the matching pair. The index of the local neighborhood feature point. For the first The matching pair contains the first two feature points. The angular difference of the neighborhood vectors For the first The first feature point contained in the matching pair is its second feature point. The distance between local neighborhood feature points For the first The matching pair contains the second feature point and its second feature point. The distance between local neighborhood feature points For all The average value, and Both are the tolerance variances for angle differences and distance deviations, which are preset. It is a natural exponential function.

[0050] In this relation: It is the mean term, for The consistency evaluation results of each neighborhood feature point are averaged to eliminate outlier interference from individual neighborhood feature points. The Gaussian decay results of each neighborhood feature point are averaged, which is equivalent to further smoothing the extreme Gaussian decay values ​​of individual points, avoiding outliers from dominating the final result. This allows the structural consistency to retain the advantage of Gaussian decay in accurately penalizing differences, while reducing the impact of random errors through the mean, resulting in a more stable consistency evaluation result. An angle penalty term is used to transform the angular differences of neighborhood vectors into a penalty term that conforms to Gaussian decay, thus addressing the matching bias caused by UAV rotation. This directly quantifies the squared error of the angle difference deviating from the ideal value of 0, and is the core of constructing the Gaussian penalty. This squared term ensures that the penalty value is non-negative, and the smaller the angle difference, the closer the penalty is to 0, which perfectly conforms to the characteristic of the Gaussian distribution to penalize around the ideal value. Dividing by This is a typical variance normalization operation in a Gaussian distribution. Corresponding to the variance of the Gaussian distribution, the larger the variance, the smaller the penalty term for the same angle difference, the smoother the Gaussian decay, and the higher the tolerance for angle differences. Conversely, the smaller the variance, the larger the penalty term, the steeper the Gaussian decay, and the more sensitive to angle differences. By following the path from angle difference to Gaussianization penalty to exponential decay, the angle consistency score conforms to the Gaussian distribution: small angle differences only cause slight decay, retaining reasonable matches; large angle differences cause drastic decay, eliminating mismatches, and improving resistance to rotation interference. The distance penalty term transforms the difference in distance ratio between two feature points and their neighboring feature points into a Gaussian penalty term, addressing image scaling interference caused by the distance of the drone. This, along with the angle term, forms a two-dimensional Gaussian constraint. , Equivalent to the mean of a Gaussian distribution, the squared term ensures that the difference from the mean is non-negative, and conforms to the core logic of the Gaussian distribution: "the farther from the mean, the lower the probability density." The closer the distance proportion is to the mean, the smaller the penalty. This corresponds to the center of the Gaussian distribution, then divided by... , It represents the tolerance variance for the degree of distance deviation, corresponding to the variance of a Gaussian distribution, and controls the decay rate of the distance ratio difference. The larger the value, the smoother the Gaussian decay, making it suitable for scenes with sparse textures and large scaling fluctuations. The smaller the value, the steeper the attenuation, making it suitable for scenes with dense textures and stable scaling. Gaussianization ensures that the distance ratio consistency score also follows a Gaussian law. Even with slight scaling of the drone, as long as the distance ratio remains close to the mean, it maintains a low penalty and high attenuation retention value, avoiding misjudging high-quality matching pairs due to scaling. For the exponential mapping term, the total Gaussian penalty term of angle and distance is transformed into the result consistency in the 0-1 interval, which is a direct embodiment of Gaussian attenuation. The total penalty term is equal to the sum of the angle Gaussian penalty term and the distance ratio penalty term, which is equivalent to the superposition of the two-dimensional deviation. When the total penalty term is equal to 0, there is no difference in two dimensions. Due to the negative sign in front of the total penalty term, the exponential mapping term is the maximum value 1, and the result consistency is the best. When the total penalty term increases, the exponential mapping term monotonically decreases according to the Gaussian curve and tends to 0. The farther the deviation, the lower the consistency score. Through Gaussian attenuation, it realizes the smooth mapping that the smaller the difference, the better the consistency.

[0051] wherein, and A determination method is as follows: since both are the variance of the normal distribution to which the geometric difference of the correct matching pair in the current scene is subjected, for the correct matching pair, the angle difference and the deviation of the distance ratio should approximately obey the normal distribution with zero mean, and The essence is the variance of the normal distribution, which reflects the dispersion degree of the geometric difference of the correct matching pair. The more stable the scene and the lower the noise, the smaller the variance. The more complex the scene and the higher the noise, the larger the variance. Therefore, the structural consistency of all matching pairs in the initial matching pair set is sorted in order from small to large, and the matching pairs with structural consistency greater than the third quartile are used to form the trusted matching pair set by using the quartile method. The angle difference and the difference of the distance ratio of each matching pair in the trusted matching pair set are obtained. The unbiased variance of of all matching pairs is taken as , and the unbiased variance of of all matching pairs is taken as If there are 100 matching pairs, there are 500 and 500 . The unbiased variance of the 500 is taken as , and the unbiased variance of the 500 is taken as .

[0052] In summary, the essence of the relationship is a three-layer structure of two-dimensional Gaussian penalty and exponential decay mapping and mean smoothing, and finally output the structure consistency score in the interval of 0-1, wherein the core carrier of Gaussian decay is the exponential term, which completely follows the characteristics of the Gaussian function "centered on the benchmark value, the greater the deviation, the more significant the decay", the smaller the total penalty term of the exponential term, the smaller the difference of the geometric configuration, and the closer the value after Gaussian decay to 1, the higher the consistency; on the contrary, the larger the total penalty term, the greater the difference of the geometric configuration, and the closer the value after Gaussian decay to 0, the lower the consistency, which is the key to realize accurate distinction between high-quality matching pairs and low-quality matching pairs.

[0053] If the structure consistency of a matching pair is greater, closer to 1, it indicates that the local neighborhood geometric configuration between its two feature points is highly isomorphic, and it is a high-quality and high-credibility matching pair, and the greater the structure consistency. On the contrary, the smaller the structure consistency of a matching pair, the closer to 0, it indicates that there is a significant difference in the local geometric structure, and it is likely to be a false matching caused by texture similarity. According to this method, the structure consistency of each matching pair in the initial matching pair set is obtained, and the structure consistency set corresponding to the initial matching pair set is obtained.

[0054] S3: Generate a screening threshold based on the statistical characteristics of the structure consistency.

[0055] In order to better adapt to the scene changes of the image, a dynamic threshold generation mechanism completely driven by data is proposed, that is, according to the statistical characteristics of the structure consistency contained in the structure consistency set, the screening threshold of the current frame image is adaptively determined. The essence of the screening threshold is to directly associate the reliability of the pose solution with the physical world features observed by the unmanned aerial vehicle. The original data of the unmanned aerial vehicle positioning comes from the scene image captured by its camera, therefore, the feature richness of the scene directly determines the quality of the visual information available for positioning. In a scene with rich texture and complex structure (such as urban building group), the unmanned aerial vehicle can obtain a large number of high-quality and high-discrimination feature points, at this time the information entropy is high, the screening threshold is appropriately relaxed, aiming to include enough effective matching pairs for robust statistical calculation to ensure the robustness of positioning. On the contrary, in a scene with sparse texture and degenerated features (such as open water surface or large wall), high-quality feature points are scarce, at this time the information entropy is low, the screening threshold is tightened to retain only the matching pairs with the highest confidence under the most stringent standard, and the core purpose is to prevent the pollution of the final pose solution caused by the fuzzy matching or false matching pairs due to the degeneration of the scene.

[0056] Specifically, it includes:

[0057] Firstly, the structure consistency set is analyzed in real time, and the high percentile point (the 90th percentile) and the middle percentile point (Median), as two statistical anchors, the 90th percentile is selected, aiming to establish the statistical anchors in the top interval of the data distribution with the best quality, ensuring the strictness of the screening benchmark. Practice has shown that the 90th percentile can effectively eliminate outliers in most scenarios while retaining enough high-quality matching pairs.

[0058] Meanwhile, the information entropy of the structural consistency set is calculated as the information entropy of the current frame image. A historical time window is formed by the first frame image to the first frame image before the current frame image in the real-time video stream of the unmanned aerial vehicle. The information entropy of each frame image in the historical time window is obtained in the same way as the information entropy of the current frame image, forming a historical information entropy sequence. The number of historical information entropy sequence less than or equal to the information entropy of the current frame image is counted, and then divided by the length of the historical information entropy sequence to obtain the empirical cumulative distribution function value of the information entropy of the current frame image in the historical information entropy sequence Using all historical data can provide the most stable statistical baseline for the calculation of the empirical cumulative distribution function, avoiding fluctuations caused by short-term scene changes.

[0059] Finally, the two statistical anchors are linearly interpolated by the empirical cumulative distribution function value to generate the screening threshold of the current frame image :

[0060]

[0061] wherein, is the screening threshold of the current frame image, and are two statistical anchors, which are the high percentile point and the median percentile point of the structural consistency set respectively, and the high percentile point and the median percentile point are determined by the percentile method, is the empirical cumulative distribution function value of the information entropy of the current frame image in the historical time window.

[0062] In this relationship, the lower the information entropy of the current frame image, the sparser the scene texture, the smaller, tending to 0, making tend to the high percentile point , a more stringent screening standard is adopted, otherwise, the higher the information entropy of the current frame image, the richer the scene texture and the more complex the details, the larger, tending to 1, making tend to the median percentile point , a relatively more inclusive standard is adopted to ensure that there are enough matching pairs for pose solving.

[0063] S4: Screening the initial matching pair set based on the screening threshold and calculating the positioning contribution weight.

[0064] This step aims to purify the initial matching pair set with a screening threshold, and evaluate the contribution weight of the screened matching pair in the positioning process.

[0065] First, use the screening threshold Screen all initial matching pairs, only keep the matching pairs with structural consistency greater than in the initial matching pair set to form a screened matching pair set, use as the serial number of the matching pair in the screened matching pair set, get the structural consistency of each matching pair in the screened matching pair set, normalize the amplitude of the structural consistency of the matching pair exceeding the adaptive screening threshold to get a normalized value, and perform a positive correlation nonlinear mapping on the normalized value. The mapping value obtained is taken as the positioning contribution weight of the matching pair.

[0066] Specifically, the nonlinear mapping is performed through the following relationship:

[0067]

[0068] Wherein, is the positioning contribution weight of the th matching pair in the screened matching pair set, is a sine function, is the structural consistency of the th matching pair in the screened matching pair set, is the screening threshold, is a parameter to prevent the denominator from being 0, usually a very small positive number .

[0069] In this relationship, is the normalized term, the numerator is the amplitude exceeding the screening threshold, and the denominator is the maximum possible exceeding amplitude. When , , , the sine function has a monotonically increasing property in the interval , when is slightly larger than , just reaches the screening standard, the value of the sine function is small, close to 0, and the positioning contribution weight of this matching pair is small. When tends to 1, the value of the sine function is larger, and the positioning contribution weight of this matching pair is large, with a maximum value of 1. At the same time, the slope of the sine function gradually decreases on , and as increases, the slope of the sine function decreases from 1 to 0, i.e. the growth rate of the positioning contribution weight becomes slower and slower, which is specifically manifested as: when the amplitude exceeding the screening threshold is small, is in the interval , positioning contribution weight grows faster, the amplitude of exceeding the screening threshold is larger, in , weight growth slows down, even close , the weight will not appear step growth.

[0070] This nonlinear adjustment based on the sine function ensures that the matching pairs just above the screening threshold obtain smaller weights, and the matching pairs with scores far above the screening threshold obtain larger weights, but the growth rate of the weights will slow down as the scores approach the upper limit. This design avoids the excessive influence of individual extremely high structural consistency matching pairs on the calculation results, thereby improving the robustness of the pose calculation.

[0071] In addition, the present application is not limited to using the sine function. Any nonlinear function with the characteristics of monotonically increasing and decreasing marginal gain in the [0, 1] interval, such as the Sigmoid function and the partial tangent function, can be used to implement the nonlinear weight distribution operation of the present application.

[0072] S5: Perform weighted pose calculation based on the positioning contribution weights of all matching pairs to obtain the instantaneous pose of the UAV in the current frame image.

[0073] In existing PnP (Perspective-n-Point) or ICP (Iterative Closest Point) based pose calculation methods, all matching feature points are usually considered to be equally reliable, that is, in constructing the cost function, the re-projection error terms of each matching point have the same weight. The present application introduces the positioning contribution weight obtained by structural consistency evaluation and embeds the weight as a weighting coefficient of the error term in the traditional pose calculation formula, thereby realizing the weighted amplification of high-reliability matching points and the suppression of low-reliability matching points. This method does not change the pose calculation framework and only optimizes the weight distribution of each term in the cost function, so it can be seamlessly integrated into the pose solver.

[0074] Specifically, the essence of traditional pose calculation is to analyze a series of matching pairs to solve the current pose of the UAV. A matching pair is composed of a three-dimensional point with a known position in the world coordinate system and a two-dimensional point observed in the current camera image. Corresponding to the current scene, the observed two-dimensional point is the feature point in the current frame image of the matching pair, denoted as the first feature point of the matching pair, and the three-dimensional point with a known position is the feature point in the reference image of the matching pair, denoted as the second feature point of the matching pair. The goal of pose calculation is to inversely calculate the optimal camera pose by minimizing the error between the theoretical projection position of the second feature point of all matching pairs and the actual observed position of the first feature point, i.e., the re-projection error. The mathematical form is usually expressed as:

[0075]

[0076] wherein, is the serial number of the matching pair, is the total number of matching pairs, is the coordinate of the first feature point of the matching pair, which is the two-dimensional coordinate of the feature point in the current frame image, is the coordinate of the second feature point of the matching pair, which is the three-dimensional coordinate of the feature point in the reference image, is the translation vector, is the symbol for calculating the distance between coordinates, and are the rotation matrix and translation vector of the camera, respectively, which together constitute the current pose of the UAV, and are the unknown quantities to be solved by the formula. is the coordinate of the feature point in the camera coordinate system, after rotation and translation , is the coordinate in the camera coordinate system, which is transformed to the virtual imaging plane of the camera by the intrinsic matrix of the camera, when followed by parentheses, it represents a projection function, and when appearing in trigonometric functions or mathematical operations, it represents the ratio of a circle's circumference to its diameter. is the final two-dimensional pixel coordinate obtained by converting the coordinate on the virtual imaging plane to the actual two-dimensional pixel coordinate through the projection function, and is the theoretical projection position under the current pose. represents the distance between the actual observed position of the first feature point and the theoretical projection position of the second feature point, and the greater the distance, the greater the error. The square of the distance is calculated to obtain the re-projection error of the matching pair. is the accumulation of the re-projection errors of all matching pairs, which is the matching cost function. The smaller the accumulated value, the more consistent the matching result is with the geometric constraints. Therefore, by minimizing the matching cost function, we obtain and , and further obtain the instantaneous pose of the UAV in the current frame image.

[0077] The instantaneous pose refers to the complete state description of the UAV in a certain specific instant in the three-dimensional space, just like the instant captured by a photo, the position and orientation of the UAV, specifically including the following two parts: one part is the position information, which describes the exact location of the UAV in the three-dimensional space, which is usually represented by a three-dimensional coordinate, and the coordinate is relative to a fixed world coordinate system (for example, the take-off point or the origin of the map); the other part is the attitude information, which describes the orientation of the UAV, or its rotation state, which has various mathematical representations, the most common one is Euler angle, which is the most intuitive way, including three independent rotation angles: roll: left and right roll of the fuselage along the forward direction; pitch: up and down pitch of the nose; yaw: left and right yaw of the nose.

[0078] The traditional pose solution completely regards all matching points as equivalent contributions, and the error of each matching pair is treated equally. The positioning contribution weight of the application is directly inserted into the objective function as a weighting coefficient of the error term, which corresponds to the following formula position:

[0079]

[0080] wherein, is the positioning contribution weight of the i-th matching pair in the matching pair set, and the meanings of the remaining characters are consistent with those in the mathematical form of the traditional pose solution.

[0081] In summary, the application does not change the mathematical framework of the pose solution itself, but introduces a matching reliability weight in the matching cost function of the traditional pose solution, so as to realize the enhancement effect of high-quality matching and the inhibition of potential mismatching, thereby improving the stability and accuracy of the pose solution without changing the algorithm structure.

[0082] S6: evaluating the visual observation reliability index of the instantaneous pose of the UAV in the current frame image, and dynamically updating the observation noise covariance in combination with the motion model to output the final positioning trajectory of the UAV.

[0083] The instantaneous pose of the UAV in the current frame image is input into a time series filtering model, and the motion model of the UAV is combined for time series fusion to output the final pose estimation of the UAV in the current frame image, so as to complete the accurate positioning of the UAV.

[0084] Specifically, the output high-quality instantaneous visual pose is combined with the motion model provided by the IMU (inertial measurement unit) of the UAV for time series fusion to output a smooth and accurate positioning trajectory.

[0085] ​The observation noise covariance matrix in a Kalman filter is explained as follows: This matrix is ​​a core parameter in the Kalman filter framework. Its function is to quantitatively describe the uncertainty or noise level of the external sensor measurements themselves. It represents the degree of distrust the filter has in the visual pose calculation results. A small observation noise covariance matrix means that the filter is told to have a high degree of trust in the current visual measurement; conversely, a large observation noise covariance matrix indicates that the filter should reduce its trust in the visual measurement and rely more on its own motion prediction. In traditional methods, the observation noise covariance matrix is ​​usually a fixed value set empirically.

[0086] This step achieves dynamic updating of the observation noise covariance at the current moment through the following process, and outputs the final positioning trajectory of the UAV:

[0087] The first step is to calculate the reliability index of the visual observation. This index is constructed based on the following: a reliable visual observation requires not only high internal matching quality but also that the scene itself provides sufficiently rich and stable information. Combining the two dimensions of scene richness and matching quality, the credibility of the current visual observation is evaluated (from 0 to 1, the closer to 1, the more reliable). Specifically, this is based on the following relationship:

[0088]

[0089] in, It is the drone in the A visual observation reliability index for instantaneous pose in frame images. The grayscale entropy of the current frame image reflects the richness of the scene's texture. The higher the value, the more complex and detailed the scene, providing a more reliable environmental basis for visual positioning. s is the number of grayscale histogram bins when calculating information entropy (the default value in engineering is 256, corresponding to an 8-bit grayscale image). It is the theoretical maximum information entropy. hour, Its function is to Normalize to the interval between 0 and 1. It is the normalized information entropy, reflecting whether the current frame image can provide reliable visual information. The closer the value is to 1, the richer the texture of the current frame image and the more reliable the environmental basis for visual observation. It is the mean of the location contribution weights of all matching pairs in the set of reliable matching pairs. It reflects how good the quality of the matching pair itself is. The closer it is to 1, the higher the overall quality of the matching pair and the better the data quality of visual observation. This means that if the texture of the current frame image is richer, and the feature point matching quality between the current frame image and the reference image is higher, then... The greater, the closer to 1, the current visual observation is reliable enough, and it can be trusted to correct the IMU drift; on the contrary, if the texture of the current frame image is sparser, and the matching quality of the feature points of the current frame image and the reference image is worse, The closer to 0, the less reliable the visual observation is, and the less it can be trusted to correct the IMU drift, and the influence of the visual observation on the filter fusion needs to be reduced.

[0090] Secondly, according to the visual observation reliability index of the instantaneous pose of the UAV in the first frame image and the prediction uncertainty of the Kalman filter itself, the observation noise covariance matrix at the current time is dynamically updated, realizing the logic that the more reliable the vision is, the higher the trust degree is.

[0091] Specifically:

[0092]

[0093] Among them, is the observation noise covariance matrix of the UAV at the time corresponding to the first frame image, is the visual observation reliability index of the instantaneous pose of the UAV in the first frame image, is the observation matrix, which is used to map the system state (such as position, attitude, velocity, etc.) of the UAV to the visual observation space. For example, if the system state is 6-dimensional (3-dimensional position + 3-dimensional attitude) and the visual observation also outputs 6-dimensional pose, then can be a 6x6 unit matrix, if the dimensions of the visual observation and the system state are different, the conversion rule needs to be designed according to the sensor model (such as the camera projection relationship), is the prior prediction covariance matrix, which is generated by the motion model of the UAV (such as IMU integration), and represents the uncertainty of the state prediction based on the state at the time, the greater the value is, the more unreliable the prediction result of the motion model is (for example, when the IMU has accumulated drift, it will gradually increase. is the motion model prediction uncertainty in the state space described by , which is projected to the observation space through the observation matrix and its transpose matrix , because describes the uncertainty in the system state space, while the visual observation is a measurement in the observation space, therefore, the mapping from the state to the observation is realized through , and the transpose matrix of The inverse mapping from observation to state is realized, and the uncertainty of the state space is projected to the observation space, ensuring that the dimension and physical meaning of the observation noise covariance are consistent with the mathematical operation requirements of the Kalman filter. The state is the internal comprehensive judgment of the filter on the entire situation of the UAV (such as speed), and the observation is part of the external evidence provided by the vision system (instantaneous pose). The observation matrix is a converter that connects the two and allows them to be compared with each other.

[0094] In the formula, is the visual distrust ratio factor, which quantifies the degree of distrust in the current vision, the higher, the smaller, the lower the distrust, indicating that the filter trusts the visual observation results more, and the noise covariance matrix is smaller. Conversely, the lower, the higher, the higher the distrust, indicating that the filter does not trust the visual observation results, and the noise covariance matrix is larger. The larger the prior prediction covariance of the motion model (such as IMU), the less reliable the prediction of the motion model (such as the existence of cumulative drift in the IMU). Through the operation of the observation matrix and its transpose , the uncertainty of the motion model in the state space is projected to the measurement space of the visual observation, ensuring that the dimension and physical meaning of the observation noise covariance are consistent.

[0095] This formula multiplies a distrust ratio factor that is negatively related to by the projection of the filter's own prediction uncertainty in the observation space, obtaining the dynamically updated observation noise covariance matrix at the current time, allowing the Kalman filter to intelligently balance the accuracy of vision and the robustness of the motion model. When the vision is good, use the vision to correct the drift, and when the vision is poor, use the motion model to maintain continuity, and finally output a smoother, more accurate and robust UAV positioning trajectory.

[0096] Third, use the updated observation noise covariance matrix to complete state fusion and generate a positioning trajectory. After obtaining the dynamic , the Kalman filter performs its standard update step. First, calculate the Kalman gain , which is inversely proportional to , then use the gain to weight and fuse the visual measurement value and the predicted value of the IMU, obtaining the optimal posterior state estimation at the current time, i.e. the corrected UAV position, attitude and speed.

[0097] This process is repeated continuously at every frame as the UAV flies. When the reliability of the visual observation is high, Thus reducing, thus leading to Kalman gain Increasing, the filter trusts the vision result more, effectively correcting the cumulative drift of the IMU, when the vision observation reliability is low, Thus increasing, thus leading to Kalman gain Reducing, the filter relies more on the short-term prediction of the IMU, effectively suppressing the interference of bad vision measurement on state estimation.

[0098] Finally, the filter generates an optimal fusion of the posterior state estimation point at each frame, that is, the most accurate UAV state at the current time. The position information of a series of continuous output state estimation points jointly constitutes the complete, smooth and high-precision UAV positioning trajectory generated by the application. Through the cooperative work of these steps, the accuracy and reliability of the autonomous positioning of the UAV in various complex and dynamically changing scenes are improved.

[0099] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A machine vision-based UAV localization method, characterized in that, include: Acquire a current frame image and a preset reference image from the real-time video stream of the drone. By detecting the first feature point set of the current frame image and the second feature point set of the reference image, establish the matching relationship between the first feature point set and the second feature point set, and generate an initial matching pair set. For each matching pair in the initial matching pair set, the structural consistency of the matching pair is calculated based on the geometric configuration of the local neighborhood feature points of the two feature points contained in the matching pair. This includes: the two feature points contained in each matching pair are located in the current frame image and the reference image, respectively; the local neighborhood feature points of each feature point are divided using a nearest neighbor detection algorithm, and the feature point is pointed to each local neighborhood feature point to construct a neighborhood vector. Taking the horizontal rightward direction of the feature point as the reference direction, all neighborhood vectors and their corresponding local neighborhood feature points are sorted in ascending order of the angle between each neighborhood vector and the reference direction; for the sorted neighborhood vectors and local neighborhood feature points of the two feature points, the angle difference between the two neighborhood vectors at the same index is calculated; the structural consistency of the matching pair is evaluated based on the angle difference and the distance between the two feature points and their respective local neighborhood feature points; the structural consistency of the matching pair satisfies the following relationship: , It is the first Structural consistency of each matching pair This refers to the number of local neighborhood feature points for each of the two feature points in the matching pair. The index of the local neighborhood feature point. For the first The matching pair contains the first two feature points. The angular difference of each neighborhood vector For the first The first feature point contained in the matching pair is related to its second feature point. The distance between local neighborhood feature points For the first The matching pair contains the second feature point and its second feature point. The distance between local neighborhood feature points For all The average value, and Both are the tolerance variances for angle differences and distance deviations, which are preset. It is a natural exponential function; Obtain the structural consistency set corresponding to the initial matching pair set; based on the statistical characteristics of the structural consistency contained in the structural consistency set, adaptively determine the screening threshold of the current frame image, including: determining the information entropy of the current frame image through the probability distribution of each structural consistency in the structural consistency set, and using the high quantile and median quantile of the structural consistency set as two statistical anchor points; based on the empirical cumulative distribution function value of the information entropy of the current frame image within a preset historical time window, perform linear interpolation between the two statistical anchor points to generate the screening threshold of the current frame image, with the following relationship: , The adaptive filtering threshold for the current frame image; and These are two statistical anchor points, namely the high quantile and the median quantile of the structural consistency set, which are determined by the percentile method. The information entropy of the current frame image is the empirical cumulative distribution function value within a preset historical time window; The initial set of matching pairs is filtered using a screening threshold to obtain a set of filtered matching pairs. The location contribution weight of each matching pair is determined based on the extent to which the structural consistency of each matching pair in the set exceeds the screening threshold, and the extent and the location contribution weight are positively correlated. Weighted pose calculation is performed based on the localization contribution weights of all matching pairs to obtain the instantaneous pose of the UAV in the current frame image. This instantaneous pose is then used as an observation and input into a temporal filtering model. The model is then combined with the UAV's motion model for temporal fusion to output the final pose estimate of the UAV in the current frame image, thereby completing the accurate localization of the UAV.

2. The UAV positioning method according to claim 1, characterized in that, Establish the matching relationship between the first feature point set and the second feature point set, and generate an initial set of matching pairs, including: The current frame image and the reference image are detected by a feature point detection algorithm to obtain a first feature point set consisting of all feature points in the current frame image and a second feature point set consisting of all feature points in the reference image. The first feature point set and the second feature point set are processed by semantic segmentation to obtain a first semantic label for each feature point in the first feature point set and a second semantic label for each feature point in the second feature point set. Based on the first semantic label and the second semantic label, the feature point matching algorithm is used to match only feature points with the same semantic label to generate an initial set of matching pairs.

3. The UAV positioning method according to claim 1, characterized in that, The empirical cumulative distribution function value of the information entropy of the current frame image within a preset historical time window is determined based on the following method: A historical time window is formed from the moment the drone begins flight to the corresponding moment of the current frame image. The information entropy of each frame image within the historical time window is obtained to form a historical information entropy sequence. The empirical cumulative distribution function value of the information entropy of the current frame image in the historical information entropy sequence is obtained by counting the number of information entropy values ​​in the historical information entropy sequence that are less than or equal to the information entropy of the current frame image, and then dividing by the length of the historical information entropy sequence.

4. The UAV positioning method according to claim 3, characterized in that, The method for determining the location contribution weight of each matching pair based on the extent to which the structural consistency of each matching pair in the set exceeds the screening threshold is as follows: For each matching pair, the magnitude by which the structural consistency of the matching pair exceeds the adaptive screening threshold is normalized. The normalization result is then subjected to a positively correlated nonlinear mapping, and the resulting mapping value is used as the localization contribution weight of the matching pair.

5. The UAV positioning method according to claim 1, characterized in that, Before performing time-series fusion with the drone's motion model, the following operations are also performed: The reliability index of the observation is calculated by combining the information entropy of the current frame image and the average localization contribution weight of all matching pairs in the filtered matching pair set. The magnitude of the reliability index is negatively correlated with the information entropy and positively correlated with the average localization contribution weight. A Kalman filter is used as the time-series filtering model. A noise benchmark proportional to the prediction uncertainty covariance of the Kalman filter itself is determined. A dynamic scaling factor is calculated based on the reliability index of the observations. The magnitude of the dynamic scaling factor is negatively correlated with the reliability index of the observations. By applying the dynamic scaling factor to the noise benchmark, the observation noise covariance in the Kalman filter is dynamically adjusted to achieve adaptive fusion between the observations and the predictions of the motion model.

6. A machine vision-based unmanned aerial vehicle (UAV) positioning system, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the UAV positioning method as described in any one of claims 1-5.

Images