Aircraft visual navigation method and device based on single-point multi-view images

Abstract

The application relates to an aircraft visual navigation method and device based on single-point multi-view images, wherein the aircraft visual navigation method comprises the following steps: acquiring aerial images of an aircraft at different view angles at the same position and constructing a multi-view aerial image set; acquiring a reference image set with geographical coordinates; performing image matching based on the reference image set and the multi-view aerial image set, so as to obtain multiple initial positioning results of the aircraft at the same position; clustering the multiple initial positioning results, and taking the average value of the largest cluster after clustering as a first positioning result; constructing an initial positioning result point group based on the multiple initial positioning results, and calculating the distance distribution of the first positioning result in the initial positioning result point group to determine the confidence of the first positioning result; if the confidence is higher than a preset threshold, taking the first positioning result as the final positioning result of the aircraft at present, and performing flight navigation on the aircraft based on the final positioning result.

Description

Technical Field

[0001] This invention relates to the field of navigation, and more particularly to a visual navigation method and apparatus for aircraft based on single-point multi-view images. Background Technology

[0002] As a novel autonomous navigation method, aircraft visual navigation offers advantages over inertial and satellite navigation, including high precision, drift-free operation, and resistance to electromagnetic interference, making it an effective supplement to both. It primarily works by establishing a geometric mapping between real-time ground imaging of the aircraft and similar (or adjacent) areas in a satellite reference map, and then utilizing the geocoding information of the reference map to obtain the aircraft's navigation and positioning information.

[0003] In the field of absolute visual navigation for UAVs, deep learning-based cross-view image matching methods have gradually become the mainstream research approach. He et al. proposed the Salient Feature Partition Network (SFPN), a classification network based on saliency-based region segmentation, combining image context information. This network divides image features into background and foreground components, achieving region segmentation learning. Wang et al., building upon image context information and incorporating neighborhood information as auxiliary information, proposed the Local Pattern Network (LPN). This network employs a square ring feature partitioning strategy to mitigate the impact of UAV image rotation on cross-view image matching. Dai et al. proposed the Feature Segmentation and Region Alignment (FSRA) method based on the Transformer architecture, automatically segmenting regions according to the heatmap distribution of feature maps, making the model robust to positional shifts and scale changes. Deuser et al. proposed a training strategy (Sample4Geo) based on contrastive learning and symmetric InfoNCE loss. This architecture eliminates the need for aggregation modules, avoiding preprocessing steps such as feature alignment and normalization, and further improving the model's generalization ability. Shen et al. proposed the MCCG (Multiple Classifier for Cross-view Geolocalization) method, which captures rich discriminative information through multi-dimensional interaction, reducing information loss during feature extraction and improving the model's robustness. Xia et al. utilized contrastive learning to obtain global information from images, while employing a neighborhood space alignment module to align fine-grained features. The comprehensive utilization of multi-granular visual information ensured better feature representation. Wu et al. proposed the CAMP (contrastive attribute mining and position-aware partitioning) method, which employs a position-aware partitioning branch (PPB) and a contrastive attribute mining (CAM) strategy to enrich the feature representation of images.

[0004] Although these existing methods have made significant progress in evaluation metrics such as recall and mean precision, they still face problems of low accuracy and reliability when performing single-image matching in actual flight missions due to the difficulty of cross-view matching and the interference of complex factors such as image scale, scene structure changes, weather, and lighting. They cannot meet the actual flight positioning needs of UAVs.

[0005] Therefore, considering that existing UAV platforms generally have single-point multi-view observation capabilities, how to utilize these capabilities for navigation can be considered a new research direction. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an aircraft visual navigation method and device based on single-point multi-view images.

[0007] To achieve the above-mentioned objectives, this invention provides an aircraft visual navigation method based on single-point multi-view images, comprising the following steps: S1. Acquire aerial images taken by the aircraft from different perspectives at the same location and construct a multi-view aerial image set; S2. Obtain a set of reference images with geographic coordinates, and perform image matching based on the set of reference images and the set of multi-view aerial images to obtain multiple initial positioning results of the aircraft at the same location; S3. Cluster the multiple initial positioning results and take the average value of the largest cluster as the first positioning result of the aircraft; S4. Based on multiple initial positioning results, an initial positioning result point group is formed, and the distance distribution of the first positioning result in the initial positioning result point group is calculated to determine the confidence level of the first positioning result. If the confidence level is higher than a preset threshold, the first positioning result is taken as the current final positioning result of the aircraft, and the aircraft is used for flight navigation based on the final positioning result.

[0008] According to one aspect of the present invention, in step S1, the step of acquiring aerial images taken by the aircraft from different perspectives at the same location and constructing a multi-view aerial image set, wherein the multi-view aerial image set is obtained based on the following steps, which include: Control the aircraft to hover in the air, and while ensuring that the center position of the aircraft remains unchanged, adjust the yaw direction of the aircraft's nose or the camera orientation and pitch angle of the onboard camera to acquire aerial images from multiple perspectives. The aerial images from multiple perspectives are labeled to obtain the multi-view aerial image set; Alternatively, in step S1, the step of acquiring aerial images taken by the aircraft from different perspectives at the same location and constructing a multi-view aerial image set is obtained based on the following steps, which include: Multiple cameras facing different directions are installed on the aircraft; wherein the orientation and / or pitch angle of the multiple cameras are different; Control the aircraft to hover in the air and acquire aerial images from multiple perspectives based on the camera; The aerial images from multiple perspectives are labeled to obtain the multi-view aerial image set.

[0009] According to one aspect of the present invention, in step S1, in the step of acquiring aerial images taken by the aircraft from different perspectives at the same position and constructing a multi-view aerial image set, the overlap rate of any two aerial images in the multi-view aerial image set is less than or equal to 1%.

[0010] According to one aspect of the present invention, step S2, which involves performing image matching based on the reference image set and the multi-view aerial image set to obtain multiple initial positioning results of the aircraft at the same location, includes: S21. Establish geometric mapping relationships between the reference images in the reference image set and the aerial images in the multi-view aerial image set, wherein the geometric mapping relationship is expressed as: ; ; ; in, Indicates the estimated position of the aircraft. Represents geometric mapping relationships, Represents the pixel coordinates in the aerial image. and This represents the pixel coordinates in the aerial image. Indicates the geometric transformation parameters of the aerial image, subscript Indicates the sequence number of the aerial image. This represents the pixel coordinates in the reference image. and This represents the pixel coordinates in the reference image; S22. Using the geometric mapping relationship and camera imaging parameters, calculate multiple initial positioning results for the aircraft.

[0011] According to one aspect of the present invention, in step S22, in the step of calculating multiple initial positioning results of the aircraft using the geometric mapping relationship and camera imaging parameters, a single-point position calculation method is used to estimate the position of the aircraft. The solution process includes: S220. After matching the reference image with the aerial image, obtain the pixel coordinates of the aerial image. Pixel coordinates in the matching reference image And, pixel coordinates are obtained based on geographic information from the reference image. The corresponding center coordinates of the aerial image ( ); S221. Obtain aerial image parameters and camera imaging parameters; wherein, the aerial image parameters include: the center coordinates of the aerial image (… Image size () UAV pitch angle UAV yaw angle Drone roll angle and drone altitude z The camera imaging parameters include: camera field of view ( ), camera pitch angle relative to the drone yaw angle of camera relative to drone The roll angle of the camera relative to the drone ; S222. Construct the UAV body attitude rotation matrix in the geographic coordinate system, and represent it as follows: ; ; ; in, These represent the rotation matrices of the UAV's body attitude in each direction under the geographic coordinate system; S223. Construct the camera mounting rotation matrix in the geographic coordinate system, and represent it as follows: ; ; ; in, These represent the camera mounting rotation matrices for each direction of the camera in the geographic coordinate system. S224. Construct a transformation matrix from world coordinates to camera coordinates by merging the body attitude rotation matrix and the camera mounting rotation matrix. Then the transformation matrix from the camera coordinate system to the world coordinate system can be expressed as the transpose: ; wherein, the transformation matrix Represented as: ; in, The transformation matrix representing the transition from Northeast Heaven to East Earth North is a known, fixed matrix; S225. Transformation matrix from world coordinates to camera coordinates Calculate the camera's position in the world coordinate system Then transform to the camera coordinate system to obtain the translation vector. ; S226. Ignoring surface undulations, transform the matrix... The first two columns and translation vectors spliced ​​into a camera projection matrix And it is represented as: ; in, , , , , , Represents the transformation matrix The first two columns of elements, , , Represents the translation vector Three elements; S227. Based on the camera projection matrix The estimated position of the aircraft corresponding to each aerial image is calculated to obtain the initial positioning results.

[0012] According to one aspect of the invention, in step S227, based on the camera projection matrix... The step of calculating the estimated position of the aircraft corresponding to each aerial image to obtain the initial positioning results includes: S2271. Based on the camera's field of view ( ) and the image size of the aerial image ( Calculate the camera's lateral equivalent focal length. and longitudinal equivalent focal length And respectively represented as: ; ; S2272. Based on lateral equivalent focal length and longitudinal equivalent focal length And combined with the center point pixel coordinates of the aerial image Construct the camera intrinsic parameter matrix And it is represented as: ; S2273. Based on the camera projection matrix and the camera intrinsic parameter matrix Construct the homography matrix And using the inverse of the homography matrix and the homogeneous coordinates of the center pixel of the aerial image, Multiplying them yields the matching position coordinates in the geographic coordinate system and the estimated position of the aircraft. The relative position vector on the horizontal plane, and expressed as: ; in, Represents a relative position vector. This represents the homogeneous pixel coordinates of the center point of the environmental image. , , Represents the relative position vector The three elements in; S2274. Obtain the estimated position of the aircraft based on the relative position vector. The initial positioning result is expressed as: ; ; in, , These represent the position coordinates of the aircraft.

[0013] According to one aspect of the present invention, step S3, which involves clustering the multiple initial positioning results and taking the average value of the largest cluster as the first positioning result of the aircraft, includes: S31. The initial positioning results are clustered using a density-based spatial clustering algorithm to divide them into multiple different clusters; S32. Obtain the largest cluster among multiple different clusters, and calculate the average value of the largest cluster to obtain the first positioning result.

[0014] According to one aspect of the present invention, in step S4, in the step of calculating the distance distribution of the first positioning result in the initial positioning result point group to determine the confidence level of the first positioning result, Mahalanobis distance is used to calculate the distance distribution of the first positioning result in the initial positioning result point group, and the confidence level of the first positioning result is determined based on the cumulative distribution function of the Mahalanobis distance; wherein, the confidence level is expressed as: ; in, Indicates the confidence level. Represents Mahalanobis distance, The cumulative distribution function representing the Mahalanobis distance, This indicates the dimension of the vector involved.

[0015] According to one aspect of the present invention, step S4, in which the distance distribution of the first positioning result in the initial positioning result point group is calculated using Mahalanobis distance, includes: S41. Calculate the initial positioning result point group. The mean vector of longitude and latitude in the initial positioning results, expressed as: ; in, Represents the mean vector of longitude and latitude. This represents the longitude value in the mean vector. This represents the dimension values ​​in the mean vector. This represents the longitude value in the initial positioning result. This represents the latitude value in the initial positioning result. A marker indicating the initial localization result; S42. Calculate the initial positioning result point group based on the mean vector. The covariance of the initial positioning results, expressed as: ; in, Describing covariance, Indicates the first The position vector of the initial positioning result; S43. The inverse covariance of the covariance is solved numerically, and the Mahalanobis distance is obtained based on the inverse covariance, expressed as: ; in, This represents the position vector of the first localization result. This indicates the inverse of the covariance.

[0016] To achieve the above-mentioned objectives, the present invention provides an aircraft visual navigation device based on single-point multi-view images, comprising: The acquisition module is used to acquire aerial images taken by the aircraft from different perspectives at the same location and to construct a multi-view aerial image set; The initial positioning module is used to acquire a reference image with geographic coordinates and perform image matching based on the reference image and the aerial images in the multi-view aerial image set to obtain multiple initial positioning results of the aircraft at the same location. The clustering module is used to cluster multiple initial positioning results and take the average value of the largest cluster after clustering as the first positioning result of the aircraft. The output module is used to obtain an initial positioning result point group based on multiple initial positioning results, and calculate the distance distribution of the first positioning result in the initial positioning result point group to determine the confidence level of the first positioning result. If the confidence level is higher than a preset threshold, the first positioning result is taken as the current final positioning result of the aircraft, and the aircraft is used for flight navigation based on the final positioning result.

[0017] According to one aspect of the present invention, this approach can fully utilize the single-point multi-view observation capability of the aircraft, fuse multi-view observation results, and effectively avoid the problem of large positioning error and low reliability caused by single image matching errors, thereby improving the robustness of visual positioning results.

[0018] According to one aspect of the present invention, this approach can fully utilize aerial images captured from multiple perspectives to assist in positioning, thereby significantly improving the accuracy and reliability of the visual navigation process.

[0019] According to one aspect of the present invention, this approach effectively improves the accuracy and reliability of the positioning system through multi-view image matching, localization result clustering, and confidence assessment.

[0020] According to one aspect of the present invention, multiple UAV images from different perspectives are captured under flight conditions. The estimated positions of multiple aircraft are obtained using a single-point localization method. These estimated positions are then input into a clustering model to obtain the final aircraft localization result. Finally, the confidence level of the final result is calculated using point clustering to evaluate the reliability of the localization result. This approach fully utilizes the differences between multi-view images under UAV flight conditions. Multiple images from different perspectives are located independently and simultaneously, effectively avoiding large localization deviations caused by errors in single-image localization. The calculation of confidence levels improves the usability of visual navigation results in engineering applications. With high confidence levels, the localization results of visual navigation are fully utilized, thus overcoming the unreliability of localization results from single-image captures under flight conditions.

[0021] According to one aspect of the present invention, this approach can effectively improve the positioning accuracy and reliability of image matching-based visual navigation algorithms. It can be applied to monocular visual navigation tasks on various flight platforms such as UAVs and airships, and has broad application prospects and economic value.

[0022] According to one aspect of the present invention, this approach classifies the initial positioning results by introducing clustering, which can effectively eliminate scattered noise points in the positioning results and improve the reliability of the positioning results.

[0023] According to one aspect of the present invention, a confidence estimation method is further introduced. This method fully utilizes the distribution among the positioning point group to calculate a specific distance of the final positioning result within the point group. The confidence level of the final measurement value is then calculated by leveraging the property that the distance distribution follows a certain distribution function. The confidence level of the final result exhibits a certain positive correlation with the positioning accuracy, thereby further reducing the positioning error of the present invention through a higher confidence level.

[0024] According to one aspect of the present invention, by taking pictures of the surrounding scene from different perspectives at the same spatial location, this approach can make full use of the scene information around the aircraft and effectively avoid the problems of insufficient matching accuracy and inaccurate position calculation that may exist in image matching and positioning algorithms based on single-frame images. Attached Figure Description

[0025] Figure 1This diagram illustrates the steps of an aircraft visual navigation method based on single-point multi-view images, according to one embodiment of the present invention. Detailed Implementation

[0026] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. The embodiments cannot be described in detail here, but the embodiments of the present invention are not limited to the following embodiments.

[0027] like Figure 1 As shown, according to one embodiment of the present invention, an aircraft visual navigation method based on single-point multi-view images includes the following steps: S1. Acquire aerial images taken by the aircraft from different perspectives at the same location and construct a multi-view aerial image set; S2. Obtain a set of reference images with geographic coordinates, and perform image matching based on the set of reference images and a set of multi-view aerial images to obtain multiple initial positioning results of the aircraft at the same location; S3. Cluster the multiple initial positioning results and take the average of the largest cluster as the first positioning result of the aircraft; S4. Based on multiple initial positioning results, an initial positioning result point group is formed, and the distance distribution of the first positioning result in the initial positioning result point group is calculated to determine the confidence level of the first positioning result. If the confidence level is higher than a preset threshold, the first positioning result is taken as the current final positioning result of the aircraft, and the aircraft is used for flight navigation based on the final positioning result.

[0028] According to one embodiment of the present invention, in step S1, the step of acquiring aerial images taken by the aircraft from different perspectives at the same location and constructing a multi-view aerial image set, the aircraft can be set as a drone, and the estimated position of the aircraft is recorded. ,in, , , These represent the position coordinates of the aircraft, ignoring surface undulations. Then the estimated position of the aircraft can also be expressed as Then, the set of multi-view aerial images obtained from different perspectives at the same location (single point) is represented as: ,in, This represents a collection of aerial images taken from multiple perspectives. , , … This refers to aerial images taken from various perspectives.

[0029] According to one embodiment of the present invention, in step S1, the step of acquiring aerial images taken by the aircraft from different perspectives at the same location and constructing a multi-view aerial image set is obtained based on the following steps, which include: Control the aircraft to hover in the air, and while ensuring that the center position of the aircraft remains unchanged, adjust the yaw direction of the aircraft's nose or the orientation and pitch angle of the onboard camera to acquire aerial images from multiple perspectives. Aerial images from multiple perspectives are labeled to obtain a multi-view aerial image set.

[0030] The above settings can effectively reduce the number of cameras carried, thereby reducing the load and cost of the aircraft. In addition, by controlling the aircraft and cameras to acquire multi-view aerial images, the acquisition method and angle are more flexible. It is more convenient to acquire a wider field of view and focus on acquiring areas of interest, which is more beneficial to ensuring the accuracy and efficiency of subsequent image matching.

[0031] According to another embodiment of the present invention, in step S1, the step of acquiring aerial images taken by the aircraft from different perspectives at the same location and constructing a multi-view aerial image set is obtained based on the following steps, which include: Multiple cameras facing different directions are installed on the aircraft; the orientation and / or pitch angle of the multiple cameras are different. Control the aircraft to hover in the air and capture aerial images from multiple perspectives using cameras; Aerial images from multiple perspectives are labeled to obtain a multi-view aerial image set.

[0032] By employing multiple cameras for multi-view photography, the aircraft can simultaneously capture aerial images from multiple perspectives while hovering in the air. This ensures a more accurate and stable center position for the aircraft, significantly improving the positioning accuracy of this solution. Furthermore, using multiple cameras for multi-view photography allows for direct annotation of each captured aerial image with its fixed installation location and corresponding camera parameters. This results in higher efficiency and better matching accuracy during the image matching calculation process.

[0033] According to one embodiment of the present invention, in step S1, in the step of acquiring aerial images taken by the aircraft from different perspectives at the same position and constructing a multi-view aerial image set, the overlap rate of any two aerial images in the multi-view aerial image set is less than or equal to 1%.

[0034] By setting the overlap rate of any two aerial images within the aforementioned range, this invention allows for significant differences in viewing angles when the aircraft images from the same location. Therefore, assuming the initial positioning results of each image are independent events, clustering them yields a maximum cluster with… n When capturing multiple aerial images, the probability of correctly matching and locating a single aerial image is: p This can increase the probability of correct matching and positioning to 100%. .

[0035] According to one embodiment of the present invention, step S2, which involves image matching based on a reference image set and a multi-view aerial image set to obtain multiple initial positioning results of the aircraft at the same location, includes: S21. Establish geometric mapping relationships between the reference images in the reference image set and the aerial images in the multi-view aerial image set. In this embodiment, the aerial images in the multi-view aerial image set are respectively mapped... Geographically labeled reference images (such as satellite or aerial images) are used for image matching to establish geometric mapping relationships between the reference images and aerial images in a multi-view aerial image set, which can be represented as: ; ; ; in, Indicates the estimated position of the aircraft. Represents geometric mapping relationships, Represents the pixel coordinates in the aerial image. and This represents the pixel coordinates in the aerial image. The subscript represents the geometric transformation parameters (such as translation, rotation, scaling, etc.) of the aerial image. Indicates the sequence number of the aerial image. This represents the pixel coordinates in the reference image. and This represents the pixel coordinates in the reference image; S22. Multiple initial positioning results of the aircraft are calculated using geometric mapping relationships and camera imaging parameters. In this embodiment, the classic PnP (Perspective-n-Point) algorithm or its variants can be used to solve for the geometric mapping relationships. And camera imaging parameters are used to calculate the estimated position of the UAV. Therefore, multiple initial positioning results for the same location can be obtained from aerial images taken from different perspectives. .

[0036] According to one embodiment of the present invention, in step S2, the step of acquiring a reference image with geographic coordinates, the geographic coordinates are absolute geographic coordinates and are expressed in longitude and latitude.

[0037] According to one embodiment of the present invention, in step S22, in the step of calculating multiple initial positioning results of the aircraft using the geometric mapping relationship and camera imaging parameters, a single-point position calculation method is used to estimate the position of the aircraft. The solution process includes: S220. After matching the reference image with the aerial image, obtain the pixel coordinates of the aerial image. Pixel coordinates in the matching reference image And, pixel coordinates are obtained based on geographic information from the reference image. The corresponding center coordinates of the aerial image ( ); S221. Obtain aerial image parameters and camera imaging parameters; wherein, the aerial image parameters include: the center coordinates of the aerial image ( Image size () UAV pitch angle UAV yaw angle Drone roll angle and drone altitude z The camera imaging parameters include: camera field of view ( ), camera pitch angle relative to the drone yaw angle of camera relative to drone The roll angle of the camera relative to the drone ; S222. Construct the UAV body attitude rotation matrix in the geographic coordinate system, and represent it as follows: ; ; ; in, These represent the rotation matrices of the UAV's body attitude in each direction under the geographic coordinate system; S223. Construct the camera mounting rotation matrix in the geographic coordinate system, and represent it as follows: ; ; ; in, These represent the camera mounting rotation matrices for each direction of the camera in the geographic coordinate system. S224. Construct a transformation matrix from world coordinates to camera coordinates by merging the body attitude rotation matrix and the camera mounting rotation matrix. Then the transformation matrix from the camera coordinate system to the world coordinate system can be expressed as the transpose: Wherein, the rotation matrix is ​​an identity orthogonal matrix, and the transformation matrix... Represented as: ; in, The transformation matrix representing the transition from Northeast Heaven to East Earth North is a known, fixed matrix; S225. Transformation matrix from world coordinates to camera coordinates Calculate the position of the camera in the world coordinate system Then transform to the camera coordinate system to obtain the translation vector. ; S226. Ignoring surface undulations, transform the matrix... The first two columns and translation vectors spliced ​​into a camera projection matrix And it is represented as: ; in, , , , , , Represents the transformation matrix The first two columns of elements, , , Represents the translation vector There are three elements in the middle, namely the first, second, and third elements; In this embodiment, the condition of ignoring surface undulations is... Thus transforming the matrix The third column can be ignored, thus enabling the translation vector to be manipulated. The stitching. In addition, the camera projection matrix... This is the extrinsic parameter matrix of the camera; S227. Based on camera projection matrix The position of the aircraft corresponding to each aerial image is calculated to obtain the initial positioning results.

[0038] According to one embodiment of the present invention, in step S227, based on the camera projection matrix... The step of calculating the position of the aircraft corresponding to each aerial image to obtain the initial positioning results includes: S2271. Based on the camera's field of view ( ) and the image size of the aerial image ( Calculate the camera's lateral equivalent focal length. and longitudinal equivalent focal length And respectively represented as: ; ; S2272. Based on lateral equivalent focal length and longitudinal equivalent focal length And combined with the center point pixel coordinates of the aerial image Construct the camera intrinsic parameter matrix And it is represented as: ; S2273. Based on camera projection matrix and the camera intrinsic parameter matrix Construct the homography matrix And using the inverse of the homography matrix and the homogeneous coordinates of the center pixel of the aerial image, Multiplying them yields the geographic coordinate system (based on the transformation matrix). The matching position coordinates (i.e., the coordinates of the center point of the aerial image in the world coordinate system, i.e., the center coordinates of the aerial image) under the inverse matrix transformation to the geographic coordinate system, and the estimated position of the aircraft. The relative position vector on the horizontal plane, and expressed as: ; in, Represents a relative position vector. This represents the homogeneous pixel coordinates of the center point of the environmental image. , , Represents the relative position vector The three elements in; S2274. Obtaining the estimated position of the aircraft based on relative position vectors. The initial positioning result is expressed as: ; .

[0039] According to one embodiment of the present invention, step S3, which involves clustering multiple initial positioning results and taking the average value of the largest cluster as the first positioning result of the aircraft, includes: S31. A density-based spatial clustering algorithm is used to cluster the initial positioning results to divide them into multiple different clusters. In this embodiment, the density-based spatial clustering algorithm is an unsupervised machine learning algorithm that can organize and classify the initial positioning results of aerial images from different perspectives at the same location into different clusters based on similarity, eliminating scattered noise points in the positioning results and improving the reliability of the positioning results. Specifically, the DBSCAN clustering method (Density-Based Spatial Clustering of Applications with Noise) can be used to cluster multiple positioning results of the same location from different UAV images. The DBSCAN clustering method can automatically divide the data into k different clusters based on data density. Furthermore, it can discover clusters of arbitrary shapes even in spatial domains with noisy data. Neighborhood-based clustering effectively captures the locality of geographic data, making it easier to group adjacent points into the same cluster and automatically identifying and eliminating noisy points. The DBSCAN clustering method only requires predefining two parameters, one of which is the distance threshold. The first is used to define the range of the "neighborhood," which determines the maximum distance that a point's neighborhood can contain other points. The second is the counting threshold, MinPts, which defines the minimum number of neighboring points required for a point to become a "core point." This allows for the elimination of single-frame matching and localization results with large errors (or erroneous results) from multi-view image localization after inputting the results, resulting in the largest cluster after clustering.

[0040] In a specific embodiment, the distance threshold of the "DBSCAN clustering model" The counting threshold parameter MinPts can be determined based on the specific requirements of the mission and flight conditions. For example, when high positioning accuracy is required, the distance threshold can be lowered, while when there are many multi-view images from the UAV, the counting threshold can be increased.

[0041] The specific steps are as follows: For the multiple initial localization results obtained (i.e., localization results from multiple single-frame images from multiple viewpoints), the distance between each pair of points is calculated using the Haversine formula. Based on distance Perform DBSCAN clustering operations.

[0042] Set all points to an unvisited state, randomly select one point to visit, and then proceed based on a distance threshold. The system uses two parameters, a counting threshold (MinPts), to determine if a point is a noise point. If it is a core point, it recursively searches for all points in its neighborhood and adds them to the same cluster. If it is a boundary point, it is assigned to the nearest cluster. This process is repeated until all points have been visited.

[0043] By combining all the final clustering results for analysis, the points within the cluster with the highest density are selected as valid predicted values, while all other points are considered noise points. The center point of this cluster (i.e., the average value of all points within the cluster) is used as the final predicted value.

[0044] If no clusters are found, it means there are no three or more predicted drone positions within the preset range (e.g., 50 meters), and the distance threshold will be further increased. Cluster the values ​​and, based on the actual task requirements, use a distance threshold. Clustering becomes meaningless when the distance threshold is too large. Therefore, when the distance threshold is too large, clustering becomes meaningless. When a certain threshold is reached, such as 200 meters, it indicates that the positioning from different perspectives is very scattered. At this point, the average value of the two closest points is used as the final prediction result.

[0045] S32. Obtain the largest cluster among multiple different clusters, and calculate the average value of the largest cluster to obtain the first localization result.

[0046] According to one embodiment of the present invention, in step S4, the step of calculating the distance distribution of the first positioning result in the initial positioning result point group to determine the confidence level of the first positioning result uses Mahalanobis distance to calculate the distance distribution of the first positioning result in the initial positioning result point group, and determines the confidence level of the first positioning result based on the cumulative distribution function of Mahalanobis distance; specifically, for the obtained multiple initial positioning results Confidence estimation is performed, and for the point group Q formed by multiple positioning results, the Mahalanobis distance of the first positioning result after clustering is calculated in point group Q. The Mahalanobis distance follows a chi-square distribution, and the confidence level of the final result is calculated using the cumulative distribution function (CDF) of the chi-square distribution; where the confidence level is expressed as: ; in, Indicates the confidence level. Represents Mahalanobis distance, The cumulative distribution function representing the Mahalanobis distance, This indicates the dimension of the vector involved.

[0047] With the above settings, the obtained confidence level can be used to evaluate the reliability of the current first positioning result.

[0048] According to one embodiment of the present invention, step S4, which involves using Mahalanobis distance to calculate the distance distribution of the first positioning result in the initial positioning result point group, includes: S41. Calculate the initial positioning result point group The mean vector of longitude and latitude in the initial positioning results, expressed as: ; in, Represents the mean vector of longitude and latitude. This represents the longitude value in the mean vector. This represents the dimension values ​​in the mean vector. This represents the longitude value in the initial positioning result. This represents the latitude value in the initial positioning result. A marker indicating the initial localization result; S42. Calculate the initial positioning result point group based on the mean vector. The covariance of the initial positioning results, expressed as: ; in, Describing covariance, Indicates the first The position vector of the initial positioning result; S43. Solve for the inverse covariance of the covariance numerically, and obtain the Mahalanobis distance based on the inverse covariance, expressed as: ; in, This represents the position vector of the first localization result. This indicates the inverse of the covariance.

[0049] According to one embodiment of the present invention, in the step of taking the first positioning result as the current final positioning result of the aircraft and performing flight navigation on the aircraft based on the final positioning result if the confidence level is higher than a preset threshold, the preset threshold used for confidence level judgment can be dynamically adjusted according to mission requirements: when high-frequency output of positioning results is required, the threshold can be reduced to improve response speed; when the positioning reliability requirement is high, the threshold can be increased to ensure accuracy.

[0050] According to one embodiment of the present invention, an aircraft visual navigation device based on single-point multi-view images includes: an acquisition module, an initial positioning module, a clustering module, and an output module. The acquisition module acquires aerial images taken by the aircraft from different perspectives at the same location and constructs a multi-view aerial image set. The initial positioning module acquires a reference image with geographic coordinates and performs image matching between the reference image and the aerial images in the multi-view aerial image set to obtain multiple initial positioning results for the aircraft at the same location. The clustering module clusters the multiple initial positioning results and takes the average value of the largest cluster as the first positioning result for the aircraft. The output module acquires a point cluster of initial positioning results based on the multiple initial positioning results and calculates the distance distribution of the first positioning result in the point cluster to determine the confidence level of the first positioning result. If the confidence level is higher than a preset threshold, the first positioning result is taken as the current final positioning result for the aircraft, and flight navigation is performed based on the final positioning result.

[0051] Specific limitations regarding the aircraft visual navigation device based on single-point multi-view images can be found in the limitations of the aircraft visual navigation method based on single-point multi-view images mentioned above, and will not be repeated here. Each module in the aforementioned aircraft visual navigation device based on single-point multi-view images can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0052] In this embodiment, the memory may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0053] In this embodiment, the processor can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0054] The above description is merely an example of a specific solution of the present invention. For any devices and structures not described in detail herein, it should be understood that they are implemented using common devices and methods already available in the art.

[0055] The above description is merely one embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An aircraft visual navigation method based on single-point multi-view images, characterized in that, Includes the following steps: S1. Acquire aerial images taken by the aircraft from different perspectives at the same location and construct a multi-view aerial image set; S2. Obtain a set of reference images with geographic coordinates, and perform image matching based on the set of reference images and the set of multi-view aerial images to obtain multiple initial positioning results of the aircraft at the same location; S3. Cluster the multiple initial positioning results and take the average value of the largest cluster as the first positioning result of the aircraft; S4. Based on multiple initial positioning results, an initial positioning result point group is formed, and the distance distribution of the first positioning result in the initial positioning result point group is calculated to determine the confidence level of the first positioning result. If the confidence level is higher than a preset threshold, the first positioning result is taken as the current final positioning result of the aircraft, and the aircraft is used for flight navigation based on the final positioning result.

2. The aircraft visual navigation method based on single-point multi-view images according to claim 1, characterized in that, In step S1, the step of acquiring aerial images taken by the aircraft from different perspectives at the same location and constructing a multi-view aerial image set is obtained based on the following steps: Control the aircraft to hover in the air, and while ensuring that the center position of the aircraft remains unchanged, adjust the yaw direction of the aircraft's nose or the camera orientation and pitch angle of the onboard camera to acquire aerial images from multiple perspectives. The aerial images from multiple perspectives are labeled to obtain the multi-view aerial image set; Alternatively, in step S1, the step of acquiring aerial images taken by the aircraft from different perspectives at the same location and constructing a multi-view aerial image set is obtained based on the following steps, which include: Multiple cameras facing different directions are installed on the aircraft; wherein the orientation and / or pitch angle of the multiple cameras are different; Control the aircraft to hover in the air and acquire aerial images from multiple perspectives based on the camera; The aerial images from multiple perspectives are labeled to obtain the multi-view aerial image set.

3. The aircraft visual navigation method based on single-point multi-view images according to claim 2, characterized in that, In step S1, the step of acquiring aerial images taken by the aircraft from different perspectives at the same location and constructing a multi-view aerial image set, wherein the overlap rate of any two aerial images in the multi-view aerial image set is less than or equal to 1%.

4. The aircraft visual navigation method based on single-point multi-view images according to claim 3, characterized in that, Step S2, which involves performing image matching based on the reference image set and the multi-view aerial image set to obtain multiple initial positioning results of the aircraft at the same location, includes: S21. Establish geometric mapping relationships between the reference images in the reference image set and the aerial images in the multi-view aerial image set, wherein the geometric mapping relationship is expressed as: ； ； ； in, Indicates the estimated position of the aircraft. Represents geometric mapping relationships, Represents the pixel coordinates in the aerial image. and This represents the pixel coordinates in the aerial image. Indicates the geometric transformation parameters of the aerial image, subscript Indicates the sequence number of the aerial image. This represents the pixel coordinates in the reference image. and This represents the pixel coordinates in the reference image; S22. Using the geometric mapping relationship and camera imaging parameters, calculate multiple initial positioning results for the aircraft.

5. The aircraft visual navigation method based on single-point multi-view images according to claim 4, characterized in that, In step S22, where multiple initial positioning results of the aircraft are calculated using the geometric mapping relationship and camera imaging parameters, a single-point position calculation method is used to estimate the aircraft's position. The solution process includes: S220. After matching the reference image with the aerial image, obtain the pixel coordinates of the aerial image. Pixel coordinates in the matching reference image And, pixel coordinates are obtained based on geographic information from the reference image. The corresponding center coordinates of the aerial image ( ); S221. Obtain aerial image parameters and camera imaging parameters; wherein, the aerial image parameters include: the center coordinates of the aerial image (… Image size () UAV pitch angle UAV yaw angle Drone roll angle and drone altitude z The camera imaging parameters include: camera field of view ( ), camera pitch angle relative to the drone yaw angle of camera relative to drone The roll angle of the camera relative to the drone ; S222. Construct the UAV body attitude rotation matrix in the geographic coordinate system, and represent it as follows: ； ； ； in, These represent the rotation matrices of the UAV's body attitude in each direction under the geographic coordinate system; S223. Construct the camera mounting rotation matrix in the geographic coordinate system, and represent it as follows: ； ； ； in, These represent the camera mounting rotation matrices for each direction of the camera in the geographic coordinate system. S224. Construct a transformation matrix from world coordinates to camera coordinates by merging the body attitude rotation matrix and the camera mounting rotation matrix. Then the transformation matrix from the camera coordinate system to the world coordinate system can be expressed as the transpose: ; wherein, the transformation matrix Represented as: ； in, The transformation matrix representing the transition from Northeast Heaven to East Earth North is a known, fixed matrix. S225. Transformation matrix from world coordinates to camera coordinates Calculate the camera's position in the world coordinate system Then transform to the camera coordinate system to obtain the translation vector. ; S226. Ignoring surface undulations, transform the matrix... The first two columns and translation vectors spliced ​​into a camera projection matrix And it is represented as: ； in, , , , , , Represents the transformation matrix The first two columns of elements, , , Represents the translation vector Three elements; S227. Based on the camera projection matrix The estimated position of the aircraft corresponding to each aerial image is calculated to obtain the initial positioning results.

6. The aircraft visual navigation method based on single-point multi-view images according to claim 5, characterized in that, In step S227, based on the camera projection matrix The step of calculating the estimated position of the aircraft corresponding to each aerial image to obtain the initial positioning results includes: S2271. Based on the camera's field of view ( ) and the image size of the aerial image ( Calculate the camera's lateral equivalent focal length. and longitudinal equivalent focal length And respectively represented as: ； ； S2272. Based on lateral equivalent focal length and longitudinal equivalent focal length And combined with the center point pixel coordinates of the aerial image Construct the camera intrinsic parameter matrix And it is represented as: ； S2273. Based on the camera projection matrix and the camera intrinsic parameter matrix Construct the homography matrix And using the inverse of the homography matrix and the homogeneous coordinates of the center pixel of the aerial image, Multiplying them yields the matching position coordinates in the geographic coordinate system and the estimated position of the aircraft. The relative position vector on the horizontal plane, and expressed as: ； in, Represents a relative position vector. This represents the homogeneous pixel coordinates of the center point of the environmental image. , , Represents the relative position vector The three elements in; S2274. Obtain the estimated position of the aircraft based on the relative position vector. The initial positioning result is expressed as: ； ； in, , These represent the position coordinates of the aircraft.

7. The aircraft visual navigation method based on single-point multi-view images according to claim 6, characterized in that, Step S3, which involves clustering the multiple initial positioning results and taking the average of the largest cluster as the first positioning result of the aircraft, includes: S31. The initial positioning results are clustered using a density-based spatial clustering algorithm to divide them into multiple different clusters; S32. Obtain the largest cluster among multiple different clusters, and calculate the average value of the largest cluster to obtain the first positioning result.

8. The aircraft visual navigation method based on single-point multi-view images according to claim 7, characterized in that, In step S4, the step of calculating the distance distribution of the first positioning result in the initial positioning result point group to determine the confidence level of the first positioning result involves using Mahalanobis distance to calculate the distance distribution of the first positioning result in the initial positioning result point group, and determining the confidence level of the first positioning result based on the cumulative distribution function of the Mahalanobis distance; wherein, the confidence level is expressed as: ； in, Indicates the confidence level. Represents Mahalanobis distance, The cumulative distribution function representing the Mahalanobis distance, This indicates the dimension of the vector involved.

9. The aircraft visual navigation method based on single-point multi-view images according to claim 8, characterized in that, Step S4, the step of using Mahalanobis distance to calculate the distance distribution of the first positioning result in the initial positioning result point group, includes: S41. Calculate the initial positioning result point group. The mean vector of longitude and latitude in the initial positioning results, expressed as: ； in, Represents the mean vector of longitude and latitude. This represents the longitude value in the mean vector. This represents the dimension values ​​in the mean vector. This represents the longitude value in the initial positioning result. This represents the latitude value in the initial positioning result. A marker indicating the initial localization result; S42. Calculate the initial positioning result point group based on the mean vector. The covariance of the initial positioning results, expressed as: ； in, Describing covariance, Indicates the first The position vector of the initial positioning result; S43. The inverse covariance of the covariance is solved numerically, and the Mahalanobis distance based on the inverse covariance is obtained, expressed as: ； in, This represents the position vector of the first localization result. This indicates the inverse of the covariance.

10. An aircraft visual navigation device based on single-point multi-view images, characterized in that, include: The acquisition module is used to acquire aerial images taken by the aircraft from different perspectives at the same location and to construct a multi-view aerial image set; The initial positioning module is used to acquire a reference image with geographic coordinates and perform image matching based on the reference image and the aerial images in the multi-view aerial image set to obtain multiple initial positioning results of the aircraft at the same location. The clustering module is used to cluster multiple initial positioning results and take the average value of the largest cluster after clustering as the first positioning result of the aircraft. The output module is used to obtain an initial positioning result point group based on multiple initial positioning results, and calculate the distance distribution of the first positioning result in the initial positioning result point group to determine the confidence level of the first positioning result. If the confidence level is higher than a preset threshold, the first positioning result is taken as the current final positioning result of the aircraft, and the aircraft is used for flight navigation based on the final positioning result.

Images