Unmanned aerial vehicle aerial photograph cross-view registration positioning method based on instance matching

Abstract

The application belongs to the field of unmanned aerial vehicle remote sensing, satellite remote sensing and photogrammetry, and discloses a method for cross-view registration and positioning of unmanned aerial vehicle aerial image based on instance matching, which comprises the following steps: step S1, target entity of a building is extracted through instance segmentation; step S2, geometric center coarse registration is used to determine instance correspondence; step S3, point matching network is used to refine and extract the top center of the instance; step S4, ground position inversion and accurate homography matrix solving; and step S5, geographic coordinate calculation. The method for cross-view registration and positioning of unmanned aerial vehicle aerial image based on instance matching is adopted, and through the complete process of instance segmentation, coarse registration and correspondence determination, top center refinement, ground inversion and accurate homography solving, and geographic coordinate calculation, various errors can be effectively eliminated, and high-precision registration and positioning of unmanned aerial vehicle non-orthographic images can be realized.

Description

Technical Field

[0001] This invention relates to the fields of UAV remote sensing, satellite remote sensing and photogrammetry, and in particular to a cross-view registration and positioning method for UAV aerial images based on instance matching. Background Technology

[0002] Drone-based tilted aerial photography can simultaneously acquire the top and side texture information of buildings in a scene, enabling the reconstruction of the scene's 3D structure. It has wide applications in emergency mapping, urban modeling, and other fields. However, because drone images are non-orthophotos, they suffer from severe perspective distortion. The segmented building instances contain both top and side 3D structural information; whereas satellite orthophotos are taken perpendicular to the ground, and the segmented building instances only contain top planar information. The dimensional differences between the two types of images make direct matching prone to feature point mismatches and low registration accuracy, ultimately leading to inaccurate positioning of drone aerial images.

[0003] Traditional image registration methods based on global manual features (such as SIFT, ORB, and SURF) exhibit extremely poor robustness to changes in viewpoint and perspective distortion. In matching scenarios involving non-orthophoto UAV images and orthophoto satellite images, reprojection errors are large, failing to meet high-precision positioning requirements. Existing instance-based matching methods mostly use the overall geometric center of the instance for matching, without considering that the overall center of the instance (including the top and sides) in the non-orthophoto UAV image is not the same physical point as the top center of the instance in the satellite image. Furthermore, they ignore projection errors caused by building elevations, resulting in insufficient accuracy in homography matrix calculation and systematic biases in the positioning results.

[0004] In recent years, instance segmentation techniques, such as the Segment Anything Model (SAM), have been able to efficiently and generalizely extract building instance outlines from various scenes. Rotation-invariant point matching networks, such as SeS-Net, can achieve accurate matching of target regions from different viewpoints, providing a technical foundation for solving the matching problem between non-orthophoto images and orthophoto satellite images. However, existing technologies have not yet formed a complete instance-level matching and localization process for non-orthophoto UAV images, and cannot effectively eliminate matching errors caused by viewpoint distortion and building elevation, making it difficult to achieve high-precision localization of UAV images.

[0005] Therefore, there is an urgent need for a registration and positioning method for UAV non-orthophoto aerial images based on instance matching to solve the problems of low matching accuracy and large positioning error in the existing technology. Summary of the Invention

[0006] The purpose of this invention is to provide a cross-view registration and positioning method for UAV aerial images based on instance matching. Through a complete process of instance segmentation, coarse registration and correspondence determination, fine extraction of top center, ground-based inversion to obtain accurate homography, and geographic coordinate calculation, various errors are effectively eliminated, and high-precision registration and positioning of UAV non-orthophoto images are achieved.

[0007] To achieve the above objectives, this invention provides a cross-view registration and localization method for UAV aerial images based on instance matching, comprising the following steps: Step S1: Instance segmentation to extract target entities of buildings: Using rigid targets of buildings as matching primitives, instance segmentation models are used to segment the input UAV non-orthophoto image and the satellite orthophoto image to be matched, respectively, to extract the mask and contour of rigid target instances of buildings. Step S2: Coarse registration of geometric centers to determine the correspondence between instances: Calculate the overall geometric center of all building instances in the UAV non-orthophoto image and the satellite orthophoto image respectively. Based on the initial homography matrix solution and mismatch elimination algorithm, complete the coarse registration to determine the one-to-one correspondence between building instances in the UAV non-orthophoto image and the satellite orthophoto image. Step S3: Point matching network refines and extracts the top center of the instance: For each pair of successfully matched building instances, a point matching network with rotation invariance is used to extract the top area of ​​the building that perfectly matches the top plane contour of the satellite orthophoto from the UAV non-orthophoto image instance, and calculate the top geometric center. Step S4: Foundation Location Inversion and Accurate Homography Matrix Solving: Based on the top geometric center of the UAV non-orthophoto image instance mask obtained in Step S3 and the overall geometric center of the UAV non-orthophoto image instance mask obtained in Step S2, combined with perspective projection geometry, the projection coordinates of the building foundation on the UAV image plane are inverted; using the UAV image plane foundation coordinates of all buildings and the corresponding satellite orthophoto image instance mask geometric center coordinates as matching point pairs, the accurate homography matrix between the UAV image plane and the satellite orthophoto ground plane is solved. The pixel position of the center of the UAV non-orthophoto image in the satellite orthophoto image is obtained by homography matrix mapping; Step S5, Geographic Coordinate Calculation: Combining the geocoding information of the satellite image, the precise pixel position of the UAV non-orthophoto image center in the satellite orthophoto obtained in Step S4 is converted into real geographic coordinates, completing the high-precision positioning of the UAV non-orthophoto aerial image.

[0008] Preferably, in step S1, the building instance obtained by segmenting the UAV non-orthophoto image includes the complete three-dimensional structural outline of the top and sides; the building instance obtained by segmenting the satellite orthophoto image only includes the top planar outline. The instance segmentation model performs end-to-end instance segmentation on UAV non-orthophoto images and satellite orthophoto images, and outputs the binary mask, bounding rectangle and contour coordinates of each building instance; The instance segmentation models include: Segment Anything Model, Mask R-CNN, and YOLO instance segmentation branch.

[0009] Preferably, in step S2, the geometric center coarse registration determines the instance correspondence, and the specific process is as follows: Step S21: Calculate the geometric center of the binary mask for each building instance in the UAV non-orthophoto image and the satellite orthophoto image respectively; (1) For non-orthophoto images from UAVs, let the total number of instances contained therein be... The binary mask for each instance is denoted as . Through instance mask Calculate the instance mask using the average coordinates of all pixels within the area. geometric center As shown below: ; in, Instance mask in unorthophoto images from UAVs The total number of pixels within; Mask for instances of non-orthophoto images from drones Inner Image plane coordinates of 1 pixel; (2) For satellite orthophotos, let the total number of instances contained be... The binary mask for each instance is denoted as . Through instance mask Calculate the instance mask using the average coordinates of all pixels within the area. geometric center As shown below: ; in, Instance mask in satellite orthophoto The total number of pixels within; For example, a mask for satellite orthophoto instances Inner Image plane coordinates of 1 pixel; Step S22: Set the geometric centers of the instance masks of the UAV non-orthophoto image. Set of geometric centers of instance masks of satellite orthophotos As the initial matching point pairs, the initial homography matrix is ​​solved using either the RANSAC or PROSAC algorithm. ; Step S23: Calculate the reprojection error for each pair of matching points. As shown below: ; in, Indicates the first non-orthophoto in the drone image Geometric center coordinates of each matching instance ; Indicates the first in the satellite orthophoto image Geometric center coordinates of each matching instance Reprojection error The calculation involves determining the geometric center of the unorthophoto image from the UAV. Through the initial homography matrix After transformation to the satellite orthophoto coordinate system, the corresponding geometric center on the satellite orthophoto image. The Euclidean distance between them; Reprojection error Used to remove erroneous matching pairs, remove erroneous matching pairs with reprojection errors greater than a preset threshold, retain instance mask matching pairs with confidence levels higher than the threshold, and determine the one-to-one correspondence between building instances in UAV non-orthophoto images and satellite orthophoto images.

[0010] Preferably, in step S3, the point matching network refines and extracts the top center of the instance, and the specific process is as follows: Step S31: For each pair of building instance masks in the UAV orthophoto and satellite orthophoto, the mask for each building instance in the UAV orthophoto is... The geometric center coordinates are In satellite orthophotos and Corresponding building instance mask The geometric center coordinates are Each pair of matching instance masks is denoted as Both correspond to the same real building, and The geometric center is the same as the center of the top plane of the actual building; Step S32: Image registration is performed using a rotation-invariant point matching network to achieve... The top outline and The top contour is accurately aligned; the point matching network integrates wavelet multimodal features and rotation-invariant coding modules, and the point matching registration process is as follows: , ; in, This is the registration mapping matrix for the output; ( ) represents the elements of the mapping matrix; This indicates the network model selected by the point matching network. Step S33: Based on the registration mapping matrix Masking satellite orthophoto instances geometric center Mapping to instance mask of drone nonorthophoto image The image plane has the following mapping relationship: ; in, For the output drone non-orthophoto image instance mask The coordinates of the top geometric center.

[0011] Preferred point matching networks include: SeS-Net, GeoDesc, and HardNet.

[0012] Preferably, in step S4, based on the top geometric center of the instance mask of the UAV non-orthophoto image obtained in step S3 and the overall geometric center of the instance mask of the UAV non-orthophoto image obtained in step S2, and combined with the perspective projection geometry, the projection coordinates of the building foundation on the UAV image plane are obtained by inversion. The specific process is as follows: (1) The top geometric center of the UAV non-orthophoto image instance mask based on step S3, the coordinates of the top center of the building are: ;in, Mask for instances of non-orthophoto images from drones The x-coordinate of the top center image plane, Mask for instances of non-orthophoto images from drones The top center image plane ordinate; (2) The overall geometric center of the instance mask based on the UAV non-orthophoto image from step S2, and the coordinates of the overall center of the building are: ;in, Mask for instances of non-orthophoto images from drones The overall geometric center of the image plane abscissa Mask for instances of non-orthophoto images from drones The overall geometric center of the image plane ordinate; (3) Solve for the projected coordinates of the building foundation center. ,in Examples of drones The horizontal coordinate of the image plane of the foundation center. Examples of drones The solution process for the ordinate of the image plane of the foundation center is as follows: For two-dimensional building instance images obtained from non-orthophoto aerial photography by UAVs, in practical engineering applications, the instance mask of the non-orthophoto UAV image can be approximated. Global geometric center coordinates , is the top center coordinate of this instance. Projected coordinates of the building foundation center The midpoint of the coordinate system is the coordinate of the overall geometric center. The central constraint equations are shown below: ; Expanding into coordinate form, we get: ; .

[0013] Preferably, the solution for the projected coordinates of the foundation center is as follows: (1) Based on the central constraint equation, the instance mask of the UAV nonorthophoto image is obtained by algebraic inversion. The projection coordinates of the foundation center on the image plane As shown below: ; ; Exact homography matrix solution and UAV image center mapping; (2) Using the homogeneous vector of the UAV image plane ground coordinates of all matching buildings Homogeneous vector of the center coordinates of the corresponding satellite orthophoto instance To match point pairs, an overdetermined system of equations is established, as shown below: ; in, For exact homography matrix; For the first The image plane x-coordinate of the ground center projection point of a matching building instance in an unorthophoto image from a UAV; For the first The image plane ordinate of the ground center projection point of a matching building instance in an unorthophoto image from a UAV; For the first The x-coordinate of the geometric center of a matching building instance in a satellite orthophoto; For the first The ordinate of the geometric center of a matching building instance in a satellite orthophoto; Solve the homography matrix using the least squares method or the LM optimization algorithm. The optimization objective is: ; in, To effectively match the number of building instances; The index for matching building instances has a range of values. =1,2,…,M; (3) Based on the homography matrix obtained by the solution The homogeneous coordinates of the center of the non-orthophoto image of the UAV By mapping to the satellite orthorectified image coordinate system, the precise pixel position of the UAV's non-orthorectified image center in the satellite orthorectified image is obtained, as shown below: ; in, The homogeneous coordinates of the center of the UAV non-orthophoto image; The x-coordinate of the center of the UAV non-orthophoto image on the image plane; The vertical coordinate of the center of the UAV non-orthophoto image on the image plane; For the center of the UAV non-orthophoto image, use an accurate homography matrix After mapping, the homogeneous coordinates in the satellite orthophoto coordinate system, in their non-homogeneous form, represent the precise pixel positions after mapping.

[0014] Preferably, in step S5, the geographic coordinates are calculated, and the specific process is as follows: Step S51: Based on the geocoding file of satellite imagery, establish a conversion model between satellite image pixel coordinates and real geographic coordinates; wherein, the conversion model adopts affine transformation, first-order polynomial or second-order polynomial transformation model; The quadratic polynomial transformation model is shown below: ; ; in, Geographic coordinates: latitude and longitude; These are the pixel coordinates of the satellite image. , , , , , and , , , , , The parameters for the polynomial transformation are obtained by solving for the ground control points in the satellite imagery. Step S52: Substitute the precise pixel position of the UAV non-orthophoto image center obtained in step S4 into the satellite orthophoto into the transformation model to obtain the geographic coordinates corresponding to the UAV image center. Step S53: Combine the attitude parameters of the UAV camera to correct the perspective error of the geographic coordinates and obtain the final positioning result.

[0015] Therefore, the present invention employs the above-mentioned instance-matching-based cross-view registration and positioning method for UAV aerial images, which has the following beneficial effects: (1) Strong perspective robustness: Using rigid building instances as matching primitives and combining rotation-invariant point matching networks, it effectively overcomes perspective distortion caused by the non-orthogonal perspective of UAVs, solves the matching problem caused by the difference in dimensions between tilted images and orthogonal satellite images, and has good adaptability to tilted shooting scenes of 10°-45°.

[0016] (2) High matching accuracy: Through the two-stage matching process of “coarse matching and correspondence - fine extraction of top center”, the matching calculation is reduced and the matching error caused by the inconsistency between the overall center of the instance and the top center is completely eliminated. The accuracy of matching point pairs is more than 60% higher than that of traditional global feature methods.

[0017] (3) Excellent positioning accuracy: By using perspective projection geometry to invert the location of the building foundation, the homography matrix is ​​solved based on the ground plane point pairs, which completely eliminates the projection error caused by the building elevation. The accuracy of the homography matrix solution is greatly improved, and the final positioning accuracy in urban scenarios can reach sub-meter level.

[0018] (4) Highly practical: It does not require high-precision GIS data, lidar point cloud and other auxiliary information. Registration and positioning can be completed only through UAV images and public satellite images. Data acquisition is convenient and it can be adapted to a variety of complex scenarios such as emergency mapping, field navigation and urban modeling.

[0019] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0020] Figure 1 This is an overall flowchart of the cross-view registration and positioning method for UAV aerial images based on instance matching according to the present invention. Figure 2 This is a schematic diagram illustrating the principle of segmentation and coarse registration in this invention. Detailed Implementation

[0021] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Example 1 like Figure 1 As shown, the present invention provides a cross-view registration and localization method for UAV aerial images based on instance matching, comprising the following steps: Step S1: Extract the target entity of the building by instance segmentation.

[0023] Using rigid, stable targets such as buildings as matching primitives, instance segmentation models are applied to the input UAV non-orthophoto images and the satellite orthophoto images to be matched for instance segmentation. This extracts the masks and contours of rigid target instances such as buildings, separating them from complex backgrounds and avoiding the influence of ground texture, lighting, and occlusion on global features. It also clarifies the dimensional differences between the two types of image instances, providing a foundation for subsequent matching. The instance segmentation model can be any one of the SegmentAnything Model (SAM), Mask R-CNN, or the YOLO instance segmentation branch.

[0024] Building instances obtained from UAV non-orthophoto image segmentation include complete 3D structural outlines of the top and sides; building instances obtained from satellite orthophoto image segmentation only include the top planar outline; the instance segmentation model performs end-to-end instance segmentation on UAV non-orthophoto images and satellite orthophoto images, and outputs the binary mask, bounding rectangle and outline coordinates of each building instance.

[0025] Step S2: Coarse registration of geometric centers to determine the correspondence between instances.

[0026] The global geometric center of all building instances in both the UAV orthophoto image and the satellite orthophoto image is calculated. Based on the initial homography matrix solution and mismatch elimination algorithm, the system quickly identifies which building instance in the UAV orthophoto image corresponds to which building instance in the satellite orthophoto image, completing coarse registration and determining the one-to-one correspondence between building instances in the UAV orthophoto image and the satellite orthophoto image. This narrows down the scope for subsequent fine matching and reduces the computational load. Figure 2 As shown.

[0027] Step S21: Calculate the geometric center of the binary mask for each building instance in the UAV non-orthophoto image and the satellite orthophoto image respectively.

[0028] (1) For non-orthophoto images from UAVs, let the total number of instances contained therein be... The binary mask for each instance is denoted as . Through instance mask Calculate the instance mask using the average coordinates of all pixels within the area. geometric center As shown below: ; in, Instance mask in unorthophoto images from UAVs The total number of pixels within; Mask for instances of non-orthophoto images from drones Inner Image plane coordinates of pixels.

[0029] (2) For satellite orthophotos, let the total number of instances contained be... The binary mask for each instance is denoted as . Through instance mask Calculate the instance mask using the average coordinates of all pixels within the area. geometric center As shown below: ; in, Instance mask in satellite orthophoto The total number of pixels within; For example, a mask for satellite orthophoto instances Inner Image plane coordinates of pixels.

[0030] Step S22: Set the geometric centers of the instance masks of the UAV non-orthophoto image. Set of geometric centers of instance masks of satellite orthophotos As the initial matching point pairs, the initial homography matrix is ​​solved using either the RANSAC or PROSAC algorithm. .

[0031] Step S23: Calculate the reprojection error for each pair of matching points. As shown below: ; in, Indicates the first non-orthophoto in the drone image The geometric center coordinates of each matching instance, i.e., the geometric center of the UAV non-orthophoto image from the initial matching point pair in step S22. ; Indicates the first in the satellite orthophoto image The geometric center coordinates of each matching instance, i.e., the geometric center of the satellite orthophoto from the initial matching point pair in step S22. .

[0032] Reprojection error The calculation involves determining the geometric center of the unorthophoto image from the UAV. Through the initial homography matrix After transformation to the satellite orthophoto coordinate system, the corresponding geometric center on the satellite orthophoto image. The Euclidean distance between them.

[0033] Reprojection error Used to remove erroneous matching pairs, remove erroneous matching pairs with reprojection errors greater than a preset threshold, retain instance mask matching pairs with confidence levels higher than the threshold, and determine the one-to-one correspondence between building instances in UAV non-orthophoto images and satellite orthophoto images.

[0034] Step S3: Point matching network refinement to extract the top center of the instance.

[0035] For each successfully matched building instance, a rotation-invariant point matching network is used to accurately extract the top area of ​​the building that perfectly matches the top plane contour of the satellite orthophoto from the 3D structural instance containing the top and sides of the UAV non-orthophoto image. The top geometric center is calculated to solve the core problem that "the overall center of the UAV instance and the center of the satellite instance are not the same physical point" and eliminate the center offset error caused by the perspective distortion.

[0036] Step S31: For each pair of building instance masks in the UAV orthophoto and satellite orthophoto, the mask for each building instance in the UAV orthophoto is... The coordinates of the geometric center (including the top and side sides) are: In satellite orthophotos and Corresponding building instance mask The coordinates of the geometric center (only the center of the top plane) are: Each pair of matching instance masks is denoted as Both correspond to the same real building, and The geometric center is the center of the top plane of the actual building.

[0037] Step S32, due to the instance mask of the UAV's non-orthophoto image An instance mask of satellite orthophoto containing the two-dimensional outline of the building's top and sides. Only the two-dimensional outline of the building's top plane is included, for each pair of matching instances. Image registration is achieved using a rotation-invariant point matching network. The top outline and The top outline is precisely aligned.

[0038] The point matching network can be any one of SeS-Net, GeoDesc, or HardNet. The network integrates wavelet multimodal features and rotation-invariant coding modules. The point matching registration process is as follows: , ; in, This is the registration mapping matrix for the output; ( ) represents the elements of the mapping matrix; This indicates the network model selected by the point matching network.

[0039] Step S33: Based on the registration mapping matrix Masking satellite orthophoto instances geometric center Mapping to instance mask of drone nonorthophoto image The image plane has the following mapping relationship: ; in, For the output drone non-orthophoto image instance mask The coordinates of the top geometric center.

[0040] Step S4: Foundation location inversion and solution of accurate homography matrix.

[0041] To address the issue that "the top of the building and the foundation are not on the same ground plane, and directly using the top center to calculate homography will introduce elevation errors," the top geometric center of the instance mask of the UAV non-orthophoto image obtained in step S3 and the overall geometric center of the instance mask of the UAV non-orthophoto image obtained in step S2 are used, combined with perspective projection geometry, to invert the projection coordinates of the building foundation on the UAV image plane.

[0042] Using the ground-based coordinates of the UAV image plane of all buildings and the geometric center coordinates of the corresponding satellite orthorectified image instance mask as matching point pairs, solve for the exact homography matrix between the UAV image plane and the satellite orthorectified ground plane. The precise pixel position of the center of the UAV non-orthophoto image in the satellite orthophoto image is obtained by homography matrix mapping.

[0043] Step S41: Define the coordinates and approximate geometric constraints of the midpoint.

[0044] Based on the top geometric center of the UAV nonorthophoto image instance mask from step S3, the coordinates of the building top center are: ;in, Mask for instances of non-orthophoto images from drones The x-coordinate of the top center image plane, Mask for instances of non-orthophoto images from drones The top center image plane ordinate.

[0045] Based on the instance mask of the UAV orthophoto image obtained in step S2, the overall geometric center of the building is: ;in, Mask for instances of non-orthophoto images from drones The overall geometric center of the image plane abscissa Mask for instances of non-orthophoto images from drones The overall geometric center of the image plane ordinate.

[0046] Solve for the projected coordinates of the building foundation center ,in Examples of drones The horizontal coordinate of the image plane of the foundation center. Examples of drones The solution process for the ordinate of the image plane of the foundation center is as follows: For two-dimensional building instance images obtained from non-orthophoto aerial photography by UAVs, in practical engineering applications, the instance mask of the non-orthophoto UAV image can be approximated. Global geometric center coordinates , is the top center coordinate of this instance. Projected coordinates of the building foundation center The midpoint of the coordinate system is the coordinate of the overall geometric center. The central constraint equations are shown below: ; Expanding into coordinate form, we get: ; ; Step S42: Solve for the projected coordinates of the foundation center.

[0047] Based on the central constraint equation in step S41, the instance mask of the UAV nonorthophoto image is obtained by algebraic inversion. The projection coordinates of the foundation center on the image plane As shown below: ; ; Step S43: Solving for the exact homography matrix and mapping the UAV image center.

[0048] Homogeneous vectors of UAV image plane ground coordinates for all matching buildings Homogeneous vector of the center coordinates of the corresponding satellite orthophoto instance To match point pairs, an overdetermined system of equations is established, as shown below: ; in, For exact homography matrix; For the first The image plane x-coordinate of the ground center projection point of a matching building instance in an unorthophoto image from a UAV; For the first The image plane ordinate of the ground center projection point of a matching building instance in an unorthophoto image from a UAV; For the first The x-coordinate of the geometric center of a matching building instance in a satellite orthophoto; For the first The ordinate of the geometric center of each matching building instance in the satellite orthophoto.

[0049] Solve the homography matrix using either the least squares method or the LM optimization algorithm. The optimization objective is: ; in, To effectively match the number of building instances; The index for matching building instances has a range of values. =1,2,…,M.

[0050] Based on the homography matrix obtained by the solution The homogeneous coordinates of the center of the non-orthophoto image of the UAV By mapping to the satellite orthorectified image coordinate system, the precise pixel position of the UAV's non-orthorectified image center in the satellite orthorectified image is obtained, as shown below: ; in, The homogeneous coordinates of the center of the UAV non-orthophoto image; The x-coordinate of the center of the UAV non-orthophoto image on the image plane; The vertical coordinate of the center of the UAV non-orthophoto image on the image plane; For the center of the UAV non-orthophoto image, use an accurate homography matrix After mapping, the homogeneous coordinates in the satellite orthophoto coordinate system, in their non-homogeneous form, represent the precise pixel positions after mapping.

[0051] Step S5: Calculate accurate geographic coordinates.

[0052] By combining the geocoding information of satellite images, the precise pixel position of the UAV non-orthophoto image center in the satellite orthophoto obtained in step S4 is converted into real geographic coordinates, thus completing the high-precision positioning of the UAV non-orthophoto aerial image.

[0053] Step S51: Based on the geocoding file of the satellite image, establish a conversion model between the pixel coordinates of the satellite image and the real geographic coordinates; wherein, the conversion model adopts any one of the affine transformation, first-order polynomial, and second-order polynomial transformation models.

[0054] The quadratic polynomial transformation model is shown below: ; ; in, Geographic coordinates: latitude and longitude; These are the pixel coordinates of the satellite image. , , , , , and , , , , , The parameters are polynomial transformation parameters, obtained by solving for ground control points in satellite imagery.

[0055] Step S52: Substitute the precise pixel position of the UAV non-orthophoto image center in the satellite orthophoto obtained in step S4 into the above conversion model to obtain the precise geographic coordinates corresponding to the UAV image center.

[0056] Step S53: Combine the attitude parameters of the UAV camera to correct the perspective error of the geographic coordinates and obtain the final high-precision positioning result.

[0057] Example 2 This invention also proposes a cross-view registration and positioning system for UAV aerial images based on instance matching, comprising: The instance segmentation module is used to segment UAV non-orthophoto images and satellite orthophoto images, and extract the mask and contour of building instances.

[0058] The coarse registration module is used to calculate the geometric center of the instance mask, complete the coarse registration, and determine the instance correspondence.

[0059] The top center extraction module is used to extract the top geometric center of drone nonorthophoto image instances through a point matching network with rotation invariance.

[0060] The precise registration module is used to invert the image plane coordinates of building foundations, solve the precise homography matrix, and obtain the precise location of the UAV image center in the satellite image.

[0061] The geographic coordinate calculation module is used to calculate the precise geographic coordinates of UAV images by combining satellite geocoding information.

[0062] Therefore, this invention adopts the above-mentioned instance-matching-based UAV aerial image cross-view registration and positioning method, using rigid building instances as matching primitives and combining rotation-invariant point matching networks to effectively overcome perspective distortion caused by non-orthogonal UAV viewpoints and solve the matching problem caused by the dimensional difference between tilted images and orthogonal satellite images. It has good adaptability to tilted shooting scenes of 10°-45°. Through a two-stage matching process of "coarse registration and correspondence - fine extraction of top center", the matching computation is reduced and the matching error caused by the inconsistency between the overall center and the top center of the instance is completely eliminated. The accuracy of matching point pairs is improved by more than 60% compared with traditional global feature methods. By using perspective projection geometric inversion to find the building foundation position and solving the homography matrix based on ground plane point pairs, the projection error caused by building elevation is completely eliminated, and the accuracy of homography matrix solution is greatly improved. The final positioning accuracy in urban scenes can reach sub-meter level. It does not rely on high-precision GIS data, lidar point clouds and other auxiliary information. Registration and positioning can be completed only through UAV images and public satellite imagery. Data acquisition is convenient and it can be adapted to various complex scenarios such as emergency mapping, field navigation and urban modeling.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A cross-view registration and localization method for UAV aerial images based on instance matching, characterized in that, Includes the following steps: Step S1: Instance segmentation to extract target entities of buildings: Using rigid targets of buildings as matching primitives, instance segmentation models are used to segment the input UAV non-orthophoto image and the satellite orthophoto image to be matched, respectively, to extract the mask and contour of rigid target instances of buildings. Step S2: Coarse registration of geometric centers to determine the correspondence between instances: Calculate the geometric centers of all building instances in the UAV non-orthophoto image and the satellite orthophoto image respectively. Based on the initial homography matrix solution and mismatch elimination algorithm, complete the coarse registration to determine the one-to-one correspondence between building instances in the UAV non-orthophoto image and the satellite orthophoto image. Step S3: Point matching network refines and extracts the top geometric center of the instance: For each pair of successfully matched building instances, a point matching network with rotation invariance is used to extract the top area of ​​the building that perfectly matches the top plane contour of the satellite orthophoto from the UAV non-orthophoto image instance, and calculate the top geometric center. Step S4, Foundation Location Inversion and Accurate Homography Matrix Solution: Based on the top geometric center of the instance mask of the UAV non-orthophoto image obtained in step S3 and the geometric center of the instance mask of the UAV non-orthophoto image obtained in step S2, combined with the perspective projection geometry, the projection coordinates of the building foundation on the UAV image plane are obtained by inversion. Using the ground-based coordinates of the UAV image plane of all buildings and the geometric center coordinates of the corresponding satellite orthorectified image instance mask as matching point pairs, solve for the exact homography matrix between the UAV image plane and the satellite orthorectified ground plane. The pixel position of the geometric center of the UAV non-orthophoto image in the satellite orthophoto image is obtained by homography matrix mapping. The specific process is as follows: Step S41: Define the approximate geometric constraints for the coordinates and midpoint; Based on the top geometric center of the UAV nonorthophoto image instance mask obtained in step S3, the coordinates of the building's top geometric center are: ;in, Mask for instances of non-orthophoto images from drones The top geometric center image plane x-coordinate, Mask for instances of non-orthophoto images from drones The top geometric center image plane ordinate; Based on the instance mask geometric center of the UAV non-orthophoto image from step S2, the coordinates of the building geometric center are: ;in, Mask for instances of non-orthophoto images from drones The geometric center of the image plane has an abscissa. Mask for instances of non-orthophoto images from drones The geometric center of the image plane ordinate; Solve for the projected coordinates of the geometric center of the building foundation ,in Mask for instances of non-orthophoto images from drones The abscissa of the geometric center of the foundation image plane. Mask for instances of non-orthophoto images from drones The solution process for determining the ordinate of the geometric center of the foundation image plane is as follows: For two-dimensional building instance images obtained from non-orthophoto aerial photography by UAVs, in practical engineering applications, the instance mask of the non-orthophoto UAV image can be approximated. Geometric center coordinates , is the coordinate of the top geometric center of this instance. Projected coordinates of the geometric center of the building foundation The midpoint of the geometric center is then the coordinates of the geometric center. The central constraint equations are shown below: ； Expanding into coordinate form, we get: ； ； Step S42: Solve for the projected coordinates of the geometric center of the foundation; Based on the constraint equations of the geometric center, the instance mask of the UAV nonorthophoto image is obtained by algebraic inversion. The geometric center of the foundation is projected onto the image plane in the following coordinates. As shown below: ； ； Step S43: Solving for the exact homography matrix and mapping the geometric center of the UAV non-orthophoto image; The homogeneous vector of the ground geometry center projected onto the image plane from the mask of all matching building UAV nonorthophoto image instances. Homogeneous vector of the geometric center coordinates of the corresponding satellite orthophoto instance To match point pairs, an overdetermined system of equations is established, as shown below: ； in, For exact homography matrix; For the first The image plane abscissa of the ground geometric center projection point of a matching building instance in an unmanned non-orthophoto image; For the first The image plane ordinate of the ground geometric center projection point of a matching building instance in an unmanned non-orthophoto image; For the first The x-coordinate of the geometric center of a matching building instance in a satellite orthophoto; For the first The ordinate of the geometric center of a matching building instance in a satellite orthophoto; Solve the homography matrix using the least squares method or the LM optimization algorithm. The optimization objective is: ； in, To effectively match the number of building instances; The index for matching building instances has a range of values. =1,2,…,M; Based on the homography matrix obtained by the solution Homogeneous coordinates of the geometric center of the UAV nonorthophoto image By mapping to the satellite orthorectified image coordinate system, the precise pixel position of the geometric center of the UAV's non-orthorectified image in the satellite orthorectified image is obtained, as shown below: ； in, The homogeneous coordinates of the geometric center of the unorthophoto image of the UAV; is the x-coordinate of the geometric center of the UAV non-orthophoto image on the image plane; The ordinate of the geometric center of the UAV non-orthophoto image on the image plane; The geometric center of the UAV non-orthophoto image is determined by the exact homography matrix. After mapping, the homogeneous coordinates in the satellite orthophoto coordinate system are the precise pixel positions after mapping. Step S5, Geographic Coordinate Calculation: Combining the geocoding information of the satellite orthophoto, the precise pixel position of the geometric center of the UAV non-orthophoto image obtained in step S4 in the satellite orthophoto is converted into real geographic coordinates, thus completing the high-precision positioning of the UAV non-orthophoto aerial image.

2. The method for cross-view registration and positioning of UAV aerial images based on instance matching according to claim 1, characterized in that, In step S1, the building instance obtained by segmenting the UAV non-orthophoto image includes the complete three-dimensional structural outline of the top and sides; the building instance obtained by segmenting the satellite orthophoto image only includes the top planar outline. The instance segmentation model performs end-to-end instance segmentation on UAV non-orthophoto images and satellite orthophoto images, and outputs the binary mask, bounding rectangle and contour coordinates of each building instance; The instance segmentation models include: Segment Anything Model, Mask R-CNN, and YOLO instance segmentation branch.

3. The method for cross-view registration and positioning of UAV aerial images based on instance matching according to claim 2, characterized in that, In step S2, the geometric center coarse registration determines the instance correspondence, and the specific process is as follows: Step S21: Calculate the geometric center of the binary mask for each building instance in the UAV non-orthophoto image and the satellite orthophoto image respectively; (1) For non-orthophoto images from UAVs, let the total number of instances contained therein be... The binary mask for each instance is denoted as . Through instance mask Calculate the instance mask using the average coordinates of all pixels within the area. geometric center As shown below: ； in, Instance mask in unorthophoto images from UAVs The total number of pixels within; Mask for instances of non-orthophoto images from drones Inner Image plane coordinates of 1 pixel; (2) For satellite orthophotos, let the total number of instances contained be... The binary mask for each instance is denoted as . Through instance mask Calculate the instance mask using the average coordinates of all pixels within the area. geometric center As shown below: ； in, Instance mask in satellite orthophoto The total number of pixels within; For example, a mask for satellite orthophoto instances Inner Image plane coordinates of 1 pixel; Step S22: Set the geometric centers of the instance masks of the UAV non-orthophoto image. Set of geometric centers of instance masks of satellite orthophotos As the initial matching point pairs, the initial homography matrix is ​​solved using either the RANSAC or PROSAC algorithm. ; Step S23: Calculate the reprojection error for each pair of matching points. As shown below: ； in, Indicates the first non-orthophoto in the drone image Geometric center coordinates of each matching instance ; Indicates the first in the satellite orthophoto image Geometric center coordinates of each matching instance Reprojection error The calculation involves determining the geometric center of the unorthophoto image from the UAV. Through the initial homography matrix After transformation to the satellite orthophoto coordinate system, the corresponding geometric center on the satellite orthophoto image. The Euclidean distance between them; Reprojection error Used to remove erroneous matching pairs, remove erroneous matching pairs with reprojection errors greater than a preset threshold, retain instance mask matching pairs with confidence levels higher than the threshold, and determine the one-to-one correspondence between building instances in UAV non-orthophoto images and satellite orthophoto images.

4. The method for cross-view registration and positioning of UAV aerial images based on instance matching according to claim 3, characterized in that, In step S3, the point matching network refines and extracts the top geometric center of the instance, as follows: Step S31: For each pair of building instance masks in the UAV orthophoto and satellite orthophoto, the mask for each building instance in the UAV orthophoto is... The geometric center coordinates are ; In satellite orthophotos and Corresponding building instance mask The geometric center coordinates are Each pair of matching instance masks is denoted as Both correspond to the same real building, and The geometric center is the same as the geometric center of the top plane of the actual building; Step S32: Image registration is performed using a rotation-invariant point matching network to achieve... The top outline and The top contour is accurately aligned; the point matching network integrates wavelet multimodal features and rotation-invariant coding modules, and the point matching registration process is as follows: ， ； in, This is the registration mapping matrix for the output; ( ) represents the elements of the mapping matrix; This indicates the network model selected by the point matching network. Step S33: Based on the registration mapping matrix Masking satellite orthophoto instances geometric center Mapping to instance mask of drone nonorthophoto image The image plane has the following mapping relationship: ； in, For the output drone non-orthophoto image instance mask The coordinates of the top geometric center.

5. The method for cross-view registration and positioning of UAV aerial images based on instance matching according to claim 4, characterized in that, Point matching networks include: SeS-Net, GeoDesc, and HardNet.

6. The method for cross-view registration and positioning of UAV aerial images based on instance matching according to claim 1, characterized in that, In step S5, the geographic coordinates are calculated, and the specific process is as follows: Step S51: Based on the geocoding file of satellite orthophoto, establish a conversion model between satellite orthophoto pixel coordinates and real geographic coordinates; wherein, the conversion model adopts affine transformation, first-order polynomial or second-order polynomial transformation model. The quadratic polynomial transformation model is shown below: ； ； in, Geographic coordinates: latitude and longitude; These are the pixel coordinates of the satellite orthophoto image. , , , , , and , , , , , The parameters for the polynomial transformation are obtained by solving for the ground control points in the satellite orthophoto. Step S52: Substitute the precise pixel position of the UAV non-orthophoto image geometric center obtained in step S4 in the satellite orthophoto image into the transformation model to obtain the geographic coordinates corresponding to the UAV image geometric center. Step S53: Combine the attitude parameters of the UAV camera to correct the perspective error of the geographic coordinates and obtain the final positioning result.

Images