Aerial vision positioning method and device on low-detail 3D map

Abstract

The invention relates to an aerial vision positioning method and device on a low-detail 3D map. The method comprises the following steps: performing building extraction on a query image according to a building segmentation network to obtain a building mask pattern; uniformly sampling in a 3-DoF space with the previous attitude as the center to obtain a sampling attitude; projecting a 3D building surface from a LoD model to a 2D plane for the sampling attitude by using a real-time rendering technology, and generating a synthetic building mask; defining mask alignment cost for the query image according to the building mask and the building mask graph; combining all mask alignment costs in a grid mode in a three-dimensional space to obtain a space cost amount; selecting a rough attitude with a maximum value from the space cost amount by using an argmax operation; and carrying out posture refinement by using a particle filtering method, and carrying out modeling and disturbance on the rough posture to obtain a fine positioning result. By adopting the method, air positioning under the condition of GPS interference can be realized.

Description

Technical Field

[0001] The present application relates to the field of aerial visual positioning technology, and in particular to an aerial visual positioning method and device on a low-detail 3D map. Background Art

[0002] Aerial visual localization is a task that relies on visual input to determine the position and orientation of an unmanned aerial vehicle (UAV) camera relative to a known georeferenced map. This task plays a vital role in the field of computer vision and remote sensing, with applications in navigation, cargo transportation, and surveillance.

[0003] Current aerial visual localization algorithms typically follow the approach of terrestrial visual localization, relying on pre-built high-quality 3D georeferenced data, such as structure from motion (SfM) or mesh models, to establish correspondences between 2D image pixels and 3D spatial points. These algorithms then use techniques such as Perspective-n-Point (PnP) or Branch-and-Bound (BnB) to compute absolute camera poses. However, it is expensive to build and frequently update high-quality 3D maps on a global scale through oblique photogrammetry. In addition, high-resolution 3D maps show a large amount of details and require large storage capacity, which raises concerns about privacy leakage and potential risks of edge device deployment. To address these issues, LoD-Loc proposes a new approach for aerial visual localization using a compact 3D representation (LoD 3D map). Unlike other methods, LoD-Loc enables drones to determine their 6-DoF pose without detailed 3D maps. LoD-Loc aligns the projected wireframe from the LoD 3D map with the predicted neural lines extracted from the query image to estimate the pose. Given an initial pose from the sensor, LoD-Loc builds a cost volume by uniformly sampling pose hypotheses and evaluating the alignment between the projected wireframe and the neural wireframe. Finally, it optimizes the estimated pose by minimizing the reprojection error using a Gauss-Newton iterative descent method. By using the LoD 3D map as a reference, LoD-Loc addresses the challenges of edge deployment, privacy preservation, and 6-DoF pose estimation through an end-to-end wire alignment approach.

[0004] However, LoD-Loc relies on high LoD 3D maps (e.g., LoD3 or LoD2), which have limited coverage around the world. In contrast, most available global LoD maps are low LoD (e.g., LoD1) and can be found in open source databases. LoD1 data is widely available in most regions. LoD3 or LoD2 data is only available for limited areas, and most of these datasets are not open source. In addition, LoD-Loc cannot address the challenges of aerial positioning in the presence of GPS interference. Summary of the invention

[0005] Based on this, it is necessary to provide an aerial visual positioning method and device on a low-detail 3D map that can achieve aerial positioning under GPS interference in order to solve the above technical problems.

[0006] An aerial visual positioning method on a low-detail 3D map, the method comprising: Obtain drone-photographed images; establish a query image of the drone-photographed images and a previous pose of the query image on a low-detail 3D map; extract buildings from the query image according to a building segmentation network to obtain a building mask map; Uniformly sample the 3-DoF space centered on the previous pose to obtain the sampled pose; project the 3D building surface from the LoD model to the 2D plane using real-time rendering technology to generate a synthetic building mask for the sampled pose; define the mask alignment cost for the query image based on the building mask and the building mask map; combine all the mask alignment costs in a grid manner in 3D space to obtain a spatial cost volume; use the argmax operation to select the coarse pose with the maximum value from the spatial cost volume; The particle filter method is used for posture refinement, and the rough posture is modeled and perturbed to obtain a fine positioning result.

[0007] An aerial visual positioning device on a low-detail 3D map, the device comprising: The building segmentation module is used to obtain the drone image; establish the query image of the drone image and the previous pose of the query image on the low-detail 3D map; extract the building from the query image according to the building segmentation network to obtain the building mask; A pose selection module is used to uniformly sample in the 3-DoF space centered on the previous pose to obtain a sampled pose; the 3D building surface is projected from the LoD model to a 2D plane using real-time rendering technology to generate a synthetic building mask for the sampled pose; a mask alignment cost is defined for the query image based on the building mask and the building mask map; all mask alignment costs are combined in a grid manner in 3D space to obtain a spatial cost volume; and the coarse pose with the maximum value is selected from the spatial cost volume using the argmax operation; The posture refinement module is used to refine the posture using the particle filtering method, model and perturb the rough posture, and obtain a fine positioning result.

[0008] The above-mentioned aerial visual positioning method and device on the low-detail 3D map, this application uses the low-detail (LoD1) 3D map and previous posture as the basis. When GPS fails, the previous posture provides an initial reference range, and the widely available LoD1 map provides a relatively stable geographic information framework for positioning, getting rid of the dependence on high-precision GPS signals. The building mask map in the query image is then extracted through the building segmentation network, and the building in the 3D map is projected based on the sampled posture to generate a synthetic building mask. As a relatively stable environmental feature, the comparison of the building's mask can effectively associate the real building information with the building information in the map, reduce the interference of unstable factors in the environment, and thus determine the approximate position and posture of the drone.

[0009] By defining the mask alignment cost and the construction space cost, an effective indicator for measuring the degree of posture matching in 3D space is formed. When there is no GPS signal, the possibility of different postures can be evaluated according to the degree of matching of building features, and the better posture options can be selected from many postures. The rough posture is selected by argmax operation, and then refined by particle filtering method. Particle filtering can handle state estimation problems in complex environments. During the positioning process, the posture estimation is continuously optimized according to the dynamic changes of the environment when the drone is flying, and finally an accurate positioning result is obtained, thereby achieving stable and reliable aerial visual positioning in complex environments with GPS interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A schematic diagram of a flow chart of an aerial visual positioning method on a low-detail 3D map in one embodiment; Figure 2 A block diagram of an aerial visual positioning method on a low-detail 3D map in one embodiment; Figure 3 A structural block diagram of an aerial visual positioning device on a low-detail 3D map in one embodiment. DETAILED DESCRIPTION

[0011] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0012] In one embodiment, Figure 1 As shown, an aerial visual positioning method on a low-detail 3D map is provided, comprising the following steps: Step 102, obtaining an image taken by a drone; establishing a query image of the image taken by the drone and a previous pose of the query image on a low-detail 3D map; extracting buildings from the query image according to a building segmentation network to obtain a building mask image.

[0013] In the case of GPS interference, traditional positioning methods that rely on high-precision GPS signals fail. LoD-Locv2 uses low-detail (LoD1) 3D maps, which are widely available and privacy-friendly. First, a query image of the drone image and its previous pose are established, which provides an initial reference frame for subsequent positioning. Because even if there is no GPS signal, the previous pose information (which may be obtained through previous positioning methods or mission initialization settings) can be used as a starting point to let the system know the approximate position and direction range.

[0014] In order to make full use of the building mask characteristics of the LoD model, this application develops a U-shaped network based on SAM2 for building segmentation to extract building masks from images taken by drones. The building segmentation network consists of a SAM2 encoder, a U-Net decoder, an adapter for efficient fine-tuning, and a receptive field block (RFB) block. Specifically, given an input query image , the encoder block outputs four levels of features ,in ,in For each , the RFB block reduces the number of channels to 64, and we get for , which enhances these lightweight features. Subsequently, the decoder block in the U-shaped structure performs sequential upsampling and concatenates each feature layer Then, the output features of each decoder block are obtained by Convolution segmentation head, produces corresponding segmentation results for These results are finally used to construct the mask from the ground truth To supervise. Loss function By binary cross entropy Loss and Weighted Union Intersection The loss components [25,80] are described as follows: (1) During the training phase, the SAM2 encoder is frozen to reduce the number of trainable parameters, thereby reducing computational and storage costs.

[0015] Buildings are extracted from the query image according to the building segmentation network to obtain a building mask. Buildings are relatively stable features in urban environments. By extracting building information, the interference of other unstable factors in the environment (such as vegetation, moving objects, etc.) can be reduced. At the same time, the sampling posture is obtained by uniformly sampling in the 3-DoF space centered on the previous posture, and then the 3D building surface is projected from the LoD model to the 2D plane using real-time rendering technology to generate a synthetic building mask. In this way, the building-related masks can be used to associate and compare the building features in the real world with the building features in the 3D map.

[0016] Step 104, uniformly sampling in the 3-DoF space centered on the previous pose to obtain a sampled pose; using real-time rendering technology to project the 3D building surface from the LoD model to a 2D plane to generate a synthetic building mask for the sampled pose; defining a mask alignment cost for the query image based on the building mask and the building mask map; combining all mask alignment costs in a grid manner in three-dimensional space to obtain a spatial cost volume; and using an argmax operation to select a coarse pose with a maximum value from the spatial cost volume.

[0017] Extract building mask After that, evenly in the previous sensor posture Surrounding Sampling Pose Assumption To build the space cost volume . Then choose the pose with the highest probability in the cost volume Proceed to the next refinement stage. Pose selection can be viewed as a coarse localization step that determines the coarse spatial translation of the query image, including , and .

[0018] The previous sensor posture It is decomposed into six angles, called ,in represents a translation in 3D space, and Represents the Euler angles (i.e. yaw, pitch, roll). Since the error in the direction of gravity of the device sensor is negligible and the coarse positioning phase focuses on estimating the 3D position of the camera, the rotation axis is fixed. and only focus on spatial translation To construct a spatial cost volume for pose selection .

[0019] Specifically, this application is based on the previous attitude Uniform sampling in the 3-DoF space centered at , the sampling count is ,in . Sampling posture along Direction axis generation: (2) in represents one of the axes, is the index of each offset relative to the starting position, Indicates stride length.

[0020] For each hypothetical pose , using real-time rendering technology OpenSceneGraph ( )[3] Project the 3D building surface from the LoD model onto a 2D plane to generate a synthetic building mask ,in . This application uses a query image for each hypothesized pose Define the mask alignment cost as follows: (3) in, Represents the building segmentation network from The binary mask extracted from (valued as 0 or 1, the same below), Indicates Composite image The binary mask of Indicates the query and The intersection area of ​​the composite mask, Is the query related to The union region of the composite masks, Indicates the number of pixels in the set. Combine all the assumed costs in a grid manner to obtain the spatial cost , whose dimensions are Finally, use Operate to select the coarse pose with the maximum value . Define the mask alignment cost for the query image based on the building mask and the building mask map, and then combine all the mask alignment costs in a grid manner in three-dimensional space to obtain a spatial cost volume. This spatial cost volume is equivalent to an indicator of the degree of pose matching in 3D space. In this way, the degree of matching of building features can be used to evaluate the possibility of different poses in the absence of GPS signals. Use the argmax operation to select the coarse pose with the maximum value from the spatial cost volume, and select the best matching coarse result from many possible poses.

[0021] Step 106, using a particle filtering method to refine the posture, modeling and perturbing the rough posture, and obtaining a fine positioning result.

[0022] Particle filtering is used to refine the pose and model and perturb the rough pose. Particle filtering can effectively handle the state estimation problem of nonlinear and non-Gaussian systems. In complex positioning environments such as GPS interference, it can continuously adjust and optimize the pose estimation based on new observation information (such as building features in subsequent images).

[0023] In the above-mentioned aerial visual positioning method on low-detail 3D maps, this application uses low-detail (LoD1) 3D maps and previous postures as the basis. When GPS fails, the previous posture provides an initial reference range, and the widely available LoD1 map provides a relatively stable geographic information framework for positioning, getting rid of the dependence on high-precision GPS signals. The building mask map in the query image is then extracted through the building segmentation network, and the building in the 3D map is projected based on the sampled posture to generate a synthetic building mask. As a relatively stable environmental feature, the comparison of the building's mask can effectively associate the real building information with the building information in the map, reduce the interference of unstable factors in the environment, and thus determine the approximate position and posture of the drone.

[0024] By defining the mask alignment cost and the construction space cost, an effective indicator for measuring the degree of posture matching in 3D space is formed. When there is no GPS signal, the possibility of different postures can be evaluated according to the degree of matching of building features, and the better posture options can be selected from many postures. The rough posture is selected by argmax operation, and then refined by particle filtering method. Particle filtering can handle state estimation problems in complex environments. During the positioning process, the posture estimation is continuously optimized according to the dynamic changes of the environment when the drone is flying, and finally an accurate positioning result is obtained, thereby achieving stable and reliable aerial visual positioning in complex environments with GPS interference.

[0025] In one embodiment, a building segmentation network includes a SAM2 encoder, a U-Net decoder, and a receptive field block; extracting buildings from a query image according to the building segmentation network includes: Input the query image into the SAM2 encoder to obtain four levels of features ,in ,in ; The four-level features The input receiving field block reduces the number of channels to 64 and obtains the feature layer ; Use U-Net decoder to perform sequential upsampling and concatenate each feature layer The output features after decoding are obtained by Convolution segmentation head, produces corresponding segmentation results ; The loss function of the building segmentation network is set using the segmentation results and the mask constructed from the ground truth, and the buildings of the query image are output using the trained building segmentation network.

[0026] In one embodiment, setting a loss function of a building segmentation network using the segmentation result and a mask constructed from ground truth includes: The loss function of the building segmentation network is set using the segmentation results and the mask constructed from the ground truth as

[0027] in, represents the segmentation result, represents the mask constructed from the ground truth, represents the binary cross entropy, represents weighted union intersection.

[0028] In one embodiment, uniform sampling is performed in a 3-DoF space centered on a previous pose to obtain a sampled pose, including: Uniform sampling in the 3-DoF space centered on the previous posture, the sampled posture is

[0029] in, represents one of the axes, is the index of each offset relative to the starting position, represents the stride, the sampling range is uniformly sampled in the 3-DoF space centered on the prior posture, the sampling range is r(d), and the sampling count is n(d).

[0030] In one embodiment, defining a mask alignment cost for a query image based on a building mask and a building mask map includes: The mask alignment cost is defined for the query image based on the building mask and the building mask map as

[0031] in, Represents the building segmentation network from the query image The building mask extracted from the image has a value of 0 or 1. Indicates open The binary mask of Indicates the query and The intersection area of ​​the composite mask, Is the query related to The union region of the composite masks, Indicates the number of pixels in the set.

[0032] In one embodiment, all mask alignment costs are combined in a grid manner in three-dimensional space to obtain a spatial cost volume, including: The spatial cost is obtained by combining all mask alignment costs in a grid manner in three-dimensional space: , whose dimensions are

[0033] in, represents the query image, Represents the building mask.

[0034] In one embodiment, a particle filter method is used to refine the posture, model and perturb the rough posture, and obtain a fine positioning result, including: From a rough gesture Start, where represents the camera center, is a quaternion that provides a stable rotation frame, applying the perturbation To generate a new set of candidate poses ; Use the cost function to evaluate and gradually optimize the candidate poses to obtain a refined positioning result.

[0035] In one embodiment, a cost function is used to evaluate and gradually optimize candidate postures to obtain a refined positioning result, including: The cost function is used to evaluate and gradually optimize the candidate postures to obtain a fine positioning result:

[0036] in, To render candidate objects, represents the candidate pose, and the loss function is the IoU loss between the query and the rendered candidate object, that is, , A manifold representing candidate poses.

[0037] In a specific embodiment, in order to further refine the rough gesture In the refinement phase, a particle filter method is used to estimate the state of the system based on observations and dynamic adjustments. In this task, the camera pose is selected as the state variable. From the initial state Start, where represents the camera center, is a quaternion that provides a stable rotation frame

[40] , and the perturbation applied To generate a new set of candidate poses These candidate poses are located at The combined rotation and translation space is represented on the manifold of . Then the cost function (such as Eq. (3)) is used to evaluate and gradually optimize these candidate poses. The problem is transformed into the following optimization method: (4) in To render candidate objects, the loss function is the IoU loss between the query and the rendered candidate objects, i.e. In this case, the particle filter will roughly The posterior distribution of , where the particle set It consists of multiple particles and their corresponding weights Composition. The weight of each particle represents its sampling probability, estimated based on formula (3). exist manifold, so they can be perturbed using Lie algebras [17,18,38]. Note that since the errors on the pitch and roll axes are small, this application keeps the pitch and roll unchanged. , And apply a rotation perturbation around the yaw axis in the subspace, and Space center Axis applies a translational perturbation.

[0038] This application conducts extensive experiments on the UAVD4L-LoDv2 dataset as well as the UAVD4L-LoD and Swiss-EPFL datasets to verify the effectiveness of the proposed model. In addition, an ablation study is performed on the UAVD4L-LoDv2 dataset to further analyze the effectiveness of the module. The released UAVD4L-LoDv2 dataset covers an area of ​​2.5 square kilometers with a precisely georeferenced LoD1 map automatically generated from the mesh model of UAVD4L

[83] . The dataset contains 3796 query images captured by different drones, as well as prior information and pose annotations provided by the UAVD4L-LoD dataset. The experiments also used the UAVD4L-LoD and Swiss-EPFL datasets, which contain semi-automatically generated LoD3 and LoD2 maps, respectively, as well as query images and pose labels captured by drones. The proposed method is compared with four visual localization baselines: UAVD4L, which performs localization on a mesh model; CadLoc, MC-Loc, and LoD-Loc, which operate on the LoD model. Both UAVD4L and CadLoc adopt keypoint-based strategies: 1) SIFT

[45] descriptor and traditional nearest neighbor (NN) matching, 2) learning-based extractors SuperPoint (SPP) and Superglue (SPG), 3) detector-free matcher LoFTR and its variants 4) e-LoFTR, and 5) dense feature matcher RoMa. In addition, MC-Loc uses two backbones as its feature extraction encoder: 6) pre-trained DINOv2

[53] encoder, and 7) RoMa dense feature extractor. 8) LoD-Loc is implemented with its default parameters. This application follows the standard localization evaluation metrics and sets the threshold to , and , the same as LoD-Loc

[94] .

[0039] During the training phase, the training dataset for the segmentation module comes from a subset of the synthetic database in UAVD4L, which has been filtered to exclude synthetic images without buildings. To further enrich the training data, additional synthetic data is generated based on the rendering method in UAVD4L, with the pitch angles set to 30 and 60 degrees. Ground truth mask labels for the training data It is constructed by projecting synthetic building masks from the UAVD4L-LoD model using matching poses, aligned to the same viewpoint to ensure accuracy and consistency. A total of For training data. In addition, the segmentation module accepts To optimize the training process, the learning rate is set to , and use the AdamW optimizer with weight decay of To prevent overfitting. The number of training cycles is set to , to ensure that the data features are fully learned. In this process, the pre-trained model of the SAM2 encoder is configured as During the inference phase, the input image size of the segmentation module is kept as , but the extracted building mask images are resized according to different processing stages. In the pose selection stage, the size of the query mask and the candidate rendering mask is set to , to capture approximate location information. The sampling interval is set to 10 meters, where In the subsequent particle filter optimization stage, the mask size is doubled to , to capture finer building mask details. This application sets the number of iterations to , using 2 beams, the initial angular perturbation is 2 degrees. The translation perturbation follows a Gaussian distribution , whose average value is 0, . Apply the attenuation factor , each iteration generates The rendering speed of LoD building mask is 602 448 resolution up to , significantly improving the inference efficiency. The entire training and testing process was performed on 2 NVIDIA RTX 4090 GPUs. To verify the effectiveness of the module, this application tried three model variants: (1) -no refine, that is, excluding the particle filter optimization stage; (2) -no select, that is, skipping pose selection and directly entering the particle filter optimization; (3) The full model represents the LoD-Loc v2 model proposed in this application. Visualization of segmentation results and overlay of different iterations. The visualized segmentation results demonstrate the excellent performance of the model in extracting building masks. In addition, the coverage of each iteration represents the pattern formed by projecting the building mask of the LoD model onto the query image based on the predicted pose of the iteration. Better alignment of building masks indicates more accurate pose prediction. Evaluation under GPS constraints. This application initialized different prior error ranges on the UAVD4L-LoDv2 dataset to simulate GPS constraints. Similar to LoD-Loc

[94] , this application sets the Z-axis error range to The X and Y axis error ranges are set to m, of which Corresponding to the normal GPS environment. As shown in Table 1, although the accuracy of LoD-Loc v2 decreases slightly with the increase of GPS error, it is still much better than LoD-Loc. This proves the effectiveness of the LoD-Loc v2 algorithm and shows that it can reliably solve the 6-DoF positioning problem of UAVs even under GPS-restricted conditions.

[0040] Table 1

[0041] Evaluation based on high-LoD dataset. This application conducted experiments on datasets with different levels of detail, including LoD3 (UAVD4L-LoD) and LoD2 (Swiss-EPFL), to verify the versatility of the algorithm of this application. Table 2 shows that LoD-Loc v2 achieves higher positioning accuracy than LoD-Loc on high-detail and medium-detail datasets. The performance of the LoD2 dataset is lower than that of the LoD3 dataset because a large part of the buildings in the query image occupy the entire screen. The method of this application has limitations in handling such situations. For more details, please refer to the Appendix. These results show that LoD-Loc v2 exhibits higher robustness and accuracy on datasets with different LoD levels, and can achieve accurate positioning even in challenging scenes, thus confirming its advantages in multi-modal UAV positioning.

[0042] Table 2

[0043] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0044] In one embodiment, Figure 3 As shown, an aerial visual positioning device on a low-detail 3D map is provided, comprising: a building segmentation module 302, a posture selection module 304 and a posture refinement module 306, wherein: The building segmentation module 302 is used to obtain the drone-photographed image; establish a query image of the drone-photographed image and a previous pose of the query image on the low-detail 3D map; extract buildings from the query image according to the building segmentation network to obtain a building mask image; The pose selection module 304 is used to uniformly sample in the 3-DoF space centered on the previous pose to obtain a sampled pose; project the 3D building surface from the LoD model to the 2D plane using real-time rendering technology for the sampled pose to generate a synthetic building mask; define a mask alignment cost for the query image according to the building mask and the building mask map; combine all mask alignment costs in a grid manner in the three-dimensional space to obtain a spatial cost volume; select a coarse pose with a maximum value from the spatial cost volume using an argmax operation; The posture refinement module 306 is used to refine the posture using a particle filtering method, model and perturb the rough posture, and obtain a fine positioning result.

[0045] For the specific definition of an aerial visual positioning device on a low-detail 3D map, please refer to the definition of an aerial visual positioning method on a low-detail 3D map mentioned above, which will not be repeated here. Each module in the above-mentioned aerial visual positioning device on a low-detail 3D map can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0046] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0047] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for aerial visual positioning on a low-detail 3D map, characterized in that: The method comprises: Acquire an image taken by a drone; create a query image of the image taken by the drone and a previous pose of the query image on a low-detail 3D map; extract buildings from the query image according to a building segmentation network to obtain a building mask image; uniformly sampling in a 3-DoF space centered on the previous pose to obtain a sampled pose; projecting a 3D building surface from a LoD model onto a 2D plane using real-time rendering technology for the sampled pose to generate a synthetic building mask; defining a mask alignment cost for a query image based on the building mask and the building mask map; combining all mask alignment costs in a grid manner in a three-dimensional space to obtain a spatial cost volume; and selecting a coarse pose with a maximum value from the spatial cost volume using an argmax operation; The particle filter method is used for posture refinement, and the rough posture is modeled and perturbed to obtain a fine positioning result.

2. The method according to claim 1, characterized in that The building segmentation network includes a SAM2 encoder, a U-Net decoder, and a receiving field block; Extracting buildings from the query image according to a building segmentation network includes: The query image is input into the SAM2 encoder to obtain four-level features ,in ,in ; The four-level features The input receiving field block reduces the number of channels to 64 and obtains the feature layer ; Use U-Net decoder to perform sequential upsampling and concatenate each feature layer The output features after decoding are obtained by Convolution segmentation head, produces corresponding segmentation results ; The loss function of the building segmentation network is set using the segmentation results and the mask constructed by the ground truth, and the buildings of the query image are output using the trained building segmentation network.

3. The method according to claim 2, characterized in that The loss function of the building segmentation network is set using the segmentation results and the mask constructed from the ground truth, including: The loss function of the building segmentation network is set using the segmentation results and the mask constructed from the ground truth as in, represents the segmentation result, represents the mask constructed from the ground truth, represents the binary cross entropy, represents weighted union intersection.

4. The method according to any one of claims 1 to 3, characterized in that Uniform sampling is performed in the 3-DoF space centered on the previous posture to obtain a sampled posture, including: Uniform sampling in the 3-DoF space centered on the previous posture gives the sampled posture: in, represents one of the axes, is the index of each offset relative to the starting position, Indicates the stride, Indicates the sampling range.

5. The method according to claim 1, characterized in that Defining a mask alignment cost for a query image based on the building mask and the building mask map, comprising: The mask alignment cost is defined for the query image according to the building mask and the building mask map as in, Represents the building segmentation network from the query image The building mask extracted from the image has a value of 0 or 1. Indicates open The binary mask of Indicates the query and The intersection area of ​​the composite mask, Is the query related to The union region of the composite masks, Indicates the number of pixels in the set.

6. The method according to claim 1, characterized in that All mask alignment costs are combined in a grid-like manner in 3D space to obtain a spatial cost volume, including: The spatial cost is obtained by combining all mask alignment costs in a grid manner in three-dimensional space: , whose dimensions are in, represents the query image, Represents the building mask.

7. The method according to claim 1, characterized in that The particle filter method is used to refine the posture, model and perturb the rough posture, and obtain fine positioning results, including: From a rough gesture Start, where represents the camera center, is a quaternion that provides a stable rotation frame, applying the perturbation To generate a new set of candidate poses ; Use the cost function to evaluate and gradually optimize the candidate postures to obtain a refined positioning result.

8. The method according to claim 7, characterized in that The candidate postures are evaluated and gradually optimized using a cost function to obtain a refined positioning result, including: The cost function is used to evaluate and gradually optimize the candidate postures to obtain a fine positioning result: in, To render candidate objects, represents the candidate pose, and the loss function is the IoU loss between the query and the rendered candidate object, that is, , A manifold representing candidate poses.

9. An aerial visual positioning device on a low-detail 3D map, characterized in that: The device comprises: A building segmentation module is used to obtain an image taken by a drone; establish a query image of the image taken by the drone and a previous pose of the query image on a low-detail 3D map; extract buildings from the query image according to a building segmentation network to obtain a building mask image; a pose selection module, configured to uniformly sample in a 3-DoF space centered on the previous pose to obtain a sampled pose; project the 3D building surface from the LoD model onto a 2D plane using real-time rendering technology for the sampled pose to generate a synthetic building mask; define a mask alignment cost for a query image according to the building mask and the building mask map; combine all mask alignment costs in a grid manner in a three-dimensional space to obtain a spatial cost volume; and select a coarse pose with a maximum value from the spatial cost volume using an argmax operation; The posture refinement module is used to refine the posture using the particle filtering method, model and perturb the rough posture, and obtain a fine positioning result.

Images