Cross-view visual place recognition method based on single uav image bev generation and repair

Abstract

The application particularly relates to a cross-view visual location recognition method based on BEV generation and repair of a single unmanned aerial vehicle image, and belongs to the fields of computer vision and remote sensing image processing. The application takes a single unmanned aerial vehicle image as input, first acquires a dense depth map by using a monocular depth estimation model, and combines camera parameters to inversely project pixels into a three-dimensional point cloud; then performs a bird's eye view projection on the point cloud to generate an initial BEV image and an effective area mask; then uses a FLUX.2 image editing model fine-tuned by LoRA to guide satellite style alignment, hole repair and missing content filling on the BEV image through a text prompt word; finally, performs feature matching on the repaired BEV image and a satellite image database to output a location recognition result. The application can complete cross-view location recognition only by using a single unmanned aerial vehicle image, effectively reduces the view angle difference between the unmanned aerial vehicle image and the satellite image, repairs the projection hole, and improves the accuracy and robustness of the cross-view visual location recognition.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and remote sensing image processing, and specifically to a cross-view visual location recognition method based on BEV generation and restoration of a single UAV image. Background Technology

[0002] Visual location recognition refers to retrieving reference images from a pre-established image database based on an input image to achieve tasks such as localization, navigation, or scene matching. In UAV positioning and remote sensing applications, a common requirement is to match ground images captured by the UAV with satellite or map images to determine the UAV's current location or to assist the navigation system in repositioning.

[0003] However, there are often significant differences in perspective between single UAV images and satellite images. UAV images may present a tilted view, a low-altitude view, or a partial overhead view, while satellite images are typically presented as an approximately orthogonal overhead view. These differences in scale, perspective, lighting, imaging height, ground object occlusion, texture style, and spatial coverage lead to problems such as low recognition accuracy and insufficient robustness when directly performing image feature matching.

[0004] Existing visual location recognition methods mainly fall into the following categories: 1. Direct Feature Matching Method This type of method directly extracts global or local features from UAV and satellite images, and then calculates similarity to complete the retrieval. While simple to implement, this method suffers from significant differences in perspective between UAV and satellite images, resulting in incomplete correspondence of image content in planar structure. This inconsistency in feature distribution can negatively impact location identification accuracy.

[0005] 2. Cross-view image conversion method These methods typically utilize deep networks to convert ground-view or UAV-view images into approximate top-down views, which are then matched with satellite images. While these methods can alleviate the perspective difference problem to some extent, when only a single image is input, the conversion results are easily affected by factors such as inaccurate depth, invisible occluded areas, projection holes, and unstable generated content.

[0006] 3. A viewpoint alignment method based on 3D reconstruction This type of method constructs a 3D scene from multi-view images or sensor data, and then projects it onto a top-down plane to form an expression that is more consistent with satellite imagery. However, this method usually relies on multiple images, pose information, or LiDAR data, resulting in high data acquisition costs and making it difficult to apply to application scenarios with only a single UAV image.

[0007] 4. Image style transfer method based on generative model This type of method can convert input images into the style of target images, such as converting drone images into satellite image styles. However, without geometric constraints, the generative model may alter the spatial structure of key features such as roads, buildings, water bodies, and vegetation, leading to incorrect matching in visual location recognition.

[0008] Therefore, existing technologies still have the following shortcomings: there are significant viewpoint differences between single UAV images and satellite images, making direct matching difficult; the 3D projection results obtained by monocular depth estimation may have occlusion areas, projection holes, and local geometric distortions; there are still differences in texture style, lighting, resolution, and ground feature representation between BEV images and real satellite images; if general image generation models are not adapted to the task, they are prone to generating illusory content that does not meet the requirements of location recognition; existing methods cannot simultaneously take into account single image input, geometric consistency, image restoration capabilities, and location recognition accuracy.

[0009] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0010] To overcome the problems existing in visual location recognition using UAV images and satellite images, such as large differences in perspective, insufficient geometric information in a single image, holes in BEV projection images, and easy alteration of key ground features by generative transformation, this invention provides a cross-perspective visual location recognition method based on BEV generation and restoration of a single UAV image.

[0011] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0012] According to a first aspect of the present invention, a cross-view visual location recognition method based on BEV generation and restoration of a single UAV image is provided, the method comprising: Acquire a single drone image and perform preprocessing; Depth estimation is performed on the preprocessed UAV images based on a monocular depth estimation model to obtain a depth map; Based on the depth map, combined with camera parameters and UAV attitude information, the image pixels are back-projected into a three-dimensional point cloud; The three-dimensional point cloud is projected onto a horizontal top-view plane to generate an initial BEV image and a corresponding effective region mask. Using an image editing model finely tuned through an image-to-image approach, the initial BEV image is subjected to text-guided image editing and repair to obtain a repaired BEV image. The repaired BEV image is matched and retrieved with a satellite image database to output visual location identification results.

[0013] In some exemplary embodiments, depth estimation is performed on the preprocessed UAV image based on a monocular depth estimation model to obtain a depth map, specifically as follows: Preprocessed drone images Input a monocular depth estimation model to obtain a dense depth map corresponding to the size of the input image. :

[0014] in, This represents the depth estimation model; Indicates the parameters of the depth estimation model; This represents the predicted depth map; When the depth estimation model outputs a relative depth, the depth map is scale-corrected using the UAV altitude, camera calibration parameters, ground reference points, or scene scale priors to obtain a scale-corrected depth map.

[0015] in, This represents the depth map after scale correction. Represents the scaling factor. This indicates the depth bias term.

[0016] In some exemplary embodiments, based on the depth map and combined with camera parameters and UAV attitude information, the image pixels are back-projected into a three-dimensional point cloud, specifically as follows: For any pixel in the input UAV image, its homogeneous pixel coordinates are denoted as

[0017] in, and These represent the x and y coordinates of a pixel in the image, respectively; Based on the camera pinhole imaging model, the pixels Combined with its depth value By back-projecting onto the camera coordinate system, we obtain three-dimensional points:

[0018] in, Represents pixels The corresponding 3D point coordinates in the camera coordinate system Represents the camera intrinsic parameter matrix; Then, based on the camera's extrinsic parameters, transform the 3D points in the camera coordinate system to the ground coordinate system or the world coordinate system:

[0019] in, Represents a three-dimensional point in the world coordinate system. This represents the rotation matrix from the camera coordinate system to the world coordinate system. This represents the translation vector from the camera coordinate system to the world coordinate system; Perform the above back projection operation on all valid pixels to obtain a 3D point cloud with color information:

[0020] in, This represents a set of 3D point clouds reconstructed from a single UAV image. This represents the RGB color corresponding to a pixel. This represents the set of valid pixels in the image.

[0021] In some exemplary embodiments, it also includes a three-dimensional point cloud. Filtering to obtain Filtering methods include: Remove low-confidence points based on depth confidence; Remove points that are too close or too far based on the depth range; Remove points that are clearly outside the ground projection range based on the height threshold; Remove interfering regions based on semantic segmentation results; Outliers are removed using statistical filtering.

[0022] In some exemplary embodiments, the 3D point cloud is projected onto a horizontal top-view plane to generate an initial BEV image and a corresponding effective region mask, specifically as follows: Effective point cloud Project the image onto a horizontal ground plane to generate a BEV image; let the point in the world coordinate system be:

[0023] in, and Represents coordinates on a horizontal ground plane. Represents the coordinates in the height direction; Let the spatial resolution of the BEV image be... , representing the actual ground distance corresponding to one pixel in the BEV image; let the coverage area of ​​the BEV image be:

[0024] Then three-dimensional points Raster coordinates in BEV images The calculation is as follows:

[0025]

[0026] in, This indicates a round-down operation; The color of each point in the point cloud Alternatively, intermediate visual features can be projected onto the corresponding BEV grid to obtain the initial BEV image:

[0027] in, This represents the projection operation from the point cloud onto the BEV plane. Represents the initial BEV image; When multiple 3D points are projected onto the same BEV pixel location, the final color of that pixel is determined using a weighted average, nearest height priority, maximum confidence priority, or distance-weighted fusion method; it is represented as:

[0028] in, Represents aggregate functions, This represents the confidence level of a pixel or depth estimate. Represents pixels The corresponding 3D points are projected onto the BEV image coordinates. ; Generate a mask for the effective BEV region:

[0029] in, This represents the effective region mask of the BEV image.

[0030] In some exemplary embodiments, the initial BEV image is subjected to text-guided image editing and repair to obtain a repaired BEV image, specifically as follows: A training dataset containing UAV images, corresponding satellite images, and geographic labels was constructed, and LoRA fine-tuning was performed on the FLUX.2 basic image editing model. During training, the parameters of the FLUX.2 basic model were frozen, and only the LoRA low-rank adaptation parameters were updated. The total loss function, which is a weighted sum of style loss, geometry preservation loss, and reconstruction loss, was used to constrain the model to learn the satellite image style, hole repair, and ground feature structure preservation capabilities. During the inference phase, the initial BEV image, the effective BEV region mask, and preset text prompts are input into the finely tuned FLUX.2 model, and the output is a repaired BEV image that has completed hole filling, texture completion, satellite style alignment, and retains the original spatial structure of ground features. The text prompts include semantic constraints such as maintaining the spatial layout of ground features, converting the satellite top-down style, repairing projection holes, and prohibiting the generation of contradictory false ground features.

[0031] In some exemplary embodiments, the repaired BEV image is matched and retrieved against a satellite image database to output visual location identification results, specifically: Establish a satellite image database:

[0032] in, Represents the first in the database Zhang Weixing image, This indicates the geographical location corresponding to the satellite image. Indicates the size of the satellite image database; Using a visual location recognition feature extraction network Extract features from the restored BEV image and satellite image respectively:

[0033]

[0034] in, This represents the normalized feature vector of the restored BEV image. Indicates the first Normalized feature vectors of Zhang Weixing's images. This indicates a feature normalization operation.

[0035] Calculate the similarity between the restored BEV image and each satellite image:

[0036] in, Indicates the image of the drone to be identified and the first Similarity between satellite images; The satellite image with the highest similarity was selected as the recognition result:

[0037] The final output is the corresponding location identification result:

[0038] in, This indicates the geographical location corresponding to the identified drone image.

[0039] According to a second aspect of the present invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the cross-view visual location recognition method for BEV generation and restoration based on a single UAV image as described in the first aspect.

[0040] According to a third aspect of the present invention, a computer program product is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the cross-view visual location recognition method for BEV generation and restoration based on a single UAV image as described in the first aspect above.

[0041] According to a fourth aspect of the present invention, an electronic device is provided, comprising: Processor; and Memory for storing the executable instructions of the processor; The processor is configured to implement the cross-view visual location recognition method based on a single UAV image for BEV generation and restoration as described in the first aspect when executing the executable instructions.

[0042] Compared with the prior art, the present invention has the following beneficial effects: 1. Cross-view location identification can be completed with only a single drone image. This invention does not rely on multi-view image sequences or LiDAR point clouds, but performs depth estimation, 3D reconstruction and BEV generation using only a single UAV image, which reduces the difficulty of data acquisition and the cost of system deployment.

[0043] 2. Reduce the viewpoint difference between UAV images and satellite images through depth estimation and 3D projection. This invention utilizes Depth Anything 3 or similar monocular depth estimation models to obtain dense depth information, and then projects the image content onto a top-down plane, transforming the input image from the original UAV perspective into a BEV representation that is closer to satellite imagery, thus alleviating the difficulty of cross-view matching from a geometric perspective.

[0044] 3. Able to repair holes and missing areas in single-image BEV projection. Due to occlusion and limited field of view in single UAV images, BEV images obtained by traditional point cloud projection are prone to holes. This invention utilizes the task-adapted FLUX.2 image editing model to fill in hole areas and locally missing content, thereby improving the integrity of BEV images.

[0045] 4. Achieve style adaptation of target area satellite imagery through LoRA fine-tuning This invention employs an image-to-image approach to fine-tune the FLUX.2 model using LoRA on the training set, enabling the generated results to learn the color, texture, and imaging style of the target satellite image, thereby reducing the appearance differences between BEV images and satellite images.

[0046] 5. Text guidance makes the image editing process controllable. This invention explicitly constrains the model to maintain the spatial structure of ground features, repairs projection holes, and converts it to a satellite top-view style through text prompts, making the generation process more in line with the needs of visual location recognition tasks.

[0047] 6. Improve image matching while maintaining geometric structure. This invention does not directly generate unconstrained images, but performs controlled restoration and style transfer on BEV images obtained by depth projection, which can preserve the layout of key features such as roads, buildings, water bodies, and vegetation as much as possible, and improve the matching accuracy with satellite image databases.

[0048] 7. High degree of modularity, easy to replace and expand. The depth estimation model, image editing model, LoRA fine-tuning method, and location recognition network in this invention can all be replaced with similar models according to actual needs, demonstrating good engineering adaptability and scalability.

[0049] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0051] Figure 1 This is a block diagram of the cross-view visual location recognition method based on BEV generation and restoration of a single UAV image according to the present invention; Figure 2 This is a flowchart of the cross-view visual location recognition method based on single-frame UAV image BEV generation and restoration according to the present invention. Detailed Implementation

[0052] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0053] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0054] To address the shortcomings and deficiencies of existing technologies, this invention provides a cross-view visual location recognition method based on single UAV image generation and restoration of BEV (Browser Emission Vehicle). Using a single UAV image as input, a dense depth map is first obtained through DepthAnything 3 or a monocular depth estimation model with equivalent functionality. Then, combining camera intrinsics, UAV attitude, or an estimated ground coordinate system, the image pixels are back-projected into a 3D point cloud. Subsequently, the 3D point cloud is projected onto a horizontal top-down plane to obtain a BEV image that more closely approximates the perspective of satellite images. Next, the FLUX.2 image editing model is used to perform text-guided image-to-image editing on the BEV image, and LoRA fine-tuning on the training set allows the model to learn the style of the target satellite image and the rules for filling in missing regions. Finally, the restored BEV image is matched and retrieved against a satellite image database to achieve cross-view visual location recognition.

[0055] Specifically, this may include the following steps: Step 1: Acquire a single RGB image of the UAV and complete preprocessing; read or estimate the camera intrinsic parameters, the rotation matrix and translation vector of the UAV camera relative to the ground coordinate system. Step 2: Input the preprocessed UAV image into the monocular depth estimation model to obtain a dense depth map, perform scale correction on the relative depth output to obtain a scale-corrected depth map, and perform filtering and noise reduction processing on the depth map; Step 3: Based on the pinhole camera imaging model, the image pixels are back-projected into the camera coordinate system along with the scale-corrected depth map, and then transformed into the world coordinate system to construct a 3D point cloud with RGB colors. Multi-dimensional filtering is then performed on the point cloud to obtain the effective point cloud. Step 4: Set the ground resolution and ground coverage of the BEV image, project the effective point cloud from above onto the horizontal ground plane to generate an initial BEV image with holes, and simultaneously generate a BEV effective area mask that marks the effective projection area and the missing hole area. Step 5: Construct a training dataset containing UAV images, corresponding satellite images, and geographic labels, and perform LoRA fine-tuning on the FLUX.2 basic image editing model; freeze the parameters of the FLUX.2 basic model during training, and only update the LoRA low-rank adaptation parameters. Use the weighted sum of style loss, geometry preservation loss, and reconstruction loss to form a total loss function to constrain the model to learn the satellite image style, hole repair, and ground feature structure preservation capabilities. Step 6: In the inference stage, the initial BEV image, the effective BEV region mask, and the preset text prompts are input into the fine-tuned FLUX.2 model, and the output is a repaired BEV image that has completed hole filling, texture completion, satellite style alignment, and retains the original spatial structure of ground features; the text prompts include semantic constraints such as maintaining the spatial layout of ground features, converting the satellite top view style, repairing projection holes, and prohibiting the generation of contradictory false ground features; Step 7: Build a satellite image database. Use a visual location recognition feature network to extract normalized feature vectors from the restored BEV image and the satellite images in the database. Calculate the cosine similarity of the feature vectors and select the geographic coordinates corresponding to the satellite image with the highest similarity as the cross-view visual location recognition result.

[0056] This invention first uses a monocular depth estimation model to estimate the depth of UAV images, then uses a camera imaging model to backproject the image pixels into a 3D point cloud, and then projects the point cloud onto a top-view plane to obtain a BEV image; then, using the FLUX.2 image editing model, which has been fine-tuned in the image-to-image manner of the training set, the BEV image is subjected to satellite image style alignment, hole repair, and missing content filling; finally, the repaired BEV image is compared with a satellite image database for visual location identification, thereby improving the accuracy and robustness of cross-view matching.

[0057] The steps in this exemplary embodiment will now be described in more detail with reference to the accompanying drawings and embodiments.

[0058] Step 1: Acquire a single drone image and perform preprocessing. Acquire a single RGB image captured by the drone, denoted as:

[0059] in, This represents the input drone image. and These represent the image height and width, respectively, and 3 represents the number of RGB color channels.

[0060] images of drones Preprocessing is performed, including but not limited to image distortion correction, size normalization, brightness normalization, color space unification, and cropping of invalid regions. If the drone image includes camera parameters, the camera intrinsic parameter matrix is ​​read.

[0061] in, and These represent the camera's focal length in the horizontal and vertical directions, respectively. and This indicates the coordinates of the camera's principal point.

[0062] If the image includes drone attitude information, then read or estimate the rotation matrix of the drone camera relative to the ground coordinate system. Translation vector If precise attitude information is unavailable, approximate extrinsic parameters can be obtained through horizon estimation, ground plane estimation, or empirical camera attitude initialization.

[0063] Step 2: Perform monocular depth estimation based on Depth Anything 3 Preprocessed drone images Input a monocular depth estimation model to obtain a dense depth map corresponding to the size of the input image:

[0064] in, The depth estimation model is preferred, either the Depth Anything 3 model or a neural network model with the same depth estimation function. Indicates the parameters of the depth estimation model; This represents the predicted depth map.

[0065] When the depth estimation model outputs a relative depth, in order to maintain consistency with the actual scale in 3D reconstruction, the depth map can be scaled using UAV altitude, camera calibration parameters, ground reference points, or scene scale priors to obtain a scale-corrected depth map.

[0066] in, This represents the depth map after scale correction. Represents the scaling factor. This represents the depth offset term. When only relative geometric projection is required, it can also be set... Using only the scaling factor Apply uniform scaling to the depth map.

[0067] To improve the stability of subsequent 3D point cloud projection, smoothing filtering, edge-preserving filtering, or confidence filtering can be applied to the depth map to remove obviously abnormal depth values.

[0068] Step 3: Perform single-image 3D reconstruction based on depth map For any pixel in the input UAV image, its homogeneous pixel coordinates are:

[0069] in, and These represent the x and y coordinates of a pixel in the image, respectively.

[0070] Based on the camera pinhole imaging model, the pixels Combined with its scale-corrected depth value By back-projecting onto the camera coordinate system, we obtain three-dimensional points:

[0071] in, Represents pixels The coordinates of the corresponding 3D point in the camera coordinate system.

[0072] Then, based on the camera's extrinsic parameters, transform the 3D points in the camera coordinate system to the ground coordinate system or the world coordinate system:

[0073] in, Represents a three-dimensional point in the world coordinate system. This represents the rotation matrix from the camera coordinate system to the world coordinate system. This represents the translation vector from the camera coordinate system to the world coordinate system.

[0074] Perform the above back projection operation on all valid pixels to obtain a 3D point cloud with color information:

[0075] in, This represents a set of 3D point clouds reconstructed from a single UAV image. This represents the RGB color corresponding to a pixel. This represents the set of valid pixels in the image.

[0076] To reduce the impact of sky, distant noise, abnormal depth, and non-ground regions on BEV generation, point clouds can be processed. Perform filtering. Filtering methods include, but are not limited to: 1. Remove low-confidence points based on depth-of-confidence. 2. Remove points that are too close or too far based on the depth range; 3. Remove points that are clearly outside the ground projection range based on the height threshold; 4. Remove interfering regions such as the sky, drone body, and invalid regions at image edges based on the semantic segmentation results; 5. Remove outliers using statistical filtering.

[0077] The effective point cloud is obtained after filtering:

[0078] Step 4: Generate BEV image through top-down projection of point cloud Effective point cloud Projecting onto a horizontal ground plane generates a BEV image. Let the point in the world coordinate system be:

[0079] in, and Represents coordinates on a horizontal ground plane. Represents the coordinates in the height direction.

[0080] Let the spatial resolution of the BEV image be... , representing the actual ground distance corresponding to one pixel in the BEV image. Let the coverage area of ​​the BEV image be:

[0081] Then three-dimensional points Raster coordinates in BEV images The calculation is as follows:

[0082]

[0083] in, This indicates a round-down operation.

[0084] The color of each point in the point cloud Alternatively, intermediate visual features can be projected onto the corresponding BEV grid to obtain the initial BEV image:

[0085] in, This represents the projection operation from the point cloud onto the BEV plane. This represents the initial BEV image.

[0086] When multiple 3D points are projected onto the same BEV pixel location, the final color of that pixel can be determined using a weighted average, nearest height priority, maximum confidence priority, or distance-weighted fusion method. Its general form can be expressed as:

[0087] in, Represents aggregate functions, This represents the confidence level of a pixel or depth estimate. Represents pixels The corresponding 3D points are projected onto the BEV image coordinates. .

[0088] Due to occlusion, limited field of view, and depth estimation errors in single-frame UAV images, the initial BEV image... The image may contain projection holes, stretched areas, missing textures, or local artifacts. Therefore, this invention simultaneously generates a mask for the effective BEV region:

[0089] in, This represents the effective region mask of the BEV image. This mask is used in subsequent image editing stages to instruct the model to retain valid structure and repair missing areas.

[0090] Step 5: Construct the training set and fine-tune LoRA for FLUX.2. To make BEV images more visually similar to satellite images and to repair holes and missing areas caused by single-image projection, this invention employs a text-guided image editing model for image-to-image editing of BEV images. Preferably, this invention uses FLUX.2 as the base image editing model and performs LoRA fine-tuning on the training set using an image-to-image approach.

[0091] The training set consists of several sets of UAV images and corresponding satellite images of the region, denoted as:

[0092] in, Indicates the first Zhang's drone image, This represents satellite images corresponding to their geographical locations. Indicates geographic location information or image matching tags, This indicates the number of training samples.

[0093] For each drone image in the training set Generate the corresponding initial BEV image according to steps 1 to 4:

[0094] Then construct the text prompt. This is used to guide the model in style alignment and missing region repair. Text prompts may include the following semantic constraints: • Maintain the relative spatial positions of features such as roads, buildings, water bodies, and vegetation; Convert the image to a satellite top-down view style; • Repair the void area in the projection; • Complete missing road edges, building textures, and ground textures; • Avoid generating new features that contradict the original geometry.

[0095] For example, the text prompt word can be set as: "Edit the input BEV drone projection image into a clear satellite top view style, keep the spatial layout of roads, buildings, water bodies, vegetation and open areas unchanged, repair projection holes and local missing areas, fill in natural textures, and do not change the main ground features." During the training phase, the main parameters of the FLUX.2 base model are frozen, and only the LoRA low-rank adaptation parameters are trained. Let the FLUX.2 base model be... Its basic parameters are The LoRA parameter is The image editing output will then be:

[0096] in, This represents the repaired pseudo-satellite style image output by the model. Indicates the effective area mask for BEV. Indicates text prompt words.

[0097] The goal of LoRA fine-tuning is to make the output image visually similar to the satellite image while preserving the main geometric structures in the BEV image. Training objectives can include:

[0098] in, Represents the total loss function. Indicates the loss of style consistency. Indicates geometric preservation loss. This represents the loss of image reconstruction or reference consistency. and This represents the weighting coefficient.

[0099] in: 1. Used to constrain the consistency between generated images and satellite images in terms of color, texture, and overall imaging style; 2. Used to constrain the spatial structure of key features such as roads, buildings, and water bodies from being altered; 3. Used to ensure that the generated image maintains consistency with the corresponding satellite image in terms of local or global features when registered or weakly registered satellite images exist.

[0100] In practical implementation, if pixel-level registration between UAV BEV images and satellite images is difficult, feature-level loss or location recognition loss can be used instead of pixel-level reconstruction loss to avoid model learning instability caused by registration errors. Step 6: Use the finely tuned FLUX.2 to perform text-guided BEV repair and filling. During the inference phase, for a single UAV image to be identified, an initial BEV image is first generated according to steps 1 to 4. and effective area mask .

[0101] Then , and text prompts Inputting the LoRA-tuned FLUX.2 image editing model yields the repaired BEV image:

[0102] in, This represents a BEV image after text-guided editing, style alignment, and missing content repair. This represents the LoRA parameters after training is complete.

[0103] Compared with the initial BEV image In comparison, the restored BEV image It has the following characteristics: 1. The image perspective is closer to the overhead view of satellite images; 2. The image's color, texture, and overall style are closer to satellite imagery; 3. Holes caused by single-image projection are filled in; 4. Missing features were appropriately filled in; 5. The relative spatial structure of major roads, buildings, water bodies, vegetation, and other features remains unchanged; 6. It exhibits higher consistency when performing visual location identification with satellite imagery.

[0104] To prevent the image editing model from generating excessive unrealistic content, this invention employs the following constraint strategies during the image editing process: • Structural preservation constraints are applied to the effective projection area; • Perform localized repair constraints on the missing areas; • The text prompt explicitly requires "no change to the layout of major features"; • Use LoRA fine-tuning to adapt the model to the style of satellite images of the training region; The effectiveness of the generated image is indirectly constrained by the feature matching results in subsequent location identification.

[0105] Step 7: Perform visual location identification by comparing the repaired BEV image with satellite imagery. Establish a satellite image database:

[0106] in, Represents the first in the database Zhang Weixing image, This indicates the geographical location corresponding to the satellite image. This indicates the size of the satellite image database.

[0107] Using a visual location recognition feature extraction network Extract features from the restored BEV image and satellite image respectively:

[0108]

[0109] in, This represents the normalized feature vector of the restored BEV image. Indicates the first Normalized feature vectors of Zhang Weixing's images. This indicates a feature normalization operation.

[0110] Calculate the similarity between the restored BEV image and each satellite image:

[0111] in, Indicates the image of the drone to be identified and the first Similarity between satellite images.

[0112] The satellite image with the highest similarity was selected as the recognition result:

[0113] The final output is the corresponding location identification result:

[0114] in, This indicates the geographical location corresponding to the identified drone image.

[0115] In practical applications, the top few candidate satellite images with the highest similarity can also be output and reordered by combining geometric verification, local feature matching or geographical priors to further improve the reliability of visual location recognition.

[0116] Example 1 Step 1: Data Acquisition and Preprocessing Acquire an RGB image captured by the drone in the target area. The image can be an approximate top-down view or a drone image with a certain tilt angle.

[0117] The image is preprocessed as follows: image distortion is corrected based on camera distortion parameters; the image size is adjusted to the input size required by the depth estimation model; image pixel values ​​are normalized; and the camera intrinsic parameter matrix is ​​read or estimated. Read or estimate camera attitude parameters .

[0118] The preprocessed image is then fed into the depth estimation module.

[0119] Step 2, Depth Estimation drone images Input the Depth Anything 3 model to obtain the depth map:

[0120] If the model output is a relative depth, then scale correction is performed based on prior knowledge of the drone's altitude or the scene scale:

[0121] in, and It can be determined by the drone altitude, camera pitch angle, ground reference point, or statistical scale of the training set.

[0122] In this embodiment, if the drone provides flight altitude This height can be used to perform scale approximation correction on the depth map, so that the ground scale obtained by projection is basically consistent with the scale of the satellite image.

[0123] Step 3: Single Image 3D Reconstruction For each valid pixel in the drone image Represent it as homogeneous coordinates:

[0124] Backproject it onto the camera coordinate system using camera intrinsic parameters and depth values:

[0125] Then transform the camera coordinate system points to the world coordinate system:

[0126] Each 3D point is associated with the color of the corresponding pixel in the original image, forming a colored point cloud:

[0127] Then, outlier filtering and ground region selection were performed on the point cloud to obtain the valid point cloud. .

[0128] Step 4: BEV Image Generation Set the ground coverage and resolution of the BEV image. , will dot clouds Projected onto the ground plane.

[0129] For any three-dimensional point Its BEV grille position is:

[0130]

[0131] Write the corresponding color to the BEV image location. The initial BEV image is obtained:

[0132] If multiple points are mapped to the same BEV pixel, the point with higher depth confidence, more reasonable distance from the ground plane, or smaller projection error should be selected first. Alternatively, a weighted average method can be used for fusion.

[0133] Simultaneously generate an effective area mask. It is used to mark which BEV pixels are obtained by projection of the real point cloud and which BEV pixels belong to the missing region.

[0134] Step 5: Training set construction Construct a dataset for LoRA fine-tuning and visual location recognition training. Each sample in the training set includes: 1. Single drone image ; 2. Satellite imagery of the area corresponding to the UAV image. ; 3. Geographic tags or matching tags ; 4. Initial BEV image generated from drone imagery ; 5. Corresponding BEV effective area mask ; 6. Text prompts .

[0135] The text prompts can be set uniformly or dynamically according to different scenarios. For example, for urban scenarios, they can be set as follows: “Convert the BEV drone projection image to a satellite top-down view style, keep the road and building layout unchanged, repair empty areas, and complete the texture of building roofs and road surfaces.” For suburban or natural scenes, it can be set as follows: "Edit the BEV image to a natural satellite image style, preserve the spatial relationships of roads, water bodies, vegetation, and open spaces, repair missing areas, and complete the surface texture." Step 6: LoRA fine-tuning of the FLUX.2 model FLUX.2 was used as the base image editing model. The parameters of the FLUX.2 base model were frozen during training. Only update LoRA parameters .

[0136] The training input is the initial BEV image. Effective area mask and text prompts The training output is a restored satellite-style image. :

[0137] The training objective is to make The style is similar to the corresponding satellite image. At the same time, it maintains the same structure as the initial BEV image. Consistent.

[0138] The total loss function is:

[0139] in, Used to constrain satellite image style Used to constrain the geometry of ground features Used to constrain the consistency between the generated image and the reference satellite image or satellite image features.

[0140] During training, LoRA fine-tuning enables the model to learn the following capabilities: convert BEV images to satellite image style; repair holes caused by point cloud projection; complete missing road, building boundaries, vegetation, and surface textures; maintain the relative spatial position of ground features; and avoid generating large areas of false content that are inconsistent with the input BEV structure.

[0141] Step 7: Text-guided repair of BEV images During the inference phase, for the drone image to be identified, an initial BEV image is first generated. and effective area mask .

[0142] Then input the FLUX.2 model after LoRA fine-tuning:

[0143] in, For text prompts, These are the LoRA parameters after training.

[0144] In this embodiment, the text prompt word is set as follows: "Edit the input BEV image to a clear satellite top view style, keep the spatial layout of roads, buildings, water bodies, vegetation and open spaces unchanged, repair projection holes, fill in missing textures, and do not add new features that contradict the original structure." Output The image is used as a query image for the subsequent location identification module.

[0145] Step 8: Visual Location Recognition Establish a satellite image database:

[0146] The visual location recognition network was used to extract features from the query image and the database image respectively.

[0147]

[0148] Calculate similarity:

[0149] The satellite image with the highest score was selected as the matching result:

[0150] Output location identification results:

[0151] If further improvements in reliability are needed, the front end can be modified. The candidate results are reordered. Reordering methods include, but are not limited to, local feature matching, road structure matching, building outline matching, geographic proximity constraints, or multi-scale image retrieval.

[0152] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0153] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is defined only by the appended claims.

Claims

1. A cross-view visual location recognition method based on BEV generation and inpainting of a single UAV image, characterized in that, The method includes: Acquire a single drone image and perform preprocessing; Depth estimation is performed on the preprocessed UAV images based on a monocular depth estimation model to obtain a depth map; Based on the depth map, combined with camera parameters and UAV attitude information, the image pixels are back-projected into a three-dimensional point cloud; The three-dimensional point cloud is projected onto a horizontal top-view plane to generate an initial BEV image and a corresponding effective region mask. Using an image editing model finely tuned through an image-to-image approach, the initial BEV image is subjected to text-guided image editing and repair to obtain a repaired BEV image. The repaired BEV image is matched and retrieved with a satellite image database to output visual location identification results.

2. The cross-view visual location recognition method based on BEV generation and restoration of a single UAV image according to claim 1, characterized in that, Depth estimation is performed on the preprocessed UAV image based on a monocular depth estimation model to obtain a depth map, specifically: Preprocessed drone images Input a monocular depth estimation model to obtain a dense depth map corresponding to the size of the input image. : in, This represents the depth estimation model; Indicates the parameters of the depth estimation model; This represents the predicted depth map; When the depth estimation model outputs a relative depth, the depth map is scale-corrected using the UAV altitude, camera calibration parameters, ground reference points, or scene scale priors to obtain a scale-corrected depth map. in, This represents the depth map after scale correction. Represents the scaling factor. This indicates the depth bias term.

3. The cross-view visual location recognition method based on BEV generation and restoration of a single UAV image according to claim 2, characterized in that, Based on the depth map, combined with camera parameters and UAV attitude information, the image pixels are back-projected into a 3D point cloud, specifically as follows: For any pixel in the input UAV image, its homogeneous pixel coordinates are denoted as in, and These represent the x and y coordinates of a pixel in the image, respectively; Based on the camera pinhole imaging model, the pixels Combined with its depth value By back-projecting onto the camera coordinate system, we obtain three-dimensional points: in, Represents pixels The corresponding 3D point coordinates in the camera coordinate system Represents the camera intrinsic parameter matrix; Then, based on the camera's extrinsic parameters, transform the 3D points in the camera coordinate system to the ground coordinate system or the world coordinate system: in, Represents a three-dimensional point in the world coordinate system. This represents the rotation matrix from the camera coordinate system to the world coordinate system. This represents the translation vector from the camera coordinate system to the world coordinate system; Perform the above back projection operation on all valid pixels to obtain a 3D point cloud with color information: in, This represents a set of 3D point clouds reconstructed from a single UAV image. This represents the RGB color corresponding to a pixel. This represents the set of valid pixels in the image.

4. The cross-view visual location recognition method based on BEV generation and restoration of a single UAV image according to claim 3, characterized in that, It also includes 3D point clouds Filtering to obtain Filtering methods include: Remove low-confidence points based on depth confidence; Remove points that are too close or too far based on the depth range; Remove points that are clearly outside the ground projection range based on the height threshold; Remove interfering regions based on semantic segmentation results; Outliers are removed using statistical filtering.

5. The cross-view visual location recognition method based on BEV generation and restoration of a single UAV image according to claim 4, characterized in that, The 3D point cloud is projected onto a horizontal top-view plane to generate an initial BEV image and a corresponding effective region mask, specifically as follows: Effective point cloud Project the image onto a horizontal ground plane to generate a BEV image; let the point in the world coordinate system be: in, and Represents coordinates on a horizontal ground plane. Represents the coordinates in the height direction; Let the spatial resolution of the BEV image be... , representing the actual ground distance corresponding to one pixel in the BEV image; let the coverage area of ​​the BEV image be: Then three-dimensional points Raster coordinates in BEV images The calculation is as follows: in, This indicates a round-down operation; The color of each point in the point cloud Alternatively, intermediate visual features can be projected onto the corresponding BEV grid to obtain the initial BEV image: in, This represents the projection operation from the point cloud onto the BEV plane. Represents the initial BEV image; When multiple 3D points are projected onto the same BEV pixel location, the final color of that pixel is determined using a weighted average, nearest height priority, maximum confidence priority, or distance-weighted fusion method; it is represented as: in, Represents aggregate functions, This represents the confidence level of a pixel or depth estimate. Represents pixels The corresponding 3D points are projected onto the BEV image coordinates. ; Generate a mask for the effective BEV region: in, This represents the effective region mask of the BEV image.

6. The cross-view visual location recognition method based on BEV generation and restoration of a single UAV image according to claim 5, characterized in that, Text-guided image editing and restoration are performed on the initial BEV image to obtain the restored BEV image, specifically as follows: A training dataset containing UAV images, corresponding satellite images, and geographic labels was constructed, and LoRA fine-tuning was performed on the FLUX.2 basic image editing model. During training, the parameters of the FLUX.2 basic model were frozen, and only the LoRA low-rank adaptation parameters were updated. The total loss function, which is a weighted sum of style loss, geometry preservation loss, and reconstruction loss, was used to constrain the model to learn the satellite image style, hole repair, and ground feature structure preservation capabilities. During the inference phase, the initial BEV image, the effective BEV region mask, and preset text prompts are input into the finely tuned FLUX.2 model, and the output is a repaired BEV image that has completed hole filling, texture completion, satellite style alignment, and retains the original spatial structure of ground features. The text prompts include semantic constraints such as maintaining the spatial layout of ground features, converting the satellite top-down style, repairing projection holes, and prohibiting the generation of contradictory false ground features.

7. The cross-view visual location recognition method based on BEV generation and restoration of a single UAV image according to claim 6, characterized in that, The repaired BEV image is then matched and retrieved against a satellite image database to output visual location identification results, specifically: Establish a satellite image database: in, Represents the first in the database Zhang Weixing image, This indicates the geographical location corresponding to the satellite image. Indicates the size of the satellite image database; Using a visual location recognition feature extraction network Extract features from the restored BEV image and satellite image respectively: in, This represents the normalized feature vector of the restored BEV image. Indicates the first Normalized feature vectors of Zhang Weixing's images. This indicates a feature normalization operation. Calculate the similarity between the restored BEV image and each satellite image: in, Indicates the image of the drone to be identified and the first Similarity between satellite images; The satellite image with the highest similarity was selected as the recognition result: The final output is the corresponding location identification result: in, This indicates the geographical location corresponding to the identified drone image.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the cross-view visual location recognition method based on single-frame UAV image BEV generation and restoration as described in any one of claims 1 to 7.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the cross-view visual location recognition method based on single-frame UAV image BEV generation and restoration as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the cross-view visual location recognition method based on single-frame UAV image BEV generation and restoration according to any one of claims 1 to 7 by executing the executable instructions.

Images