A geolocation method and system based on hierarchical model training and multi-scale visual matching

Abstract

This application discloses a geolocation method and system based on hierarchical model training and multi-scale visual matching. It employs three key designs: hierarchical decoupled training, panoramic dense matching and unified coordinate transformation, and multi-scale optical magnification fusion matching. Simultaneously, it introduces DEM terrain data to construct a constraint model. Hierarchical decoupled training completely separates the pixel coordinate to camera coordinate transformation from camera coordinate to geographic coordinate positioning tasks, ensuring no parameter sharing or joint optimization. Panoramic dense matching and unified coordinate transformation use the same physical model to uniformly convert the pixel coordinates of all matching points into simulated camera coordinates, eliminating coordinate deviations from multi-source matching points. Multi-scale fusion matching balances global coverage and local accuracy. This application enables end-to-end real-time high-precision positioning of cameras in large-scale natural scenes across the entire area. It can be deployed in field inspection equipment, emergency rescue terminals, etc., providing reliable technical support for positioning scenarios such as forest fire prevention, field inspection, and emergency rescue.

Description

Technical Field

[0001] This application relates to the field of computer vision positioning technology, specifically to a high-precision geolocation method and system based on hierarchical decoupling training and multi-scale visual matching. It is mainly applied to high-precision camera geolocation scenarios such as UAV aerial surveying, field inspection, forest fire prevention, emergency rescue, and farmland protection in large natural scenes such as mountains, forests, grasslands, deserts, and river and lake shorelines. Background Technology

[0002] In natural space applications such as forestry resource monitoring, forest fire early warning, field inspection, and emergency rescue, it is necessary to utilize PTZ cameras deployed in the field to achieve large-scale, multi-angle continuous image acquisition and target monitoring. The core technical prerequisite is to accurately map any target point in the camera's image to its real geographic coordinates (longitude, latitude, and elevation), thereby supporting target tracing, collaborative operation, and intelligent decision-making. Existing technologies generally adopt the conventional process of "feature matching - coordinate transformation - model training," resulting in various solutions such as end-to-end single model, single-scale matching + simple coordinate transformation, and multi-level coordinate transformation + joint training.

[0003] The end-to-end single-model approach couples all tasks, such as pixel coordinate transformation, feature matching, and geographic pose calculation, into a single deep learning model, directly outputting geographic coordinates. This model has a large number of parameters and complex training logic, making it prone to problems such as gradient vanishing, fitting difficulties, and overfitting in natural scenes with complex terrain. At the same time, it does not introduce terrain constraints, which cannot compensate for the positioning errors caused by terrain undulations. The positioning error will increase sharply or even fail in scenes with changes in lighting and vegetation occlusion.

[0004] Single-scale matching + simple coordinate transformation scheme: This scheme uses only panoramic images or local single images for feature matching and relies on simple camera intrinsic parameter formulas to complete coordinate transformation. Panoramic image matching suffers from local detail blurring and cannot capture coordinate shifts caused by abrupt terrain changes; local magnified image matching lacks global scene constraints and is prone to matching ambiguity and positioning drift. Neither approach can simultaneously meet the requirements of full coverage and high-precision local positioning.

[0005] Multi-level coordinate transformation + joint training scheme: This scheme employs a multi-level coordinate transformation of "pixel-camera-geography" and jointly trains the transformation parameters with the positioning model. This scheme suffers from a serious error accumulation problem. Small errors at each transformation level are amplified step by step, reaching errors of tens or even hundreds of meters in large-scale mountainous and desert scenes. At the same time, multi-source matching data such as panoramic images and local images use independent coordinate transformation rules, resulting in inconsistent coordinate systems and the inability to integrate and utilize effective feature data, further reducing positioning accuracy.

[0006] While these existing technologies are effective under specific conditions, they have significant limitations when dealing with large-scale natural scenes with complex terrain (significant undulations, abrupt changes in terrain), variable environments (changing lighting, vegetation obstruction), and vast areas (no artificial markers).

[0007] (1) Model task coupling leads to high training difficulty and poor stability: Traditional end-to-end models couple tasks such as pixel transformation, feature matching, and geospatial computation, resulting in a large number of parameters and complex structure. Gradient vanishing is prone to occur during training, making it impossible to learn effective coordinate mapping relationships in large natural scenes. This results in poor localization stability and a high risk of overfitting, making it difficult to adapt to the generalization requirements of different types of large natural scenes (forests, deserts, mountains). In addition, when optimizing the accuracy of a certain level of coordinate transformation, the entire model needs to be retrained, which is extremely inefficient.

[0008] (2) Single-scale matching cannot meet the needs of both global and local terrain: When only panoramic image matching is used, the details of local terrain are blurred, making it difficult to obtain high-precision matching points and capture the positioning deviation caused by terrain changes; when only local single image matching is used, there is a lack of global scene constraints and terrain constraints, which can easily lead to matching ambiguity and positioning drift, and cannot simultaneously meet the dual needs of full-area natural large scene coverage and high-precision positioning of local terrain.

[0009] (3) Accumulation of multi-level coordinate transformation errors without independent calibration mechanism: The transformation parameters from pixel to camera coordinates and from camera to geographic coordinates are interrelated, and no independent calibration and optimization mechanism has been designed. Furthermore, no error compensation is performed in conjunction with terrain data. Small errors in the first-level transformation will continue to be amplified in subsequent transformations. Combined with the influence of terrain undulations, this will eventually lead to a decrease in overall positioning accuracy, which cannot meet the decimeter-level positioning requirements of natural large-scale scenes. At the same time, the use of different coordinate transformation methods for multi-source matching points results in inconsistent coordinates, further aggravating the accumulation of errors.

[0010] (4) The coordinates of multi-source matching points (such as panoramic and local images) are not consistent and cannot be integrated and utilized: The matching points of panoramic and local single images use different coordinate transformation methods, are not in the same coordinate system, and are not integrated with terrain features. This makes it impossible to achieve the coordinated utilization of global scene features and local terrain detail features, which not only wastes effective matching data, but may also cause conflicts in positioning results, limiting the improvement of positioning accuracy.

[0011] Therefore, there is an urgent need for a geolocation method that can break through the existing framework, adapt to the complex characteristics of large-scale natural scenes across the entire domain, and combine high accuracy, high robustness, and low deployment cost. Summary of the Invention

[0012] The purpose of this application is to overcome the shortcomings of existing technologies and provide a geolocation method and system based on hierarchical model training and multi-scale visual matching. It introduces DEM terrain constraints to solve the positioning deviation caused by the undulation of natural large-scale scenes. It reduces the training difficulty and improves the training stability by hierarchical decoupling training. It combines panoramic dense matching and unified coordinate transformation to eliminate the coordinate deviation and error accumulation of multi-source matching points. It adopts multi-scale fusion matching to take into account both global coverage and local accuracy, so as to achieve high-precision and robust positioning of cameras in natural large-scale scenes across the entire domain. At the same time, it reduces deployment costs and adapts to the deployment needs of equipment with limited field resources.

[0013] The first objective of this application is to provide a geolocation method based on hierarchical model training and multi-scale visual matching.

[0014] The aforementioned objective of this application is achieved through the following technical solution:

[0015] A geolocation method based on hierarchical model training and multi-scale visual matching, the method comprising:

[0016] S1: Acquire DEM terrain data, remote sensing images, and scene images captured by the camera to be located for the target scene. The remote sensing images include panoramic remote sensing images and local remote sensing images. The scene images include multi-angle image sequences, panoramic images, and local magnified images.

[0017] S2: Construct a first mapping model and train the first mapping model independently based on the multi-angle image sequence and its corresponding camera metadata. The first mapping model is used to convert the target pixel coordinates in the scene image into simulated camera coordinates under a unified camera coordinate system.

[0018] S3: Perform feature matching between the panoramic image and the magnified local image and the remote sensing image respectively, and use the first mapping model to convert the pixel coordinates of the matching points on the panoramic image and the magnified local image into the coordinates of the simulated camera. At the same time, combine the DEM terrain data to convert the matching points on the remote sensing image into geographic coordinates to generate an initial set of matching points.

[0019] S4: Construct a second mapping model and train the second mapping model independently based on the initial set of matching points. The second mapping model is used to convert the simulated camera coordinates into geographic coordinates.

[0020] S5: Based on the trained second mapping model, generate a positioning mapping file, and realize bidirectional positioning of the target point between pixel coordinates and geographic coordinates based on the first mapping model and the positioning mapping file. The bidirectional positioning includes forward positioning from the pixel coordinates of the target point to the geographic coordinates, and reverse positioning from the geographic coordinates of the target point to the camera coordinates corresponding to the image taken with the pixel coordinates of the target point as the center.

[0021] Preferably, step S2 specifically includes:

[0022] S21: Based on the field of view parameters of the camera to be located, the camera coordinates corresponding to the multi-angle image sequence are used as the classification basis to classify the multiple camera coordinates obtained by sampling, and each sampling point is used as the center camera coordinate, and the sampling points in the neighborhood of the sampling point in the sampling grid are used as the adjacent camera coordinates.

[0023] S22: Based on the coordinates of the center camera and the coordinates of the adjacent cameras, extract the corresponding center image and adjacent images from the multi-angle image sequence, construct an image pair dataset, and perform feature matching on each image pair in the image pair dataset to generate a mapping dataset. The mapping dataset records the pixel coordinates of the center pixel of the adjacent image in the center image and the camera coordinates corresponding to the adjacent image.

[0024] S23: Construct a physical model containing a lens distortion correction module and a camera mounting tilt correction module as the first mapping model;

[0025] S24: Using the mapping dataset, the parameters of the first mapping model are independently trained using the least squares optimization algorithm until convergence, thus obtaining the trained first mapping model.

[0026] Preferably, in step S23,

[0027] The lens distortion correction module is used to fit the radial distortion coefficient and tangential distortion coefficient of the camera to be positioned, so as to perform distortion correction on the input pixel coordinates.

[0028] The camera mounting tilt correction module is used to fit the three-dimensional components of the gimbal arm length, the gimbal tilt angle, and the three-dimensional mounting tilt parameters of the camera relative to the gimbal, so as to perform tilt correction on the coordinates after distortion correction.

[0029] The first mapping model is configured to take the camera coordinates of the scene image and the target pixel coordinates on the scene image as input, and output the simulated camera coordinates under the unified camera coordinate system after distortion correction and tilt correction.

[0030] Preferably, step S3 specifically includes:

[0031] S31, Local Matching and Transformation: Based on the matching result between the panoramic image and the panoramic remote sensing image in the remote sensing image, determine the geographically corresponding region of the local magnified image, and perform feature matching between the local magnified image and the local remote sensing image within the geographically corresponding region; use the first mapping model to convert the pixel coordinates of the matching points on the local magnified image into the coordinates of the simulated camera, and convert the matching points on the local remote sensing image into geographical coordinates according to the local remote sensing image and the DEM terrain data, generating a subset of local matching points;

[0032] S32, Global Matching and Conversion: Feature matching is performed between the panoramic image and the panoramic remote sensing image in the remote sensing image to obtain panoramic image matching points and their corresponding panoramic remote sensing image matching points; a pre-generated index mapping file is queried to map the pixel coordinates of the panoramic image matching points to the corresponding local magnified image pixel coordinates; the local magnified image pixel coordinates are converted to the simulated camera coordinates using the first mapping model; simultaneously, the panoramic remote sensing image matching points are converted to geographic coordinates based on the panoramic remote sensing image and the DEM terrain data, generating a global matching point subset;

[0033] S33, Dataset fusion: The local matching point subset and the global matching point subset are fused to obtain the initial matching point set.

[0034] Preferably, the index mapping file is generated in the following manner:

[0035] Based on the field of view of the magnified local image, a corresponding sub-image is cropped from the panoramic image, and the sub-image is combined with the magnified local image to form an image pair;

[0036] For each image pair, dense feature matching is performed, and the correspondence between the successfully matched panoramic image pixel coordinates and the corresponding magnified local image pixel coordinates is stored as an index mapping file that supports nearest neighbor search, so as to achieve fast mapping from panoramic image pixel coordinates to magnified local image pixel coordinates.

[0037] Preferably, the first mapping model and the second mapping model do not share parameters and do not perform joint optimization during the training and inference phases. Accordingly, the method further includes:

[0038] S6: When the installation state of the camera to be positioned changes, only step S2 is re-executed to retrain or fine-tune the first mapping model, while keeping the second mapping model unchanged.

[0039] Preferably, step S5 specifically includes:

[0040] S51: Based on the trained second mapping model, the target geographic area is divided into discrete grid points according to a preset step size, and the optimal simulated camera coordinates corresponding to each grid point are calculated to generate a reverse mapping file from geographic coordinates to simulated camera coordinates.

[0041] S52: Based on the trained second mapping model, the observation angle range under the unified camera coordinate system is divided into discrete angle grid points according to a preset step size, and the geographic coordinates corresponding to each angle grid point are calculated to generate a forward mapping file from simulated camera coordinates to geographic coordinates.

[0042] S53: Associate and store the reverse mapping file and the forward mapping file to obtain the positioning mapping file;

[0043] S54: Based on the first mapping model and the forward mapping file in the positioning mapping file, realize the forward positioning from the pixel coordinates of the target point to the geographic coordinates, and based on the reverse mapping file in the positioning mapping file, realize the reverse positioning from the geographic coordinates of the target point to the camera coordinates corresponding to the image taken with the pixel coordinates of the target point as the center.

[0044] Preferably, step S4 specifically includes:

[0045] S41: Construct a white-box physical model based on the WGS84 ellipsoidal coordinate system and camera installation priors as the second mapping model;

[0046] S42: Use the initial set of matching points to perform a first-stage coarse training on the second mapping model to obtain an initial second mapping model;

[0047] S43: Using the initial second mapping model, perform regional loss calculation and filtering on the initial matching point set to obtain the first filtered matching point subset;

[0048] S44: Use the first set of selected matching points to perform a second-stage fine training on the initial second mapping model to obtain an optimized second mapping model;

[0049] S45: Repeat steps S43 to S44 at least once until the model accuracy meets the preset conditions and a trained second mapping model is obtained.

[0050] Preferably, the resolution of the panoramic image is 4K to 8K, and the magnified local image is a local high-resolution image obtained through 2-5x optical zoom.

[0051] The second objective of this application is to provide a geolocation system based on hierarchical model training and multi-scale visual matching.

[0052] The second objective of this application is achieved through the following technical solution:

[0053] A geolocation system based on hierarchical model training and multi-scale visual matching, the system comprising:

[0054] The data acquisition module is used to acquire DEM terrain data, remote sensing images, and scene images collected by the camera to be located for the target scene. The remote sensing images include panoramic remote sensing images and local remote sensing images, and the scene images include multi-angle image sequences, panoramic images, and local magnified images.

[0055] The first model training module is used to construct a first mapping model and independently train the first mapping model based on the multi-angle image sequence and its corresponding camera metadata. The first mapping model is used to convert the target pixel coordinates in the scene image into simulated camera coordinates under a unified camera coordinate system.

[0056] The matching and coordinate unification module is used to perform feature matching between the panoramic image and the magnified local image and the remote sensing image respectively, and use the first mapping model to convert the pixel coordinates of the matching points on the panoramic image and the magnified local image into the simulated camera coordinates. At the same time, it combines the DEM terrain data to convert the matching points on the remote sensing image into geographic coordinates, and generates an initial set of matching points.

[0057] The second model training module is used to construct a second mapping model and independently train the second mapping model based on the initial matching point set. The second mapping model is used to convert the simulated camera coordinates into geographic coordinates.

[0058] The positioning execution module is used to generate a positioning mapping file based on the trained second mapping model, and to realize bidirectional positioning of the target point between pixel coordinates and geographic coordinates based on the first mapping model and the positioning mapping file. The bidirectional positioning includes forward positioning from the pixel coordinates of the target point to the geographic coordinates, and reverse positioning from the geographic coordinates of the target point to the camera coordinates corresponding to the image taken with the pixel coordinates of the target point as the center.

[0059] Preferably, the first model training module is specifically used for:

[0060] Based on the field of view parameters of the camera to be located, the camera coordinates corresponding to the multi-angle image sequence are used as the classification basis to classify the multiple camera coordinates obtained by sampling. Each sampling point is used as the center camera coordinate, and the sampling points in the neighborhood of the sampling grid are used as the adjacent camera coordinates.

[0061] Based on the coordinates of the center camera and the coordinates of the adjacent cameras, the corresponding center image and adjacent images are extracted from the multi-angle image sequence to construct an image pair dataset. Feature matching is performed on each image pair in the image pair dataset to generate a mapping dataset. The mapping dataset records the pixel coordinates of the center pixel of the adjacent image in the center image and the camera coordinates corresponding to the adjacent image.

[0062] A physical model containing a lens distortion correction module and a camera mounting tilt correction module is constructed as the first mapping model;

[0063] Using the mapping dataset, the parameters of the first mapping model are independently trained using the least squares optimization algorithm until convergence, thus obtaining the trained first mapping model.

[0064] Preferably, the lens distortion correction module is used to fit the radial distortion coefficient and tangential distortion coefficient of the camera to be positioned, so as to perform distortion correction on the input pixel coordinates;

[0065] The camera mounting tilt correction module is used to fit the three-dimensional components of the gimbal arm length, the gimbal tilt angle, and the three-dimensional mounting tilt parameters of the camera relative to the gimbal, so as to perform tilt correction on the coordinates after distortion correction.

[0066] The first mapping model is configured to take the camera coordinates of the scene image and the target pixel coordinates on the scene image as input, and output the simulated camera coordinates under the unified camera coordinate system after distortion correction and tilt correction.

[0067] Preferably, the matching and coordinate unification module is specifically used for:

[0068] Local matching and transformation: Based on the matching results between the panoramic image and the panoramic remote sensing image in the remote sensing image, the geographic corresponding region of the local magnified image is determined. Feature matching is performed between the local magnified image and the local remote sensing image within the geographic corresponding region. The pixel coordinates of the matching points on the local magnified image are converted into the simulated camera coordinates using the first mapping model. The matching points on the local remote sensing image are converted into geographic coordinates based on the local remote sensing image and the DEM terrain data, generating a subset of local matching points.

[0069] Global Matching and Transformation: Feature matching is performed between the panoramic image and the panoramic remote sensing image in the remote sensing image to obtain panoramic image matching points and their corresponding panoramic remote sensing image matching points; a pre-generated index mapping file is queried to map the pixel coordinates of the panoramic image matching points to the corresponding local magnified image pixel coordinates; the local magnified image pixel coordinates are converted into the simulated camera coordinates using the first mapping model; simultaneously, the panoramic remote sensing image matching points are converted into geographic coordinates based on the panoramic remote sensing image and the DEM terrain data to generate a global matching point subset;

[0070] Dataset fusion: The local matching point subset and the global matching point subset are fused to obtain the initial matching point set.

[0071] Preferably, the matching and coordinate unification module is further configured to generate the index mapping file in the following manner:

[0072] Based on the field of view of the magnified local image, a corresponding sub-image is cropped from the panoramic image, and the sub-image is combined with the magnified local image to form an image pair;

[0073] For each image pair, dense feature matching is performed, and the correspondence between the successfully matched panoramic image pixel coordinates and the corresponding magnified local image pixel coordinates is stored as an index mapping file that supports nearest neighbor search, so as to achieve fast mapping from panoramic image pixel coordinates to magnified local image pixel coordinates.

[0074] Preferably, the first mapping model and the second mapping model do not share parameters and do not perform joint optimization during the training and inference phases. Accordingly, the system further includes:

[0075] An adaptive model training module is used to trigger only the first model training module to retrain or fine-tune the first mapping model when the installation state of the camera to be located changes, while keeping the second mapping model unchanged.

[0076] Preferably, the positioning execution module is specifically used for:

[0077] Based on the trained second mapping model, the target geographic area is divided into discrete grid points according to a preset step size, and the optimal simulated camera coordinates corresponding to each grid point are calculated to generate a reverse mapping file from geographic coordinates to simulated camera coordinates.

[0078] Based on the trained second mapping model, the observation angle range under the unified camera coordinate system is divided into discrete angle grid points according to a preset step size, and the geographic coordinates corresponding to each angle grid point are calculated to generate a forward mapping file from simulated camera coordinates to geographic coordinates.

[0079] The reverse mapping file and the forward mapping file are associated and stored to obtain the positioning mapping file;

[0080] Based on the first mapping model and the forward mapping file in the positioning mapping file, the forward positioning of the target point from pixel coordinates to geographic coordinates is realized, and based on the reverse mapping file in the positioning mapping file, the directional positioning of the target point from geographic coordinates to the camera coordinates corresponding to the image taken with the pixel coordinates of the target point as the center is realized.

[0081] Preferably, the second model training module is specifically used for:

[0082] A white-box physical model based on the WGS84 ellipsoidal coordinate system and camera installation priors is constructed as the second mapping model;

[0083] The initial matching point set is used to perform a first-stage coarse training on the second mapping model to obtain the initial second mapping model;

[0084] The initial second mapping model is used to perform regional loss calculation and filtering on the initial matching point set to obtain the first filtered matching point subset;

[0085] The initial second mapping model is fine-trained in the second stage using the first subset of selected matching points to obtain the optimized second mapping model.

[0086] Repeat the above filtering and retraining process at least once until the model accuracy meets the preset conditions, and obtain the trained second mapping model.

[0087] Preferably, the resolution of the panoramic image is 4K to 8K, and the magnified local image is a local high-resolution image obtained through 2-5x optical zoom.

[0088] Compared with the prior art, this application has the following beneficial effects:

[0089] (1) The training difficulty is significantly reduced and the stability is greatly improved: By splitting the training into independent tasks through hierarchical decoupling, the number of parameters of a single model is reduced and the task objectives are clear, avoiding problems such as gradient vanishing and fitting difficulties in traditional end-to-end models. The training convergence speed is faster, the model stability is stronger, and subsequent optimization is convenient without the need for overall retraining.

[0090] (2) Positioning accuracy is greatly improved and error accumulation is effectively controlled: panoramic dense matching ensures global consistency, unified coordinate transformation solves the problem of inconsistent coordinates of multi-source matching points, multi-scale fusion matching takes into account both global and local accuracy, combined with terrain constraint calibration, and geographic coordinate transformation of matching points through DEM terrain data, so as to improve positioning accuracy and effectively solve the problem of positioning deviation and error accumulation caused by the undulation of natural large-scale terrain.

[0091] (3) Strong adaptability to complex scenes and high robustness: The multi-scale fusion matching mechanism can adapt to complex situations such as changes in lighting, vegetation occlusion, and blurring of local details in natural large scenes. The global constraints of the panoramic image avoid positioning drift, and the local magnification matching ensures the accuracy of details, which greatly improves the adaptability and robustness of the method in various natural large scenes.

[0092] (4) Convenient deployment and low cost: The number of parameters of a single model after splitting is small and the inference speed is fast. It does not require high-performance computing equipment and can be easily deployed in scenarios with limited resources such as drones, field inspection equipment, and emergency rescue terminals, reducing deployment costs and barriers.

[0093] (5) Strong generalization ability: The layered decoupling design and multi-scale fusion strategy can adapt to different types of large natural scenes such as mountains, forests, deserts, rivers and lakes. There is no need to adjust the model structure for a single scene. The generalization ability is significantly better than the existing technology.

[0094] (6) Achieve efficient bidirectional positioning through positioning mapping files: After completing the training of high-precision models, the complex physical model calculations are discretized and pre-generated as positioning mapping files. In actual deployment, bidirectional accurate positioning between geographic coordinates and pixel coordinates of scene images captured by the camera can be quickly completed through efficient table lookup operations, perfectly adapting to real-time application requirements. Attached Figure Description

[0095] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0096] Figure 1 This is a flowchart illustrating a geolocation method based on hierarchical model training and multi-scale visual matching in an embodiment of this application.

[0097] Figure 2 This is an original scene image from a specific embodiment of this application where effective feature matching cannot be performed using locally magnified images;

[0098] Figure 3 for Figure 2 The image shown is the result of processing the scene image using the method described in this application.

[0099] Figure 4 This is a schematic diagram of the structure of a geolocation system based on hierarchical model training and multi-scale visual matching in an embodiment of this application. Detailed Implementation

[0100] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0101] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described below are merely illustrative. For example, the division of units and modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or modules can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, and can be electrical, mechanical, or other forms.

[0102] In addition, each functional unit in the various embodiments of this application can be integrated into a single processor, or each unit can be a separate device, or two or more units can be integrated into a single device; each functional unit in the various embodiments of this application can be implemented in hardware or in the form of hardware plus software functional units.

[0103] Those skilled in the art will understand that all or part of the steps of the following method embodiments can be implemented by program instructions and related hardware. The aforementioned program instructions can be stored in a computer-readable storage medium. When the program instructions are executed, they perform the steps of the following method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0104] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "a plurality of" or "several" means two or more, unless otherwise explicitly specified.

[0105] In this embodiment, the scenario of the camera to be located being deployed on the top of a forest fire lookout tower is used as an example. In this scenario, the camera to be located is a pan-tilt camera with 360° horizontal rotation and ±90° pitch adjustment capability, installed at a height of about 30 meters, used for all-weather monitoring of the surrounding forest area to achieve accurate location of fire points and suspicious targets.

[0106] like Figure 1 As shown in the figure, this application provides a geolocation method based on hierarchical model training and multi-scale visual matching, which may include the following steps:

[0107] S1: Acquire DEM terrain data, remote sensing images, and scene images captured by the camera to be located for the target scene. The remote sensing images include panoramic remote sensing images and local remote sensing images. The scene images include multi-angle image sequences, panoramic images, and local magnified images.

[0108] DEM (Digital Elevation Model) terrain data provides elevation information for geographic coordinates, helping to resolve positioning errors caused by terrain undulations. Remote sensing images are satellite or aerial remote sensing images covering the monitoring area, including panoramic and local remote sensing images, with a resolution of no less than 1 meter, used to provide a reference base map with precise geographic coordinates. Scene images are acquired by the camera of the pan-tilt unit to be positioned, including three types: multi-angle image sequences, panoramic images, and local magnified images. To improve the accuracy of subsequent feature matching, preprocessing operations such as deblurring and illumination normalization can be performed on the acquired image data.

[0109] In this specific implementation, a forest fire monitoring area in a mountainous region is taken as an example. The pan-tilt camera to be located is deployed at the top of a lookout tower, at a height of approximately 30 meters. The system acquires satellite remote sensing images with a resolution of 0.5 meters from the National Geographic Information Public Service Platform, covering a radius of 10 kilometers centered on the lookout tower. Simultaneously, the pan-tilt camera acquires multi-angle image sequences (for subsequent model training), 4K-8K resolution panoramic images (for global constraints), and 2-5x optically magnified local images (for local fine-tuning).

[0110] S2: Construct the first mapping model and train the first mapping model independently based on the multi-angle image sequence and its corresponding camera metadata. The first mapping model is used to convert the target pixel coordinates in the scene image into simulated camera coordinates (i.e., spatial pointing angles, including horizontal angle P and pitch angle T) in a unified camera coordinate system.

[0111] In this specific implementation, the first mapping model is a white-box model based on physical imaging principles, capable of learning the camera's internal geometric parameters (such as distortion coefficients and focal length) and mounting posture parameters (such as tilt angle). During training, the pixel correspondences between different images in a multi-angle image sequence and their corresponding camera coordinate differences are used as supervision signals, and the least squares optimization algorithm is employed to independently train the model parameters. After training, the first mapping model can receive pixel coordinates, camera center coordinates, and zoom factor from any scene image as input, and output the simulated camera coordinates corresponding to that pixel. The training of this model does not depend on geographic coordinate information, achieving independent modeling from pixel coordinates to spatial pointing angles.

[0112] S3: Perform feature matching between the panoramic image and the magnified local image and the remote sensing image respectively, and use the first mapping model to convert the pixel coordinates of the matching points on the panoramic image and the magnified local image into simulated camera coordinates. At the same time, combine the DEM terrain data to convert the matching points on the remote sensing image into geographic coordinates (including longitude, latitude and elevation) to generate an initial set of matching points.

[0113] In this specific implementation, a deep learning feature detector and a graph neural network matcher are used for cross-modal feature matching. For locally magnified images, feature matching is performed between them and the remote sensing images of the corresponding regions to obtain local matching point pairs; for panoramic images, feature matching is performed between them and the remote sensing images to obtain global matching point pairs. Both types of matching points are converted into unified simulated camera coordinates using a first mapping model, and corresponding geographic coordinates are obtained from the remote sensing images and DEM data to form a matching point dataset. In this way, the simulated camera coordinates of matching points in both panoramic and locally magnified images are calculated using the same first mapping model, achieving coordinate unification of heterogeneous data. After fusing the two types of matching points, an initial matching point set is formed, where each matching point contains the correspondence between simulated camera coordinates (P, T) and geographic coordinates (lng, lat, height).

[0114] S4: Construct a second mapping model and train it independently based on the initial set of matching points. The second mapping model is used to convert the simulated camera coordinates into geographic coordinates.

[0115] In this specific implementation, the second mapping model is a white-box physical model based on the WGS84 ellipsoidal coordinate system and camera installation priors. This model can map simulated camera coordinates (spatial pointing angle) to geographic coordinates (longitude, latitude, and elevation). Using the initial set of matching points generated by S3 as training data, the parameters of the second mapping model are trained independently. After training, the second mapping model can receive simulated camera coordinates as input and output the corresponding geographic coordinates. This model establishes a mapping relationship between spatial pointing angles and real geographic locations, forming a complete positioning chain together with the first mapping model.

[0116] S5: Based on the trained second mapping model, generate a positioning mapping file, and realize bidirectional positioning of the target point between pixel coordinates and geographic coordinates based on the first mapping model and the positioning mapping file. The bidirectional positioning includes forward positioning from the pixel coordinates of the target point to the geographic coordinates, and reverse positioning from the geographic coordinates of the target point to the camera coordinates corresponding to the image taken with the pixel coordinates of the target point as the center.

[0117] In this specific implementation, a static positioning mapping file is generated using a trained second mapping model. A circular area with a radius of 10 kilometers, centered on the watchtower, is used as the target geographic area. Discretization sampling is performed at a certain step size, and the optimal observation angle and geographic coordinates corresponding to each sampling point are calculated, generating forward and reverse mapping files. In practical applications, when a user clicks on a point in the monitoring screen, the system uses the pixel coordinates of that point, first converting them into simulated camera coordinates via the first mapping model, and then into geographic coordinates via the positioning mapping file, returning the precise location in milliseconds. Conversely, after inputting geographic coordinates, the system can also quickly calculate the observation angle the camera should point at based on the positioning mapping file, achieving precise turning.

[0118] The core of this embodiment lies in splitting the camera localization task into two independent sub-tasks: the conversion from pixel coordinates to simulated camera coordinates and the conversion from simulated camera coordinates to geographic coordinates, which are independently completed by the first mapping model and the second mapping model, respectively. The two models do not share parameters, do not perform joint optimization, and are each trained independently using a dedicated dataset.

[0119] During the data acquisition phase, three types of image data were collected: multi-angle image sequences were used to train the first mapping model, and panoramic and magnified images were used for multi-scale matching. The first mapping model learns the camera's internal geometric parameters and mounting attitude parameters through the multi-angle image sequences, establishing an accurate mapping from pixel coordinates to simulated camera coordinates. The second mapping model learns the mapping from simulated camera coordinates to geographic coordinates through an initial set of matching points. The two models are used in series to form a complete localization chain: pixel coordinates → first mapping model → simulated camera coordinates → second mapping model → geographic coordinates.

[0120] In the multi-scale matching stage, panoramic images provide large-scale global features, ensuring global consistency in localization, while magnified local images provide high-resolution local details, ensuring local accuracy in localization. Both types of matching points are transformed to a unified camera coordinate system using the same first mapping model, solving the problem of inconsistent coordinates among multi-source matching points in traditional methods.

[0121] During the positioning execution phase, the complex physical model calculations are transformed into efficient table lookup operations through pre-generated static positioning mapping files, achieving millisecond-level bidirectional positioning response.

[0122] The geolocation method based on hierarchical model training and multi-scale visual matching in this embodiment first acquires DEM terrain data, remote sensing images, and scene images containing multi-angle image sequences, panoramic images, and local magnified images. Then, it constructs and independently trains a first mapping model based on the multi-angle image sequences to convert pixel coordinates into simulated camera coordinates in a unified coordinate system. Next, it matches the panoramic image and the local magnified image with the remote sensing image, and uses the first mapping model to uniformly convert all matching points to simulated camera coordinates. Then, it combines the DEM to generate an initial set of matching points. Next, it constructs and independently trains a second mapping model based on the initial set of matching points to convert simulated camera coordinates into geographic coordinates. Finally, it generates a positioning mapping file based on the second mapping model to achieve bidirectional positioning between pixel coordinates and geographic coordinates. This scheme first decomposes the complex localization task into two independent sub-tasks through layered decoupling training, significantly reducing the model training difficulty and eliminating the need for joint optimization. Then, a unified first mapping model transforms panoramic and local matching points to the same camera coordinate system, eliminating the problem of inconsistent coordinates among multi-source matching points. Next, DEM terrain data is used to perform geographic coordinate transformation on the matching points, effectively utilizing terrain constraints to offset localization errors caused by natural scene undulations. Simultaneously, the panoramic image provides global feature constraints to ensure localization consistency, while the local magnified image provides high-resolution details to ensure local accuracy; these two complement each other to achieve multi-scale fusion matching. Finally, a lookup-based bidirectional localization is achieved through a pre-generated static mapping file. In summary, the method in this embodiment has the advantages of low training difficulty, high localization accuracy, strong robustness to complex scenes, low deployment cost, and adaptability to large-scale natural scenes.

[0123] In one embodiment, step S2 specifically includes:

[0124] S21: Based on the field of view parameters of the camera to be located, the camera coordinates corresponding to the multi-angle image sequence are used as the classification basis to classify the multiple camera coordinates obtained by sampling. Each sampling point is used as the center camera coordinate, and the sampling points in the neighborhood of the sampling point in the sampling grid are used as the adjacent camera coordinates.

[0125] In this specific implementation, taking a horizontal field of view of 60 degrees and a vertical field of view of 35 degrees for the camera to be located as an example, grid sampling is performed with a horizontal interval of 20 degrees and a vertical interval of 10 degrees, forming approximately 180 sampling points. When the sampling point (0°, 0°) is used as the coordinates of the center camera, its adjacent camera coordinates include (20°, 0°), (0°, 10°), (20°, 10°), etc.

[0126] S22: Based on the coordinates of the center camera and the coordinates of the adjacent cameras, extract the corresponding center image and adjacent images from the multi-angle image sequence, construct an image pair dataset, and perform feature matching on each image pair in the image pair dataset to generate a mapping dataset. The mapping dataset records the pixel coordinates of the center pixel of the adjacent image in the center image and the camera coordinates corresponding to the adjacent image.

[0127] In this specific implementation, feature matching is performed between the central image and adjacent images to find the projection position (x, y) of the center pixel of the adjacent image onto the central image. The center camera coordinates of this set of central images, the pixel coordinates, and the camera coordinates (pan, tilt) of the adjacent images when they were captured are recorded to form a mapping data pair.

[0128] S23: Construct a physical model containing a lens distortion correction module and a camera mounting tilt correction module as the first mapping model.

[0129] In the specific implementation of this embodiment, the first mapping model is a white-box model based on the physical imaging principle, which includes two modules: distortion correction and tilt correction. It can learn the camera's internal geometric parameters (such as distortion coefficient, focal length, sensor size, etc.) and mounting posture parameters (such as the three-dimensional components of the gimbal arm length, tilt angle, etc.).

[0130] S24: Using the mapping dataset, the parameters of the first mapping model are independently trained using the least squares optimization algorithm until convergence, thus obtaining the trained first mapping model.

[0131] In the specific implementation of this embodiment, the mapping dataset is input into the first mapping model, and the TRF (Trust-Region Reflective Algorithm) least squares optimization algorithm is used to train the model parameters. The maximum number of iterations is set to 3000, the convergence accuracy is 0.001, and parameter boundary constraints are set until the model converges, thus obtaining the trained first mapping model.

[0132] In this embodiment, the first mapping model acquires multi-angle image sequences through gridded sampling and uses the pixel correspondence between different images and their corresponding camera coordinate differences as supervision signals for training. The model adopts a white-box structure based on physical imaging formulas and includes two modules: distortion correction and mount tilt correction. The distortion correction module fits the radial and tangential distortion coefficients of the camera to correct the distortion of the input pixel coordinates; the mount tilt correction module fits the three-dimensional components of the gimbal arm length, the tilt angle, and the three-dimensional mount tilt parameters of the camera relative to the gimbal to correct the tilt of the coordinates after distortion correction. The model parameters are learned from the mapping dataset through a least-squares optimization algorithm, ultimately achieving a high-precision conversion from pixel coordinates to simulated camera coordinates.

[0133] The training of the first mapping model in this embodiment is based entirely on the pixel correspondence between images and the differences in camera coordinates, without relying on geographic coordinate information, thus achieving independent modeling from pixel coordinates to simulated camera coordinates; the physical model parameters have clear geometric meanings, and the model has strong interpretability; compared with traditional black-box neural networks, the physical model has fewer parameters and faster inference speed, making it suitable for deployment on resource-limited field equipment.

[0134] In one embodiment, in step S23,

[0135] The lens distortion correction module is used to fit the radial and tangential distortion coefficients of the camera to be positioned in order to correct the distortion of the input pixel coordinates.

[0136] Specifically, the lens distortion correction module performs lens distortion correction as follows:

[0137] Step 1: Obtain training parameters

[0138] Obtain the training parameters for the lens distortion correction module, including: camera optical center coordinates (cx, cy), sensor width (sensor_width) and height (sensor_height), one focal length (f), radial distortion coefficients (k1, k2, k3, k4), and tangential distortion coefficients (p1, p2). These parameters are manually set during initialization and used for subsequent distortion correction calculations.

[0139] Step 2: Obtain input data

[0140] The input data received by this module includes: the reference camera coordinates (pan_ref, tilt_ref) of the gimbal camera when the current image was captured, the pixel coordinates (x, y) of the target point to be corrected on the image, the camera coordinates of the center image of the group, and the zoom factor (default 1x).

[0141] Step 3: Pixel coordinate normalization processing

[0142] The pixel coordinates of the target point are converted into normalized coordinates. Specifically, the difference between the pixel coordinates of the target point and the coordinates of the camera's optical center is calculated, and then divided by the width or height of the image to obtain the normalized x-coordinate x_c and y-coordinate y_c.

[0143] Step 4: Calculate the square of the radial radius

[0144] Based on the normalized coordinates obtained in step three, calculate the squared radial distance r from the point to the center of the image, which is the sum of the square of the normalized abscissa x_c and the square of the normalized ordinate y_c.

[0145] Step 5: Calculate the radial distortion correction factor

[0146] Based on the squared radial radius r obtained in step four, and combined with the radial distortion coefficients (k1, k2, k3, k4), the radial distortion correction coefficient radial is calculated. This coefficient is 1 plus the sum of the radial distortion terms of each order, where the squared radial radius is multiplied by the first-order coefficient k1, the second-order coefficient k2, the third-order coefficient k3, and the fourth-order coefficient k4 in sequence.

[0147] Step 6: Calculate the tangential distortion correction term

[0148] Based on the normalized coordinates (x_c, y_c), the squared radial radius r, and the tangential distortion coefficients (p1, p2), calculate the two tangential distortion correction terms respectively.

[0149] Step 7: Calculate distortion-free normalized coordinates

[0150] Multiply the original normalized coordinates (x_c, y_c) obtained in step 3 by the radial distortion correction coefficient radial obtained in step 5, and then add the tangential distortion correction term obtained in step 6 to obtain the distortion-free normalized abscissa x_ud and ordinate y_ud after distortion correction.

[0151] Step 8: Calculate the physical dimension offset

[0152] Based on the distortion-free normalized coordinates (x_ud, y_ud) obtained in step seven, multiply them by the sensor width (sensor_width) and sensor height (sensor_height) respectively to obtain the physical size offset of the target point on the sensor plane, namely the lateral offset dx and the longitudinal offset dy.

[0153] Step 9: Calculate the intermediate pitch angle

[0154] Based on the longitudinal offset dy and one focal length f obtained in step eight, the intermediate pitch angle T_Ay_0 is calculated using the arctangent function.

[0155] Step 10: Overlay reference pitch angle

[0156] After converting the pitch reference angle tilt_ref in the input data from angle to radians, add it to the intermediate pitch angle T_Ay_0 obtained in step nine to obtain the superimposed pitch angle T_Ay.

[0157] Step 11: Calculate the longitudinal projection length

[0158] Calculate the longitudinal projection length L_Ay based on the longitudinal offset dy and one focal length f.

[0159] Step 12: Calculate the effective components in the horizontal direction

[0160] Based on the longitudinal projection length L_Ay obtained in step eleven and the superimposed pitch angle T_Ay obtained in step ten, calculate the effective component L_Ay_ of the projection length in the horizontal direction.

[0161] Step 13: Calculate the straight-line distance in space

[0162] Calculate the complete spatial straight-line distance L_A from the target point to the camera optical center based on the horizontal offset dx, the vertical offset dy, and one focal length f.

[0163] Step Fourteen: Calculate the horizontal projection distance

[0164] Based on the lateral offset dx and the effective horizontal component L_Ay_ obtained in step 12, calculate the projected distance L_A_ of the target point in the horizontal direction.

[0165] Step 15: Calculate the horizontal offset prediction angle

[0166] Based on the lateral offset dx obtained in step eight and the effective horizontal component L_Ay_ obtained in step twelve, the horizontal offset prediction angle dp_pred is calculated using the arctangent function and converted into degrees.

[0167] Step 16: Calculate the horizontal angle without installation tilt.

[0168] Add the horizontal reference angle pan_ref in the input data to the horizontal offset prediction angle dp_pred obtained in step 15, and take the modulus of 360 degrees to obtain the horizontal angle P0 under the condition of no installation tilt.

[0169] Step 17: Calculate the pitch angle without installation tilt.

[0170] Based on the horizontal projection distance L_A_ obtained in step fourteen and the spatial straight-line distance L_A obtained in step thirteen, the pitch angle is calculated using the inverse cosine function and converted into degrees to obtain the pitch angle T0 under the condition of no installation tilt.

[0171] Step 18: Output Results

[0172] The output, after distortion correction and before considering camera tilt, is the simulated camera coordinates (P0, T0), which serves as the final output of the lens distortion correction module. This output will then be used as the input for the camera tilt correction module, performing subsequent tilt correction processing.

[0173] This module takes as input the camera coordinates (pan_ref, tilt_ref) of the camera center and the pixel coordinates (x, y) of the target point. After the above eighteen steps of calculation, it outputs the simulated camera coordinates (P0, T0) without mounting tilt. The entire correction process sequentially completes operations such as pixel coordinate normalization, radial distortion correction, tangential distortion correction, physical size conversion, and angle calculation, achieving accurate compensation for lens distortion and providing a foundation for subsequent mounting tilt correction.

[0174] The camera mounting tilt correction module is used to fit the three-dimensional components of the gimbal arm length, the gimbal tilt angle β, and the three-dimensional mounting tilt parameters of the camera relative to the gimbal, so as to perform tilt correction on the coordinates after distortion correction.

[0175] Specifically, the process for performing camera installation tilt correction using the camera tilt correction module is as follows:

[0176] Step 1: Obtain training parameters

[0177] The training parameters for the camera mounting tilt correction module are obtained, including: the lowest point horizontal angle P_ determined by the gimbal camera rotating one full turn while keeping the pitch angle T=0 constant, and its corresponding three-dimensional components of the gimbal arm length (rx0, ry0, rz0), the gimbal tilt angle β, and the three-dimensional mounting tilt parameters of the camera relative to the gimbal (cam_rx, cam_ry, cam_rz). These parameters are obtained through pre-training and are used for subsequent tilt correction calculations.

[0178] Step 2: Obtain input data

[0179] The analog camera coordinates (P0, T0) without installation tilt, output by the distortion calculation module, are used as the input of this module, where P0 is the horizontal angle and T0 is the pitch angle.

[0180] Step 3: Calculate the three-dimensional components of the gimbal arm length when rotated to the target horizontal angle.

[0181] Based on the three-dimensional component parameters (rx0, ry0, rz0) of the gimbal arm length and the gimbal tilt angle β obtained in step one, as well as the current target horizontal angle P (i.e., the horizontal angle P0 in the input coordinates of the unmounted tilted simulated camera), calculate the component values ​​of the arm length in three dimensions when the gimbal rotates to this horizontal angle, including the first horizontal component, the second horizontal component, and the vertical component. The vertical component is affected by the gimbal tilt angle β.

[0182] Step 4: Construct the tilt direction vector of the gimbal arm

[0183] The three arm length components calculated in step three are combined into a three-dimensional vector in a predetermined order, which is used as the current gimbal arm tilt direction vector v_a.

[0184] Step 5: Obtain the camera mounting tilt direction vector

[0185] Combine the camera 3D mounting tilt parameters (cam_rx, cam_ry, cam_rz) obtained in step one into a camera mounting tilt direction vector v_c.

[0186] Step 6: Calculate the total tilt direction vector

[0187] Using the vertical upward direction [0,0,1] as the reference vector v0, first calculate the dot product of the reference vector v0 and the camera mounting tilt direction vector v_c to obtain the projection component of the reference vector in the camera tilt direction; then calculate the component of the reference vector perpendicular to the camera tilt direction, and perform an operation with the gimbal arm tilt direction vector v_a; finally, add the results of the above two parts to obtain the total tilt direction vector v_total, which comprehensively reflects the gimbal arm tilt and the camera mounting tilt.

[0188] Step 7: Calculate the spatial pointing vector in the untilted state.

[0189] Based on the input coordinates (P0, T0) of the untilted simulated camera, calculate the component values ​​of the pointing in the three dimensions and combine them to form the spatial pointing vector v0 in the untilted state.

[0190] Step 8: Convert the total tilt direction vector into a rotation matrix

[0191] Convert the total tilt direction vector v_total obtained in step six into the corresponding three-dimensional rotation matrix R_tilt.

[0192] Step 9: Apply tilt rotation to obtain the actual pointing vector.

[0193] Multiply the spatial pointing vector v0 obtained in step seven (without tilt) with the rotation matrix R_tilt obtained in step eight to obtain the actual spatial pointing vector v1 after considering the effect of installation tilt.

[0194] Step 10: Calculate the coordinates of the simulated camera after considering tilt.

[0195] Based on the actual spatial pointing vector v1 obtained in step nine, the horizontal angle P1 and the pitch angle T1, considering the tilt effect, are calculated respectively. The horizontal angle P1 is calculated using the arctangent function based on the first and third components of the vector; the pitch angle T1 is calculated using the arctangent function based on the root of the square sum of the second component and the first and third components of the vector.

[0196] Step 11: Output Results

[0197] The output is the simulated camera coordinates (P1, T1) after taking into account the tilt effect, which is the final output of the camera installation tilt correction module.

[0198] The input to this module is the simulated camera coordinates (P0, T0) without installation tilt, output from the distortion calculation section. After the above eleven steps of calculation, the output is the simulated camera coordinates (P1, T1) considering the tilt effect. The entire correction process integrates the arm length change during gimbal rotation, the gimbal's own tilt angle, and the camera's installation tilt relative to the gimbal, achieving precise compensation for camera installation attitude deviations.

[0199] The first mapping model is configured to take the camera coordinates of the scene image (i.e., the pan_ref and tilt_ref of the gimbal when the image is captured) and the target pixel coordinates (x, y) on the scene image as input, and output the simulated camera coordinates (P1, T1) after distortion correction and tilt correction in a unified camera coordinate system.

[0200] In this specific implementation, the input pixel coordinates are first normalized, and then sequentially passed through two independent but cascaded sub-modules: a lens distortion correction module and a camera mount tilt correction module. Specifically:

[0201] First, the input is fed into the lens distortion correction module. This module uses the pre-trained camera optical center, sensor size, focal length, and radial / tangential distortion coefficients to sequentially perform normalization processing, distortion correction, and physical imaging calculations, outputting an intermediate simulated camera coordinate (P0, T0) without mounting tilt.

[0202] Subsequently, (P0,T0) is fed into the camera mounting tilt correction module. This module uses the pre-trained gimbal arm length parameters, gimbal tilt angle, and camera three-dimensional mounting tilt parameters to perform three-dimensional spatial attitude correction on (P0,T0) to compensate for the tilt deviation caused by the actual installation of the gimbal arm and camera, and finally outputs the fully corrected simulated camera coordinates (P1,T1).

[0203] In this embodiment, the entire correction process is clearly divided into four logical stages: normalization, distortion correction, physical imaging calculation, and tilt correction. The distortion correction and physical imaging stages eliminate nonlinear errors caused by the internal optical characteristics of the lens, while the tilt correction stage compensates for system deviations caused by the external mounting posture of the camera.

[0204] The physical model parameters in this embodiment have clear geometric meanings, and the model is highly interpretable. The decoupled design of distortion correction and tilt correction makes the model training more stable, the parameter estimation more accurate, and facilitates subsequent independent optimization or retraining of a single module. Compared with black-box neural networks, the physical model has fewer parameters and faster inference speed, making it suitable for deployment on resource-limited field equipment. The model can accurately fit the distortion characteristics and installation deviations of various camera models.

[0205] In one embodiment, step S3 specifically includes:

[0206] S31, Local Matching and Transformation, the specific process is as follows:

[0207] S311: Perform feature matching between the panoramic image and the panoramic remote sensing image in the remote sensing image to obtain the global offset dp of the panoramic image relative to the panoramic remote sensing image.

[0208] S312: Based on the geographical location and installation height information of the camera to be located, as well as the camera coordinates (p,t) and the offset dp when the magnified image was taken, calculate the geographical corresponding region of the magnified image, and perform feature matching between the magnified image and the local remote sensing image within the geographical corresponding region to obtain a set of matching points.

[0209] S313: Use the first mapping model to convert the pixel coordinates of the matching points on the local magnified image into simulated camera coordinates, and convert the matching points on the local remote sensing image into geographic coordinates (longitude, latitude, elevation) based on the local remote sensing image and DEM terrain data, generating a local matching point subset {P,T,lng,lat,height}.

[0210] S32, global matching and transformation, the specific process is as follows:

[0211] S321, Panoramic Image Cropping and Local Enlarged Image Matching (Index Mapping File Generation): Based on the field of view and center camera coordinates of the local enlarged image, a corresponding sub-image is cropped from the panoramic image, and the sub-image is combined with the local enlarged image to form an image pair. Dense feature matching is performed on each image pair, and the correspondence between the successfully matched panoramic image pixel coordinates and the corresponding local enlarged image pixel coordinates is stored as an index mapping file (npy file) supporting nearest neighbor search. The key is the panoramic image pixel coordinates, and the value is the center camera coordinates and pixel coordinates of the local enlarged image, enabling fast mapping from panoramic image pixel coordinates to local enlarged image pixel coordinates.

[0212] S322, Panoramic Image and Remote Sensing Image Matching: Feature matching is performed between the panoramic image and the panoramic remote sensing image within the remote sensing image to obtain matching point data (panoramic image pixel coordinates - remote sensing image pixel coordinates). Based on the panoramic remote sensing image and DEM topographic data, the remote sensing image pixel coordinates are converted into latitude, longitude, and elevation to obtain panoramic image pixel coordinate - geographic coordinate mapping pairs.

[0213] S323, Panoramic Image Matching Point Unified Coordinate Transformation: The panoramic image pixel coordinates in the panoramic image pixel coordinate-geographic coordinate mapping pair are searched in the index mapping file using the nearest neighbor search algorithm and converted into the center camera coordinates and pixel coordinates of the local magnified image. Then, according to the first mapping model, they are converted into simulated camera coordinates, and finally the global matching point subset {P,T,lng,lat,height} is obtained.

[0214] In this way, whether it is the matching point of a locally magnified image or the matching point of a panoramic image, the simulated camera coordinates are calculated through the same first mapping model, thus achieving coordinate unification of heterogeneous data.

[0215] S33: Dataset Fusion

[0216] By merging the local matching point subset and the global matching point subset, an initial matching point set is obtained.

[0217] By integrating the two, the problems of blurred details and insufficient local positioning accuracy in panoramic images are solved, as well as the problems of lack of global constraints and ambiguity in matching in individual local images.

[0218] In the specific implementation of this embodiment, a forest fire monitoring scenario is taken as an example. For nearby tree targets, a locally magnified image with 2-5x optical magnification is used. After determining the corresponding area in the remote sensing image based on the matching results of the panoramic image and the remote sensing image, it is matched with the remote sensing image in the corresponding area to obtain high-precision local feature matching points. For distant ridgelines, a global matching is performed between 4K-8K panoramic images and remote sensing images to obtain a large range of global feature matching points.

[0219] During the coordinate unification transformation stage, local matching points are directly converted into unified simulated camera coordinates using the first mapping model; global matching points are first mapped to corresponding local magnified image pixel coordinates using an index mapping file, and then converted into unified simulated camera coordinates using the first mapping model. Both types of matching points ultimately obtain simulated camera coordinates through the same first mapping model, achieving coordinate unification of heterogeneous data. Finally, the two types of matching points are merged to form an initial set of matching points covering the entire domain.

[0220] In this embodiment, local matching utilizes the high resolution of magnified local images to obtain high-precision local feature matching points, ensuring the target's positioning accuracy. Global matching utilizes the large field of view of panoramic images to obtain a wide range of global feature matching points, achieving global positioning constraints. Both types of matching points are converted into a unified simulated camera coordinate system through the same first mapping model, solving the problem of inconsistent coordinates among multi-source matching points in traditional methods. Finally, subsets of the two types of matching points are merged to form an initial matching point set with unified coordinate system.

[0221] This embodiment of multi-scale fusion matching takes into account both global coverage and local accuracy, solving both the problem of blurred local details in panoramic images and the problem of lack of global constraints in magnified local images; by performing coordinate transformation through a unified first mapping model, the problem of inconsistent coordinates of multi-source matching points is eliminated; the fused matching point set covers the entire domain, providing a high-quality data foundation for the training of the second mapping model.

[0222] like Figure 2 As shown in a specific application example of this application, when direct feature matching is performed using only a magnified local image and a remote sensing image, the lack of significant and stable texture features in the local area (such as large areas of vegetation cover or gently undulating terrain) results in a small number of matching points with uneven distribution, making it difficult to establish a reliable pixel coordinate-geographic coordinate correspondence. To address this issue, this application introduces a panoramic image for global feature constraints and uses an index mapping file to uniformly transform the global matching points to the camera coordinate system of the magnified local image, ultimately obtaining the following... Figure 3 The diagram shows a dense, uniform, and high-precision distribution of matching points. This demonstrates that the multi-scale fusion matching strategy proposed in this application effectively solves the matching failure problem of a single local image in feature-scarce scenarios, significantly improving the coverage and localization robustness of matching points.

[0223] In one embodiment, the index mapping file is generated in the following manner:

[0224] Based on the field of view of the magnified local image, the corresponding sub-image is cropped from the panoramic image, and the sub-image is combined with the magnified local image to form an image pair.

[0225] Specifically, the magnified image has a certain field of view (e.g., 20° horizontally and 15° vertically), and a sub-image corresponding to the field of view is cropped from the panoramic image based on this field of view.

[0226] For each image pair, dense feature matching is performed, and the correspondence between the successfully matched panoramic image pixel coordinates and the corresponding magnified local image pixel coordinates is stored as an index mapping file that supports nearest neighbor search, so as to achieve fast mapping from panoramic image pixel coordinates to magnified local image pixel coordinates.

[0227] In this specific implementation, the panoramic image resolution is 8000×4000, and the zoomed-in image resolution is 1920×1080. Through dense matching, approximately one million corresponding point pairs are established and stored as an index mapping file in .npy format. The key is the pixel coordinates of the panoramic image, and the values ​​are the corresponding pixel coordinates of the zoomed-in image and the camera coordinates at the time the zoomed-in image was captured. In practical use, given a pixel on the panoramic image, the corresponding zoomed-in image pixel coordinates can be quickly obtained through nearest neighbor search.

[0228] In this embodiment, a geometric correspondence exists between the panoramic image and the magnified local image. However, the accuracy of directly converting from panoramic image pixel coordinates to simulated camera coordinates is limited. This solution establishes a precise mapping relationship between the two images through dense matching, generating an index mapping file. Dense matching differs from sparse feature matching; it matches every pixel or small region in the image, thus establishing a denser and more accurate correspondence. Using this index file, a fast and accurate conversion from panoramic image pixel coordinates to magnified local image pixel coordinates can be achieved. This is then converted to simulated camera coordinates using a first mapping model, thereby obtaining high-precision panoramic image matching points.

[0229] The index mapping file in this embodiment is generated through dense matching, resulting in high matching accuracy and high density; it supports nearest neighbor search, enabling fast query speed and meeting the needs of real-time applications; it implements a high-precision conversion path from panoramic image pixel coordinates to analog camera coordinates, providing high-quality matching point data for global matching; and it effectively solves the problem of insufficient matching accuracy caused by blurred local details in panoramic images.

[0230] In one embodiment, the first mapping model and the second mapping model do not share parameters and do not perform joint optimization during the training and inference phases. Accordingly, the method further includes:

[0231] S6: When the installation status of the camera to be positioned changes (for example, the camera's installation angle shifts due to strong winds, or it needs to be reinstalled for maintenance), only step S2 is re-executed to retrain or fine-tune the first mapping model, while keeping the second mapping model unchanged.

[0232] In this specific implementation, a pan-tilt camera on a forest fire lookout tower is taken as an example. The camera is exposed outdoors for extended periods, and its installation angle may shift slightly due to factors such as strong winds and earthquakes. Using a traditional end-to-end model would require re-collecting a large amount of data and retraining the entire model, which is time-consuming and labor-intensive. However, using the layered decoupling design of this application, since the first mapping model is responsible for converting pixel coordinates to simulated camera coordinates (mainly affected by the camera's internal geometric parameters and installation posture), and the second mapping model is responsible for converting simulated camera coordinates to geographic coordinates (mainly affected by the camera's geographical location), when the installation state changes but the geographical location remains the same, only the first mapping model needs to be retrained to complete system calibration.

[0233] In this embodiment, the first mapping model and the second mapping model do not share parameters or undergo joint optimization during the training and inference phases. That is, each model uses its own dedicated dataset and training strategy, and is trained independently without interference. During the inference phase, the two models are used in series, but their parameters remain independent and are not jointly adjusted.

[0234] This embodiment's layered decoupling design splits the localization task into two independent sub-tasks, with the parameters of the two models being independent of each other. Changes in camera mounting status (such as changes in tilt angle or arm length) mainly affect the distortion and tilt parameters of the first mapping model, but do not affect the second mapping model (where camera latitude, longitude, and elevation remain unchanged). Therefore, only the first mapping model needs to be retrained to quickly complete system calibration, while the second mapping model can remain unchanged.

[0235] This embodiment significantly reduces system maintenance costs, eliminating the need for overall retraining after camera repair or reinstallation; it boasts high model optimization efficiency, requiring no overall retraining when optimizing a specific level of coordinate transformation accuracy; and its layered design provides excellent scalability and maintainability, in stark contrast to existing technologies that require overall model retraining when optimizing a specific level of coordinate transformation accuracy.

[0236] In one embodiment, step S5 specifically includes:

[0237] S51, Generate reverse mapping file: Based on the trained second mapping model, the target geographic area is divided into discrete grid points according to a preset step size, and the optimal simulated camera coordinates corresponding to each grid point are calculated to generate a reverse mapping file from geographic coordinates to simulated camera coordinates.

[0238] S52, Generate a forward mapping file: Based on the trained second mapping model, the observation angle range under the unified camera coordinate system is divided into discrete angle grid points according to a preset step size. The geographic coordinates corresponding to each angle grid point are calculated to generate a forward mapping file from simulated camera coordinates to geographic coordinates.

[0239] S53, Associated Storage: Associate the reverse mapping file and the forward mapping file into a unified location mapping file;

[0240] S54, Bidirectional positioning: Based on the first mapping model and the forward mapping file in the positioning mapping file, the forward positioning of the target point from pixel coordinates to geographic coordinates is realized, and based on the reverse mapping file in the positioning mapping file, the reverse positioning of the target point from geographic coordinates to the camera coordinates corresponding to the image taken with the pixel coordinates of the target point as the center is realized.

[0241] In this specific implementation, a circular area with a radius of 10 kilometers centered on the watchtower is used as the target geographical area. The area is divided into approximately 700,000 grid points with a step size of 10 meters. The optimal simulated camera coordinates corresponding to each grid point are calculated to generate a reverse mapping file. Simultaneously, within the camera PT angle space (horizontal 0-360°, vertical -60° to +30°), sampling is performed with a step size of 0.1° to generate a forward mapping file.

[0242] This embodiment discretizes the complex physical model calculations and pre-generates two mapping files. In practical applications, during forward positioning, the simulated camera coordinates corresponding to the pixel coordinates of the target point are first calculated using the first mapping model, and then the geographic coordinates corresponding to the simulated camera coordinates are found using the forward mapping file in the positioning mapping file. During reverse positioning, the simulated camera coordinates corresponding to the geographic coordinates of the target point are found using the reverse mapping file in the positioning mapping file (these simulated camera coordinates are the camera coordinates corresponding to when the image centered on the pixel coordinates of the target point was taken). The entire process is mainly completed through table lookups, achieving millisecond-level bidirectional positioning response.

[0243] This embodiment features a fast positioning response speed, achieving millisecond-level bidirectional positioning; it significantly reduces the computing power requirements of terminal devices, allowing ordinary mobile terminals to use it smoothly; through discretized pre-computation and static file storage, it realizes the design concept of being complex during training and simple during inference; it is perfectly adapted to application scenarios with extremely high real-time requirements, such as forest fire prevention and emergency command.

[0244] In one embodiment, step S4 specifically includes:

[0245] S41: Construct a white-box physical model based on the WGS84 ellipsoidal coordinate system and camera installation priors as the second mapping model. This model can map simulated camera coordinates (spatial pointing angle) to geographic coordinates (longitude, latitude, and elevation).

[0246] S42: The second mapping model is coarsely trained in the first stage using the initial set of matching points to obtain the initial second mapping model. This stage uses a large learning rate and a relaxed convergence condition to quickly obtain a basically usable model.

[0247] S43: Utilize the initial second mapping model to perform regional loss calculation and filtering on the initial set of matching points to obtain the first subset of selected matching points. Specifically, the camera field of view is divided into multiple horizontal sub-regions at equal intervals in the horizontal direction, and into multiple vertical sub-regions in the vertical direction using a nonlinear transformation based on the pitch angle distribution characteristics. For each matching point, the loss between its actual geographic coordinates and the model-predicted geographic coordinates is calculated. Based on the loss of matching points within each sub-region, the average loss of each sub-region is calculated. Matching points whose loss is less than the average loss of their respective sub-regions and less than a preset multiple (e.g., 2 times) of the global average loss are retained.

[0248] S44: The initial second mapping model is fine-trained in the second stage using the first selected subset of matching points to obtain the optimized second mapping model. A small learning rate is used in this stage to finely optimize the model parameters.

[0249] S45: Repeat steps S43 to S44 at least once until the model accuracy meets the preset conditions and a trained second mapping model is obtained.

[0250] In this specific implementation, a mountain forestry monitoring area is taken as an example. The initial set of matching points may contain a large number of noisy points, especially in remote mountainous areas, where the mismatch rate is high. After obtaining the initial model through the first stage of coarse training, the matching points are filtered by region: the horizontal direction is divided into sub-regions of 5°, and the vertical direction is divided using a quadratic nonlinear curve, making the distant region (absolute value of elevation angle) more densely divided. After retaining the high-quality subset of matching points, the second stage of fine training is performed. This process is repeated, and the model accuracy is gradually improved.

[0251] This embodiment employs a two-stage alternating optimization architecture of "coarse training → filtering → fine training → iteration". The coarse training stage quickly obtains a usable model; the filtering stage uses the model to evaluate matching points, eliminating noise points based on region-specific loss thresholds and retaining high-quality matching points; the fine training stage uses high-quality matching points to finely optimize model parameters; and through multiple iterations, the model accuracy is gradually improved. The region-specific filtering strategy fully considers the differences in observation characteristics between near / far distances and different regions in natural scenes.

[0252] This embodiment effectively resolves the contradiction between inaccurate initial models and high noise in initial data; through the bidirectional promotion of models and data, it achieves gradual convergence from coarse to fine; the regional filtering algorithm fully considers the differences in observation characteristics of different regions in natural scenes, avoiding overfitting of the model to specific regions; the iterative optimization mechanism ensures that the model eventually converges to a high-precision state.

[0253] In one embodiment, the panoramic image has a resolution of 4K to 8K to provide a wide range of global features. During actual acquisition, multiple images are captured by rotating a gimbal camera and then stitched together to generate a high-resolution panoramic image covering a 360° area around the camera.

[0254] The magnified local images are high-resolution images acquired through 2-5x optical zoom, used to provide detailed local features. In actual data acquisition, for key monitoring areas (such as suspected fire spots, illegal buildings, etc.), high-resolution local images are acquired through optical zoom to retain more texture details.

[0255] In this specific implementation, forest fire monitoring is taken as an example. 4K-8K panoramic images can cover a wide area of ​​5 kilometers around the observation tower, providing global visual constraints and avoiding positioning drift; 2-5x optically magnified local images can clearly show detailed features such as trees and smoke plumes on distant mountain peaks, providing high-precision local matching points. The combined use of these two types of images ensures both global consistency in positioning and accuracy in local positioning.

[0256] The multi-scale fusion matching mechanism in this embodiment acquires feature information at different levels through images of two different scales. The panoramic image utilizes 4K-8K high resolution to provide a wide range of global features, ensuring global consistency in positioning; the magnified local image utilizes 2-5x optical zoom to acquire local detailed features, improving positioning accuracy. The two types of information are fused through a unified first mapping model to achieve precise positioning from coarse to fine.

[0257] This embodiment balances global coverage and local accuracy, achieving high-precision positioning in large-scale natural scenes; it enhances the robustness of matching in complex scenes (lighting changes, vegetation occlusion, and blurry local details); the global constraints of the panoramic image effectively avoid positioning drift, while local magnification matching ensures detailed positioning accuracy; the parameter range has been verified in practice and can achieve a good balance between positioning accuracy and computational efficiency.

[0258] like Figure 4 As shown in the figure, this application provides a geolocation system based on hierarchical model training and multi-scale visual matching, which may include:

[0259] The data acquisition module 201 is used to acquire DEM terrain data, remote sensing images, and scene images collected by the camera to be positioned for the target scene. The remote sensing images include panoramic remote sensing images and local remote sensing images, and the scene images include multi-angle image sequences, panoramic images, and local magnified images.

[0260] The first model training module 202 is used to construct the first mapping model and independently train the first mapping model based on the multi-angle image sequence and its corresponding camera metadata. The first mapping model is used to convert the target pixel coordinates in the scene image into simulated camera coordinates under a unified camera coordinate system.

[0261] The matching and coordinate unification module 203 is used to perform feature matching between the panoramic image and the magnified local image and the remote sensing image respectively, and to use the first mapping model to convert the pixel coordinates of the matching points on the panoramic image and the magnified local image into simulated camera coordinates. At the same time, it combines DEM terrain data to convert the matching points on the remote sensing image into geographic coordinates and generate an initial set of matching points.

[0262] The second model training module 204 is used to construct the second mapping model and independently train the second mapping model based on the initial set of matching points. The second mapping model is used to convert the simulated camera coordinates into geographic coordinates.

[0263] The positioning execution module 205 is used to generate a positioning mapping file based on the trained second mapping model, and to realize bidirectional positioning of the target point between pixel coordinates and geographic coordinates based on the first mapping model and the positioning mapping file. The bidirectional positioning includes forward positioning from the pixel coordinates of the target point to the geographic coordinates, and reverse positioning from the geographic coordinates of the target point to the camera coordinates corresponding to the image taken with the pixel coordinates of the target point as the center.

[0264] In one embodiment, the first model training module 202 is specifically used for:

[0265] Based on the field of view parameters of the camera to be located, the camera coordinates corresponding to the multi-angle image sequence are used as the classification basis to classify the multiple camera coordinates obtained by sampling. Each sampling point is used as the center camera coordinate, and the sampling points in the neighborhood of the sampling grid are used as the adjacent camera coordinates.

[0266] Based on the coordinates of the center camera and the coordinates of adjacent cameras, the corresponding center image and adjacent images are extracted from the multi-angle image sequence to construct an image pair dataset. Feature matching is performed on each image pair in the image pair dataset to generate a mapping dataset. The mapping dataset records the pixel coordinates of the center pixel of the adjacent image in the center image and the camera coordinates corresponding to the adjacent image.

[0267] A physical model containing a lens distortion correction module and a camera mount tilt correction module is constructed as the first mapping model;

[0268] Using the mapping dataset, the parameters of the first mapping model are independently trained using the least squares optimization algorithm until convergence, thus obtaining the trained first mapping model.

[0269] In one embodiment, the lens distortion correction module is used to fit the radial distortion coefficient and tangential distortion coefficient of the camera to be positioned in order to perform distortion correction on the input pixel coordinates.

[0270] The camera mounting tilt correction module is used to fit the three-dimensional components of the gimbal arm length, the gimbal tilt angle, and the three-dimensional mounting tilt parameters of the camera relative to the gimbal, so as to perform tilt correction on the coordinates after distortion correction.

[0271] The first mapping model is configured to take the camera coordinates of the scene image and the target pixel coordinates on the scene image as input, and output simulated camera coordinates in a unified camera coordinate system after distortion correction and tilt correction.

[0272] In one embodiment, the matching and coordinate unification module 203 is specifically used for:

[0273] Local matching and transformation: Based on the matching results of panoramic remote sensing images in panoramic images and remote sensing images, the geographic corresponding region of the local magnified image is determined. Feature matching is performed between the local magnified image and the local remote sensing image within the geographic corresponding region. The pixel coordinates of the matching points on the local magnified image are converted into simulated camera coordinates using the first mapping model. Based on the local remote sensing image and DEM terrain data, the matching points on the local remote sensing image are converted into geographic coordinates to generate a subset of local matching points.

[0274] Global Matching and Transformation: Feature matching is performed between the panoramic image and the panoramic remote sensing image to obtain the matching points in the panoramic image and their corresponding matching points in the panoramic remote sensing image; a pre-generated index mapping file is queried to map the pixel coordinates of the matching points in the panoramic image to the corresponding pixel coordinates of the magnified local image; the pixel coordinates of the magnified local image are converted into simulated camera coordinates using the first mapping model; simultaneously, the matching points in the panoramic remote sensing image are converted into geographic coordinates based on the panoramic remote sensing image and DEM terrain data to generate a subset of global matching points;

[0275] Dataset fusion: Merge the local matching point subset and the global matching point subset to obtain the initial matching point set.

[0276] In one embodiment, the matching and coordinate unification module 203 is also used to generate an index mapping file in the following manner:

[0277] Based on the field of view of the magnified local image, the corresponding sub-image is cropped from the panoramic image, and the sub-image is combined with the magnified local image to form an image pair;

[0278] For each image pair, dense feature matching is performed, and the correspondence between the successfully matched panoramic image pixel coordinates and the corresponding magnified local image pixel coordinates is stored as an index mapping file that supports nearest neighbor search, so as to achieve fast mapping from panoramic image pixel coordinates to magnified local image pixel coordinates.

[0279] In one embodiment, the first mapping model and the second mapping model do not share parameters and do not perform joint optimization during the training and inference phases. Accordingly, the system further includes:

[0280] The adaptive model training module is used to trigger the first model training module to retrain or fine-tune the first mapping model when the installation state of the camera to be located changes, while keeping the second mapping model unchanged.

[0281] In one embodiment, the positioning execution module 205 is specifically used for:

[0282] Based on the trained second mapping model, the target geographic area is divided into discrete grid points according to a preset step size. The optimal simulated camera coordinates corresponding to each grid point are calculated, and a reverse mapping file from geographic coordinates to simulated camera coordinates is generated.

[0283] Based on the trained second mapping model, the observation angle range under the unified camera coordinate system is divided into discrete angle grid points according to a preset step size. The geographic coordinates corresponding to each angle grid point are calculated to generate a forward mapping file from simulated camera coordinates to geographic coordinates.

[0284] The reverse mapping file and the forward mapping file are stored together to obtain the positioning mapping file;

[0285] Based on the first mapping model and the forward mapping file in the positioning mapping file, the forward positioning of the target point from pixel coordinates to geographic coordinates is realized, and based on the reverse mapping file in the positioning mapping file, the reverse positioning of the target point from geographic coordinates to the camera coordinates corresponding to the image taken with the pixel coordinates of the target point as the center is realized.

[0286] In one embodiment, the second model training module 204 is specifically used for:

[0287] A white-box physical model based on the WGS84 ellipsoidal coordinate system and camera installation priors is constructed as the second mapping model;

[0288] The initial set of matching points is used to perform the first stage of coarse training on the second mapping model to obtain the initial second mapping model.

[0289] The initial second mapping model is used to perform regional loss calculation and filtering on the initial matching point set to obtain the first filtered matching point subset;

[0290] The initial second mapping model is fine-trained in the second stage using the first set of matched points to obtain the optimized second mapping model.

[0291] Repeat the above filtering and retraining process at least once until the model accuracy meets the preset conditions, and obtain the trained second mapping model.

[0292] In one embodiment, the panoramic image has a resolution of 4K to 8K, and the magnified image is a local high-resolution image obtained through 2-5x optical zoom.

[0293] It should be noted that the geolocation system based on hierarchical model training and multi-scale visual matching in the above embodiments has the same working principle and technical effect as the geolocation method based on hierarchical model training and multi-scale visual matching in the above embodiments, and will not be repeated here.

[0294] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0295] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0296] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0297] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A geolocation method based on hierarchical model training and multi-scale visual matching, characterized in that, The method includes: S1: Acquire DEM terrain data, remote sensing images, and scene images captured by the camera to be located for the target scene. The remote sensing images include panoramic remote sensing images and local remote sensing images. The scene images include multi-angle image sequences, panoramic images, and local magnified images. S2: Construct a first mapping model and train the first mapping model independently based on the multi-angle image sequence and its corresponding camera metadata. The first mapping model is used to convert the target pixel coordinates in the scene image into simulated camera coordinates under a unified camera coordinate system. S3: Perform feature matching between the panoramic image and the magnified local image and the remote sensing image respectively, and use the first mapping model to convert the pixel coordinates of the matching points on the panoramic image and the magnified local image into the coordinates of the simulated camera. At the same time, combine the DEM terrain data to convert the matching points on the remote sensing image into geographic coordinates to generate an initial set of matching points. S4: Construct a second mapping model and train the second mapping model independently based on the initial set of matching points. The second mapping model is used to convert the simulated camera coordinates into geographic coordinates. S5: Based on the trained second mapping model, generate a positioning mapping file, and realize bidirectional positioning of the target point between pixel coordinates and geographic coordinates based on the first mapping model and the positioning mapping file. The bidirectional positioning includes forward positioning from the pixel coordinates of the target point to the geographic coordinates, and reverse positioning from the geographic coordinates of the target point to the camera coordinates corresponding to the image taken with the pixel coordinates of the target point as the center.

2. The geolocation method based on hierarchical model training and multi-scale visual matching according to claim 1, characterized in that, Step S2 specifically includes: S21: Based on the field of view parameters of the camera to be located, the camera coordinates corresponding to the multi-angle image sequence are used as the classification basis to classify the multiple camera coordinates obtained by sampling, and each sampling point is used as the center camera coordinate, and the sampling points in the neighborhood of the sampling point in the sampling grid are used as the adjacent camera coordinates. S22: Based on the coordinates of the center camera and the coordinates of the adjacent cameras, extract the corresponding center image and adjacent images from the multi-angle image sequence, construct an image pair dataset, and perform feature matching on each image pair in the image pair dataset to generate a mapping dataset. The mapping dataset records the pixel coordinates of the center pixel of the adjacent image in the center image and the camera coordinates corresponding to the adjacent image. S23: Construct a physical model containing a lens distortion correction module and a camera mounting tilt correction module as the first mapping model; S24: Using the mapping dataset, the parameters of the first mapping model are independently trained using the least squares optimization algorithm until convergence, thus obtaining the trained first mapping model.

3. The geolocation method based on hierarchical model training and multi-scale visual matching according to claim 2, characterized in that, In step S23, The lens distortion correction module is used to fit the radial distortion coefficient and tangential distortion coefficient of the camera to be positioned, so as to perform distortion correction on the input pixel coordinates. The camera mounting tilt correction module is used to fit the three-dimensional components of the gimbal arm length, the gimbal tilt angle, and the three-dimensional mounting tilt parameters of the camera relative to the gimbal, so as to perform tilt correction on the coordinates after distortion correction. The first mapping model is configured to take the camera coordinates of the scene image and the target pixel coordinates on the scene image as input, and output the simulated camera coordinates under the unified camera coordinate system after distortion correction and tilt correction.

4. The geolocation method based on hierarchical model training and multi-scale visual matching according to claim 1, characterized in that, Step S3 specifically includes: S31, Local Matching and Transformation: Based on the matching result between the panoramic image and the panoramic remote sensing image in the remote sensing image, determine the geographically corresponding region of the local magnified image, and perform feature matching between the local magnified image and the local remote sensing image within the geographically corresponding region; use the first mapping model to convert the pixel coordinates of the matching points on the local magnified image into the coordinates of the simulated camera, and convert the matching points on the local remote sensing image into geographical coordinates according to the local remote sensing image and the DEM terrain data, generating a subset of local matching points; S32, Global Matching and Conversion: Feature matching is performed between the panoramic image and the panoramic remote sensing image in the remote sensing image to obtain panoramic image matching points and their corresponding panoramic remote sensing image matching points; a pre-generated index mapping file is queried to map the pixel coordinates of the panoramic image matching points to the corresponding local magnified image pixel coordinates; the local magnified image pixel coordinates are converted to the simulated camera coordinates using the first mapping model; simultaneously, the panoramic remote sensing image matching points are converted to geographic coordinates based on the panoramic remote sensing image and the DEM terrain data, generating a global matching point subset; S33, Dataset fusion: The local matching point subset and the global matching point subset are fused to obtain the initial matching point set.

5. The geolocation method based on hierarchical model training and multi-scale visual matching according to claim 4, characterized in that, The index mapping file is generated in the following way: Based on the field of view of the magnified local image, a corresponding sub-image is cropped from the panoramic image, and the sub-image is combined with the magnified local image to form an image pair; For each image pair, dense feature matching is performed, and the correspondence between the successfully matched panoramic image pixel coordinates and the corresponding magnified local image pixel coordinates is stored as an index mapping file that supports nearest neighbor search, so as to achieve fast mapping from panoramic image pixel coordinates to magnified local image pixel coordinates.

6. The geolocation method based on hierarchical model training and multi-scale visual matching according to claim 1, characterized in that, The first mapping model and the second mapping model do not share parameters and do not perform joint optimization during the training and inference phases. Accordingly, the method further includes: S6: When the installation state of the camera to be positioned changes, only step S2 is re-executed to retrain or fine-tune the first mapping model, while keeping the second mapping model unchanged.

7. The geolocation method based on hierarchical model training and multi-scale visual matching according to claim 1, characterized in that, Step S5 specifically includes: S51: Based on the trained second mapping model, the target geographic area is divided into discrete grid points according to a preset step size, and the optimal simulated camera coordinates corresponding to each grid point are calculated to generate a reverse mapping file from geographic coordinates to simulated camera coordinates. S52: Based on the trained second mapping model, the observation angle range under the unified camera coordinate system is divided into discrete angle grid points according to a preset step size, and the geographic coordinates corresponding to each angle grid point are calculated to generate a forward mapping file from simulated camera coordinates to geographic coordinates. S53: Associate and store the reverse mapping file and the forward mapping file to obtain the positioning mapping file; S54: Based on the first mapping model and the forward mapping file in the positioning mapping file, realize the forward positioning from the pixel coordinates of the target point to the geographic coordinates, and based on the reverse mapping file in the positioning mapping file, realize the reverse positioning from the geographic coordinates of the target point to the camera coordinates corresponding to the image taken with the pixel coordinates of the target point as the center.

8. The geolocation method based on hierarchical model training and multi-scale visual matching according to claim 1, characterized in that, Step S4 specifically includes: S41: Construct a white-box physical model based on the WGS84 ellipsoidal coordinate system and camera installation priors as the second mapping model; S42: Use the initial set of matching points to perform a first-stage coarse training on the second mapping model to obtain an initial second mapping model; S43: Using the initial second mapping model, perform regional loss calculation and filtering on the initial matching point set to obtain the first filtered matching point subset; S44: Use the first set of selected matching points to perform a second-stage fine training on the initial second mapping model to obtain an optimized second mapping model; S45: Repeat steps S43 to S44 at least once until the model accuracy meets the preset conditions and a trained second mapping model is obtained.

9. The geolocation method based on hierarchical model training and multi-scale visual matching according to any one of claims 1-8, characterized in that, The panoramic image has a resolution of 4K to 8K, and the magnified local image is a local high-resolution image obtained through 2-5x optical zoom.

10. A geolocation system based on hierarchical model training and multi-scale visual matching, characterized in that, The system includes: The data acquisition module is used to acquire DEM terrain data, remote sensing images, and scene images collected by the camera to be located for the target scene. The remote sensing images include panoramic remote sensing images and local remote sensing images, and the scene images include multi-angle image sequences, panoramic images, and local magnified images. The first model training module is used to construct a first mapping model and independently train the first mapping model based on the multi-angle image sequence and its corresponding camera metadata. The first mapping model is used to convert the target pixel coordinates in the scene image into simulated camera coordinates under a unified camera coordinate system. The matching and coordinate unification module is used to perform feature matching between the panoramic image and the magnified local image and the remote sensing image respectively, and use the first mapping model to convert the pixel coordinates of the matching points on the panoramic image and the magnified local image into the simulated camera coordinates. At the same time, it combines the DEM terrain data to convert the matching points on the remote sensing image into geographic coordinates, and generates an initial set of matching points. The second model training module is used to construct a second mapping model and independently train the second mapping model based on the initial matching point set. The second mapping model is used to convert the simulated camera coordinates into geographic coordinates. The positioning execution module is used to generate a positioning mapping file based on the trained second mapping model, and to realize bidirectional positioning of the target point between pixel coordinates and geographic coordinates based on the first mapping model and the positioning mapping file. The bidirectional positioning includes forward positioning from the pixel coordinates of the target point to the geographic coordinates, and reverse positioning from the geographic coordinates of the target point to the camera coordinates corresponding to the image taken with the pixel coordinates of the target point as the center.

Images