Global natural scene bidirectional precise positioning method and device fusing visual physical model constraint, electronic equipment and storage medium

Abstract

The application discloses a global natural scene bidirectional precise positioning method and device fusing visual physical model constraints, electronic equipment and storage medium, the method comprises the following steps: acquiring a pan-tilt camera panoramic image and a corresponding remote sensing image; cross-modal feature matching is performed to obtain an initial matching point set of associated observation angles and geographic coordinates; an initial white box physical model under a unified world coordinate system is constructed and the model is trained; a two-stage training mechanism is adopted, and the model is alternately optimized through a regional matching point filtering algorithm; the final matching point subset is fine-tuned in height disturbance, and the target positioning model is obtained by fine-tuning the model; and a static mapping file is generated based on the target positioning model to realize bidirectional positioning. Under the condition that the camera device has low precision installation and the terrain elevation data is inaccurate, the application can construct a high-precision bidirectional positioning model without manual height measurement, greatly reduce the calibration cost, and realize full-automatic positioning in a full-natural complex environment.

Description

Technical Field

[0001] This application relates to the field of computer vision and geographic information technology, specifically to a method, device, electronic device and storage medium for spatial mapping of cameras and maps for precise two-way positioning in large-scale land space scenes such as natural resources, agriculture and rural areas, water conservancy, ecological environment, transportation, forestry, and emergency response. Background Technology

[0002] In natural space applications such as forestry resource monitoring, forest fire early warning, wildlife tracking, border patrol, and emergency command, pan-tilt cameras with horizontal rotation and pitch adjustment capabilities are widely deployed in the field to achieve continuous image acquisition and target monitoring over a wide area and from multiple angles. The core prerequisite for such applications is accurately mapping any target point in the camera's view to its true geographic coordinates (longitude, latitude, and elevation), thereby supporting precise target location tracking, cross-regional collaborative operations, and intelligent decision-making. The implementation logic typically involves: obtaining corresponding point pairs (i.e., matching points) between the pan-tilt camera's view and the remote sensing image through image matching; training a positioning model based on these matching points; and finally calculating the geographic coordinates of the observed target at a given pan-tilt angle and pitch angle.

[0003] Currently, the industry generally adopts a fixed technical process of "matching point extraction → geometric matrix filtering → localization model training," which has led to the development of various typical solutions. For example:

[0004] ORB+RANSAC+homography matrix+least squares scheme: This scheme is a simplified approach commonly used in low-cost monitoring and localization. It employs the ORB feature matching algorithm (replacing SIFT to reduce computation) to extract matching points from the image. The RANSAC algorithm (Random Sample Consensus) assists the homography matrix in geometric constraint filtering, eliminating obvious mismatches. Finally, using the filtered matching points, a linear mapping model between camera pixel coordinates and geographic coordinates is directly fitted using the least squares algorithm to output the localization parameters. While this scheme is computationally efficient, ORB features result in sparse matching points in weak-texture scenes such as dense forests, and the direct removal of points with missing elevation data leads to significant loss of effective data.

[0005] The KAZE + homography matrix + simple physical model scheme, designed for mid-range positioning and monitoring, uses the KAZE algorithm (with better scale invariance) to extract matching points. After filtering with a homography matrix, a simple physical model is constructed based on camera intrinsics and installation height (ignoring complex factors such as terrain slope), and the positioning model is directly trained. While this scheme improves scale robustness, it does not distinguish between near and far target loss and lacks a compensation mechanism for elevation errors caused by steep slopes, buildings, and other terrain features.

[0006] The SIFT + Fundamental Matrix + Single-Stage Neural Network Solution: For scenarios requiring high precision, this solution uses the SIFT algorithm to improve the stability of matching points, filters based on 3D geometric constraints using a fundamental matrix, and then inputs the matching points into a single-stage neural network to train the localization model. While this solution offers high matching stability, it lacks an iterative optimization mechanism. Improperly set initial filtering thresholds can easily discard correct matching points, and the network structure is not customized for natural scenes, making it prone to overfitting in complex environments.

[0007] Simplified visual SLAM schemes (such as a modified version of ORB-SLAM): This scheme is used for some large-scale localization monitoring. It constructs a scene map in real time and extracts matching points through SLAM technology, and uses map constraints to replace homography matrix filtering to directly train the localization model. Although this scheme can construct a local map to assist matching, the map is prone to failure under dense vegetation occlusion, and it does not introduce the collaborative optimization of the physical imaging model and matching points, resulting in limited convergence accuracy.

[0008] However, while the above solutions may be effective under specific conditions, they all face insurmountable systemic defects in large-scale natural environments (such as mountainous areas, dense forests, and areas with intersecting ravines), specifically manifested as follows:

[0009] 1. Preset position matching has a cumulative error problem: Preset position matching achieves positioning by fitting positioning model parameters. As the positioning range expands and the task duration increases, the error of the model parameters will continue to accumulate, resulting in a continuous decrease in positioning accuracy, which cannot meet the needs of high-precision positioning in large natural space scenarios.

[0010] 2. Poor filtering effect of unavailable elevation data: Existing technologies often directly remove missing or unreliable elevation data matching points without considering the special characteristics of elevation data in forestry scenarios. This can easily lead to a large loss of effective matching points, further affecting the reliability of model training.

[0011] 3. Limited matching features lead to insufficient correct matching points: In natural scenes, dense forest areas have monotonous vegetation textures, large areas of vegetation occlusion, and highly homogeneous landforms (such as contiguous forests and similar mountainous terrain), resulting in very few unique feature points available for feature matching. Existing matching algorithms struggle to extract enough effective features in such scenarios, leading to a large number of incorrect matches. Even after filtering through the fundamental matrix or homography matrix, there are still many incorrect matches and the problem of filtering out correct matches, which fails to provide sufficient and reliable data support for model training. This further exacerbates the problem of insufficient model training and poor generalization ability, ultimately affecting positioning accuracy.

[0012] 4. The matrix filtering mode has inherent defects: Existing technologies rely on geometric matrices such as homography matrices to filter matching points. However, in natural scenes, matching points are affected by terrain undulations and vegetation occlusion, and may not strictly follow the homography transformation rules, which can easily lead to a large number of correct matching points being filtered incorrectly. At the same time, geometric matrices can only achieve geometric constraints and cannot combine the physical characteristics of forestry scenes (such as camera installation height, terrain slope, gravity direction, etc.) for precise filtering, so the constraint effect is limited.

[0013] 5. Imbalance between regional differences and near-far distance loss: In natural scenes, the camera observation angle loss for near targets (such as nearby trees) and far targets (such as distant mountains or distant fire points) differs significantly. Furthermore, the influence weight of matching points in different regions (such as flat land and steep slopes) on model training is not distinguished. It is difficult to directly use distance as a measure of loss weight when constructing the loss function, which can easily lead to excessive deviation in the model's target localization in some regions or at certain distances.

[0014] 6. Significant impact of initial model random error: Existing technologies mostly adopt a single-stage training mode, resulting in a large random error in the initial model. Training the model with a large amount of incorrect matching data (low-threshold matrix filtering) will lead to a large overall error in the localization model; training the model with a small amount of correct matching data (high-threshold matrix filtering) will lead to overfitting of the localization model, resulting in large errors in some areas. Both of these factors will affect the convergence of model training.

[0015] 7. No targeted correction for elevation errors: In natural scenes, features such as houses, forests, and elevated roads can cause the actual elevation to be higher than the measured elevation. Existing technologies do not have an elevation error correction mechanism designed for such scenarios. Directly using the original elevation data to train the model will introduce significant positioning deviations.

[0016] In summary, regardless of optimizations made to feature extraction, geometric constraints, or model training, existing technologies remain confined to the inherent framework of "direct training after matrix filtering," failing to integrate the physical characteristics of natural terrain as active constraints into the entire localization and modeling process. Therefore, a novel, precise bidirectional localization method combining cameras and maps is urgently needed. This method can deeply integrate terrain priors, effectively address feature sparsity and unreliable elevation data, and possess adaptive optimization capabilities to overcome the localization bottleneck in large-scale natural scenes. Summary of the Invention

[0017] To overcome the shortcomings of existing technologies, such as large cumulative error in pre-set position matching, poor elevation data filtering effect, insufficient matching points due to feature sparsity, inherent defects in matrix filtering mode, regional loss imbalance, significant impact of initial model error, and uncorrected elevation error, the core of this application lies in abandoning the modeling approach that relies on local pre-set positions, and instead constructing a globally convergent physical model based on a unified world coordinate system. By designing a two-way iterative optimization closed loop of physical model training and matching point selection, the problem of accurate positioning in large-scale natural scenes across the entire domain is fundamentally solved.

[0018] To address this, this application provides a method, device, electronic device, and storage medium for bidirectional precise positioning of large-scale natural scenes across the entire domain, which integrates visual-physical model constraints. By constructing a white-box physical model, designing a two-stage alternating iterative optimization architecture, and introducing an elevation perturbation fine-tuning mechanism, the method achieves a two-way promotion of matching point quality and model accuracy, effectively solving the problem of precise positioning in large-scale natural scenes across the entire domain.

[0019] The first objective of this application is to provide a method for bidirectional precise localization of large-scale natural scenes across the entire domain, incorporating constraints from a visual-physical model.

[0020] The aforementioned objective of this application is achieved through the following technical solution:

[0021] A method for bidirectional precise localization of large-scale natural scenes across the entire domain, incorporating constraints from a visual-physical model, the method comprising:

[0022] S1, acquire a panoramic image captured by the camera of the pan-tilt unit to be positioned, and acquire a remote sensing image that at least partially covers the geographical area corresponding to the panoramic image;

[0023] S2, by cross-modal feature matching, an initial correspondence between the panoramic image and the remote sensing image is established to obtain an initial matching point set, wherein each matching point in the initial matching point set is associated with observation angle information determined based on the panoramic image and geographic coordinate information determined based on the remote sensing image;

[0024] S3. Construct an initial white-box physical model under a unified world coordinate system. This model is used to characterize the physical constraint relationship of global spatial positioning mapping between geographic coordinates and camera observation position, and is initialized based on camera installation priors. Then, use the initial matching point set to train the parameters of the initial white-box physical model to obtain an initial positioning model containing gimbal camera installation deviation parameters.

[0025] S4, adopts a two-stage training mechanism, iteratively filters matching points through a regional matching point filtering algorithm, and alternately optimizes and trains the initial positioning model to obtain the optimized positioning model;

[0026] S5, the elevation perturbation of the matching point subset finally obtained by iterative screening is fine-tuned to compensate for the low elevation error of the digital elevation model caused by the obstruction of ground objects, and the optimized positioning model is finally fine-tuned based on the fine-tuned matching point subset to obtain the target positioning model for accurate positioning of the gimbal camera.

[0027] S6. Generate a static mapping file based on the target positioning model, and realize bidirectional positioning of camera observation angle and geographic coordinates through the static mapping file.

[0028] Preferably, step S3 includes:

[0029] Based on the WGS84 ellipsoidal coordinate system, the initial installation position and nominal installation height of the gimbal camera, an initial white-box physical model is constructed. The WGS84 ellipsoidal coordinate system serves as the model input, the theoretical observation angle of the gimbal camera in a unified world coordinate system serves as the model output, and the initial installation position and nominal installation height serve as the model's initial parameters. The initial white-box physical model is a parameterized model that maps geographic coordinates to the camera's theoretical observation angle, and its parameters to be optimized characterize the actual installation deviation of the camera relative to the installation prior.

[0030] The initial set of matching points is input into the initial white-box physical model, and the model parameters are iteratively trained using the stochastic gradient descent algorithm until the model loss converges to obtain the initial positioning model. The model parameters include camera latitude and longitude offset, installation height correction, and installation attitude tilt angle.

[0031] Preferably, in step S4, the two-stage training mechanism includes:

[0032] The first stage of coarse training involves using the initial localization model to perform regional loss calculation and filtering on the initial matching point set to obtain a first filtered matching point subset. The initial localization model is then retrained using the first filtered matching point subset until the first convergence condition is met, thus obtaining the first stage localization model.

[0033] The second stage of fine training involves using the first stage localization model to perform regional loss calculation and filtering on the initial matching point set again, obtaining a second set of filtered matching points, and then using the second set of filtered matching points to retrain the first stage localization model until the second convergence condition is met, thus obtaining the optimized localization model.

[0034] Preferably, the regional loss calculation and filtering includes:

[0035] The field of view of the gimbal camera is divided into multiple horizontal sub-regions at equal intervals in the horizontal direction, and into multiple vertical sub-regions in the vertical direction according to the pitch angle distribution characteristics using a nonlinear transformation method.

[0036] For each matching point, calculate the loss between its actual observed angle and the angle predicted by the current positioning model;

[0037] Calculate the mean loss for each sub-region based on the loss of the matching points within each sub-region;

[0038] Matching points whose losses are less than the average loss of their respective sub-regions and less than a preset multiple of the global average loss are retained, forming a filtered subset of matching points.

[0039] Preferably, in the step of dividing the vertical direction into multiple vertical sub-regions using a nonlinear transformation method based on the pitch angle distribution characteristics, a quadratic curve transformation is used for the division, so that the division density in the region with a larger absolute pitch angle value is higher than that in the region with a smaller absolute pitch angle value.

[0040] Preferably, in step S5, the elevation perturbation fine-tuning of the subset of matching points finally obtained by iterative screening includes:

[0041] Fix all parameters of the optimized positioning model except for elevation;

[0042] For each matching point in the subset of matching points, the optimal elevation disturbance amount that minimizes the model prediction loss is solved in reverse.

[0043] The original elevation data of each matching point is corrected using the optimal elevation disturbance to obtain a subset of matching points with fine-tuned elevation.

[0044] Preferably, in step S6, the step of generating a static mapping file based on the target localization model includes:

[0045] Based on the target localization model, the mapping relationship between the geographic coordinates within the target area and the optimal observation angle of the camera is discretized and calculated to generate a reverse mapping file;

[0046] Based on the target positioning model, the mapping relationship between the camera observation angle and geographic coordinates is discretized and calculated to generate a forward mapping file;

[0047] The reverse mapping file and the forward mapping file are stored in an array as two structured static files that support fast loading and indexing.

[0048] The second objective of this application is to provide a bidirectional precise positioning device for large-scale natural scenes that integrates visual-physical model constraints.

[0049] The second objective of this application is achieved through the following technical solution:

[0050] A bidirectional precise positioning device for large-scale natural scenes across the entire domain, incorporating visual-physical model constraints, the device comprising:

[0051] The image acquisition module is used to acquire panoramic images captured by the camera of the pan-tilt unit to be positioned, and to acquire remote sensing images that at least partially cover the geographical area corresponding to the panoramic image.

[0052] The image matching module is used to establish an initial correspondence between the panoramic image and the remote sensing image and obtain an initial matching point set, wherein each matching point in the initial matching point set is associated with observation angle information determined based on the panoramic image and geographic coordinate information determined based on the remote sensing image.

[0053] The model building module is used to construct an initial white-box physical model under a unified world coordinate system. This model is used to characterize the global spatial positioning mapping physical constraint relationship between geographic coordinates and camera observation positions, and is initialized based on camera installation priors. The module also uses the initial matching point set to train the parameters of the initial white-box physical model to obtain an initial positioning model containing gimbal camera installation deviation parameters.

[0054] The iterative optimization module is used to employ a two-stage training mechanism, iteratively filtering matching points through a regional matching point filtering algorithm, and alternately optimizing and training the initial positioning model to obtain the optimized positioning model.

[0055] The elevation perturbation fine-tuning module is used to perform elevation perturbation fine-tuning on the subset of matching points finally obtained by iterative screening, so as to compensate for the low elevation error of the digital elevation model caused by the occlusion of ground objects, and to perform final fine-tuning on the optimized positioning model based on the fine-tuned subset of matching points to obtain the target positioning model for accurate positioning of the gimbal camera.

[0056] The positioning execution module generates a static mapping file based on the target positioning model, and uses the static mapping file to achieve bidirectional positioning of the camera observation angle and geographic coordinates.

[0057] Preferably, the model building module is specifically used for:

[0058] Based on the WGS84 ellipsoidal coordinate system, the initial installation position and nominal installation height of the gimbal camera, an initial white-box physical model is constructed. The WGS84 ellipsoidal coordinate system serves as the model input, the theoretical observation angle of the gimbal camera in a unified world coordinate system serves as the model output, and the initial installation position and nominal installation height serve as the model's initial parameters. The initial white-box physical model is a parameterized model that maps geographic coordinates to the camera's theoretical observation angle, and its parameters to be optimized characterize the actual installation deviation of the camera relative to the installation prior.

[0059] The initial set of matching points is input into the initial white-box physical model, and the model parameters are iteratively trained using the stochastic gradient descent algorithm until the model loss converges to obtain the initial positioning model. The model parameters include camera latitude and longitude offset, installation height correction, and installation attitude tilt angle.

[0060] Preferably, the two-stage training mechanism includes:

[0061] The first stage of coarse training involves using the initial localization model to perform regional loss calculation and filtering on the initial matching point set to obtain a first filtered matching point subset. The initial localization model is then retrained using the first filtered matching point subset until the first convergence condition is met, thus obtaining the first stage localization model.

[0062] The second stage of fine training involves using the first stage localization model to perform regional loss calculation and filtering on the initial matching point set again, obtaining a second set of filtered matching points, and then using the second set of filtered matching points to retrain the first stage localization model until the second convergence condition is met, thus obtaining the optimized localization model.

[0063] Preferably, the regional loss calculation and filtering includes:

[0064] The field of view of the gimbal camera is divided into multiple horizontal sub-regions at equal intervals in the horizontal direction, and into multiple vertical sub-regions in the vertical direction according to the pitch angle distribution characteristics using a nonlinear transformation method.

[0065] For each matching point, calculate the loss between its actual observed angle and the angle predicted by the current positioning model;

[0066] Calculate the mean loss for each sub-region based on the loss of the matching points within each sub-region;

[0067] Matching points whose losses are less than the average loss of their respective sub-regions and less than a preset multiple of the global average loss are retained, forming a filtered subset of matching points.

[0068] Preferably, in the step of dividing the vertical direction into multiple vertical sub-regions using a nonlinear transformation method based on the pitch angle distribution characteristics, a quadratic curve transformation is used for the division, so that the division density in the region with a larger absolute pitch angle value is higher than that in the region with a smaller absolute pitch angle value.

[0069] Preferably, when the elevation perturbation fine-tuning module performs elevation perturbation fine-tuning on the subset of matching points finally obtained from iterative screening, it is specifically used for:

[0070] Fix all parameters of the optimized positioning model except for elevation;

[0071] For each matching point in the subset of matching points, the optimal elevation disturbance amount that minimizes the model prediction loss is solved in reverse.

[0072] The original elevation data of each matching point is corrected using the optimal elevation disturbance to obtain a subset of matching points with fine-tuned elevation.

[0073] Preferably, when the positioning execution module generates a static mapping file based on the target positioning model, it specifically performs the following:

[0074] Based on the target localization model, the mapping relationship between the geographic coordinates within the target area and the optimal observation angle of the camera is discretized and calculated to generate a reverse mapping file;

[0075] Based on the target positioning model, the mapping relationship between the camera observation angle and geographic coordinates is discretized and calculated to generate a forward mapping file;

[0076] The reverse mapping file and the forward mapping file are stored in an array as two structured static files that support fast loading and indexing.

[0077] The third objective of this application is to provide an electronic device.

[0078] The aforementioned objective three of this application is achieved through the following technical solution:

[0079] An electronic device, comprising:

[0080] The method includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the bidirectional precise localization method for global natural large-scale scenes that integrates visual-physical model constraints as described in any of the first objectives of this application.

[0081] The fourth objective of this application is to provide a computer-readable storage medium.

[0082] The fourth objective of this application is achieved through the following technical solution:

[0083] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the bidirectional precise localization method for a large-scale natural scene with fusion of visual-physical model constraints as described in any of the first objectives of this application.

[0084] Compared with the prior art, this application has the following beneficial effects:

[0085] 1. A core technology closed loop of physical model and matching point selection was constructed to fundamentally solve the positioning bottleneck in large-scale natural scenes.

[0086] This application abandons the traditional approach of direct training after a single geometric matrix filtering, and innovatively designs a two-stage alternating optimization architecture (i.e., training-filtering-retraining). This closed-loop collaborative mechanism ensures positioning accuracy from two dimensions: sample quality and model convergence. On the one hand, the physical model is used to perform precise regional constraint filtering of matching points, effectively avoiding the problem of incorrectly eliminating matching points due to terrain undulations and vegetation occlusion causing them to not follow homography transformations. On the other hand, the high-quality matching point set feeds back into the model training, gradually converging it from coarse training to fine training. The two promote each other, jointly ensuring that the model eventually converges to a correct and stable state, which is a necessary and irreplaceable technical means to achieve full-coverage and high-precision positioning.

[0087] 2. It achieves efficient utilization of unavailable elevation data and accurate compensation for elevation errors, significantly improving robustness in complex scenarios.

[0088] To address the challenges of missing elevation data in natural settings such as forestry, or the generally low elevation of DEMs due to features like buildings, trees, and elevated structures, this application proposes a dual solution:

[0089] In the filtering stage, the theoretical observation angle is calculated through a physical model, which replaces the crude processing of directly eliminating unusable elevation points by the traditional method, and greatly reduces the loss of effective matching points.

[0090] In the fine-tuning stage, an innovative elevation perturbation fine-tuning mechanism is introduced. By solving in reverse the optimal elevation perturbation amount that minimizes the model prediction loss, the original elevation data is specifically corrected.

[0091] This combined strategy effectively overcomes the elevation error problem that is common in open natural spaces, and significantly improves the model's positioning accuracy and generalization ability in harsh environments such as dense vegetation and complex terrain.

[0092] 3. An adaptive regional matching point filtering algorithm was designed to effectively balance the positioning stability across the entire scene.

[0093] To address the issue of significant differences in the magnitude of observation angle loss for near / far targets and flat / steep slope regions in natural scenes, which can easily lead to model overfitting or regional bias, this application proposes a refined field-of-view segmentation and a dual threshold filtering mechanism:

[0094] The horizontal direction is equally spaced, and the vertical direction is divided into regions using quadratic curve nonlinear transformation, which makes the division more dense in distant regions with large absolute values ​​of pitch angle, thus conforming to the actual observation characteristics.

[0095] By combining a dual threshold of the local sub-region loss mean and a preset multiple of the global loss mean for filtering, adaptive evaluation of matching quality in different regions is achieved, while a global absolute benchmark is set to prevent local noise interference.

[0096] This algorithm can adapt to the accuracy differences of targets in different regions and at different distances, effectively balancing the localization performance across the entire scene and avoiding overfitting of the model to specific regions.

[0097] 4. Achieve efficient bidirectional positioning through static mapping files, perfectly adapting to real-time application requirements.

[0098] After completing high-precision model training, this application discretizes the complex physical model calculations and pre-generates static mapping files (including forward and reverse mappings). In actual deployment, the system can complete bidirectional precise positioning between geographic coordinates and camera observation angles in milliseconds through efficient table lookup operations. This solution greatly improves response speed while ensuring positioning accuracy, perfectly meeting the stringent real-time requirements of demanding application scenarios such as forest fire early warning, border patrol, and emergency command. Attached Figure Description

[0099] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0100] Figure 1 This is a flowchart illustrating a method for bidirectional precise localization of a large-scale natural scene that integrates visual-physical model constraints, as described in an embodiment of this application.

[0101] Figure 2 This is a schematic diagram of a bidirectional precise positioning device for a large-scale natural scene that integrates visual-physical model constraints, as described in an embodiment of this application.

[0102] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0103] Figure 4 The image shows the matching point results obtained using the method described in this application (A in the image is a remote sensing image, and B is a panoramic image).

[0104] Figure 5 The image obtained using the method described in this application is a fused remote sensing image.

[0105] Figure 6 The image shows the positioning results using the method described in this application when the target distance is 2915.38 meters. Detailed Implementation

[0106] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0107] In the embodiments provided in this application, it should be understood that the disclosed methods and systems can be implemented in other ways. The system embodiments described below are merely illustrative. For example, the division of units and modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or modules can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, and can be electrical, mechanical, or other forms.

[0108] In addition, each functional unit in the various embodiments of this application can be integrated into a single processor, or each unit can be a separate device, or two or more units can be integrated into a single device; each functional unit in the various embodiments of this application can be implemented in hardware or in the form of hardware plus software functional units.

[0109] Those skilled in the art will understand that all or part of the steps of the following method embodiments can be implemented by program instructions and related hardware. The aforementioned program instructions can be stored in a computer-readable storage medium. When the program instructions are executed, they perform the steps of the following method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0110] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "a plurality of" or "several" means two or more, unless otherwise explicitly specified.

[0111] like Figure 1 As shown in the figure, this application provides a method for bidirectional accurate localization of a large-scale natural scene that integrates visual-physical model constraints. The method may include the following steps:

[0112] S1, acquire a panoramic image captured by the camera of the pan-tilt unit to be positioned, and acquire a remote sensing image that at least partially covers the geographical area corresponding to the panoramic image;

[0113] In this embodiment, taking a forest fire prevention scenario as an example, the pan-tilt camera to be positioned is deployed at the top of a forest fire lookout tower, at a height of approximately 30 meters. This camera is equipped with a high-resolution lens and 360° horizontal rotation and ±90° pitch adjustment capabilities, and can periodically (e.g., every 5 minutes) or triggered by a fire warning to acquire a panoramic image. Simultaneously, the system acquires satellite remote sensing images with a resolution of 0.5 meters from national geographic information public service platforms (such as Tianditu or Google Earth) or dedicated satellite imagery services. The coverage area is centered on the lookout tower with a radius of 10 kilometers, ensuring as complete a coverage as possible of the geographical area observable by the panoramic image. The remote sensing images are accompanied by corresponding digital elevation model (DEM) data, used to extract elevation information from the geographic coordinates.

[0114] S2, through cross-modal feature matching, establish the initial correspondence between the panoramic image and the remote sensing image, and obtain the initial matching point set. Each matching point in the initial matching point set is associated with the observation angle information determined based on the panoramic image and the geographic coordinate information determined based on the remote sensing image.

[0115] This embodiment employs the SuperPoint deep learning feature detector to extract key points from both ground-view panoramic images and top-view remote sensing images. Then, the SuperGlue graph neural network matcher is used for cross-view and cross-modal feature point matching. After successful matching, for each key point in the panoramic image, its pixel coordinates are converted into horizontal angle (Pan, denoted by the letter P) and pitch angle (Tilt, denoted by the letter T) using the camera intrinsic parameter matrix, i.e., observation angle information. For the corresponding key point in the remote sensing image, its pixel coordinates are used to query the remote sensing image metadata and DEM to obtain the longitude (lng), latitude (lat), and elevation (h) in the WGS84 coordinate system. This forms an initial matching point set D, which can be represented as... Each matching point .

[0116] Each matching point All include the observation angle obtained by transforming the pixel coordinates of the panoramic image. And WGS84 geographic coordinates extracted from remote sensing images and their digital elevation models (DEMs). , Let be the horizontal angle relative to the camera device after the pixel coordinates of the i-th matching point in the panoramic image have been transformed. Let be the pitch angle relative to the camera device after the pixel coordinates of the i-th matching point in the panoramic image have been transformed. Let be the longitude of the i-th matching point in the WGS84 geographic coordinate system. Let i be the latitude of the i-th matching point in the WGS84 geographic coordinate system. This is the elevation data corresponding to the i-th matching point in the WGS84 geographic coordinate system.

[0117] Specifically, in some other embodiments, cross-modal feature matching algorithms such as SIFT, ORB, or KAZE can be used instead of the SuperPoint+SuperGlue algorithm.

[0118] S3. Construct an initial white-box physical model under a unified world coordinate system. This model is used to characterize the physical constraints of global spatial positioning mapping between geographic coordinates and camera observation positions, and is initialized based on camera installation priors. Then, use the initial matching point set to train the parameters of the initial white-box physical model to obtain an initial positioning model containing gimbal camera installation deviation parameters.

[0119] This embodiment first constructs an initial white-box physical model M based on a geographic coordinate system (such as WGS84, CGCS2000, ITRF series, GRS80, etc.) and the installation parameters of the gimbal camera. P The model is a deterministic mathematical function capable of converting arbitrary geographic coordinates... The theoretical observation angle is mapped through a series of coordinate transformations (such as WGS84 → ECEF → ENU → camera coordinate system considering attitude deflection → spherical coordinates). The parameters to be optimized in the model (i.e., installation deviation parameters) include latitude and longitude offsets. Altitude correction amount and two small-angle attitude tilt angles Then, the initial set of matching points D is input into M. P The model is optimized using algorithms such as stochastic gradient descent. When the total loss of the model (the sum of squared angle errors of all points) converges to a threshold (e.g., 1.0), the iteration stops, and the initial localization model P0 is obtained.

[0120] S4 employs a two-stage training mechanism, iteratively filtering matching points through a regional matching point filtering algorithm and alternately optimizing and training the initial localization model to obtain the optimized localization model.

[0121] This embodiment introduces a two-stage training mechanism to improve model robustness. First, the loss (i.e., the Euclidean distance between the actual observed angle and the model's predicted angle) of each point in the initial matching point set D is calculated using the initial localization model P0. Then, a region filtering algorithm is executed to select a high-quality matching point subset E1. Next, P0 is retrained using E1 to obtain the first-stage localization model P1. Subsequently, the original set D is filtered and retrained again using P1 to obtain the second-stage, higher-precision optimized localization model P2. This process achieves bidirectional promotion between the model and the data.

[0122] S5. The elevation perturbation of the matching point subset obtained by the iterative screening is fine-tuned to compensate for the low elevation error of the digital elevation model caused by the occlusion of ground objects. Based on the fine-tuned matching point subset, the optimized positioning model is finally fine-tuned to obtain the target positioning model for accurate positioning of the gimbal camera.

[0123] Considering that in forest scenarios, the elevation data of remote sensing DEMs typically only reflects the top of the tree canopy or the exposed ground, while the actual elevation of the observed targets (such as forest fire spots or animals) may be much lower than the DEM value, this embodiment, for the final matching point subset E2 selected in the second stage, fixes all parameters of the optimized positioning model P2 except for elevation, and only sets the elevation of each matching point. As a variable, the optimal elevation disturbance that minimizes the model prediction loss at that point can be solved in reverse. Update the original elevation to This yields the matching point set E2' after elevation fine-tuning. Finally, P2 is fine-tuned using E2' to obtain the final target localization model P3.

[0124] S6 generates a static mapping file based on the target positioning model, and uses the static mapping file to achieve bidirectional positioning of the camera observation angle and geographic coordinates.

[0125] To meet real-time positioning requirements, this embodiment utilizes the final target positioning model P3, discretizing the sampling within a circular area centered on the watchtower and with a radius of 10 kilometers using a 1m x 1m grid. For each grid point, the corresponding optimal observation angle is calculated, generating a reverse mapping file from geographic coordinates to PT angles. Simultaneously, the PT angle space within the camera's field of view is discretized, and the geographic coordinates corresponding to each angle are calculated, generating a forward mapping file from PT angles to geographic coordinates. These two mapping relationships are stored as two static files (such as .npy format files) that support fast loading and indexing.

[0126] In summary, the global natural large-scale scene bidirectional precise positioning method of this embodiment, which integrates visual physical model constraints, first acquires a panoramic image captured by the gimbal camera to be positioned, and then acquires a remote sensing image that at least partially covers the geographical area corresponding to the panoramic image; then, through cross-modal feature matching, an initial correspondence is established between the panoramic image and the remote sensing image to obtain an initial matching point set; next, an initial white-box physical model based on the geographic coordinate system and camera installation priors is constructed, and the initial white-box physical model is trained using the initial matching point set to obtain an initial positioning model including gimbal camera installation deviation parameters; then... A two-stage training mechanism is adopted. The matching points are iteratively filtered through a regional matching point filtering algorithm, and the initial positioning model is alternately optimized and trained to obtain an optimized positioning model. Then, the elevation perturbation of the finally obtained matching point subset is fine-tuned to compensate for the low elevation error of the digital elevation model caused by ground object occlusion. Based on the fine-tuned matching point subset, the optimized positioning model is finally fine-tuned to obtain the target positioning model for accurate positioning of the gimbal camera. Finally, a static mapping file is generated based on the target positioning model, and bidirectional positioning of camera observation angle and geographic coordinates is achieved through the static mapping file.

[0127] This embodiment constructs a complete, end-to-end positioning process, tightly integrating complex physical modeling, data optimization, and engineering deployment. Its core lies in using the physical model as a constraint, and through multi-stage data cleaning and model iteration, ultimately generating a statically deployable map. This solution effectively overcomes the model bias problem caused by inaccurate single-filtering in traditional methods, and specifically addresses the elevation errors commonly found in natural scenes through an elevation fine-tuning mechanism. Thus, in complex, all-domain scenarios such as forest fire prevention, it achieves a leap from basic positioning to precise positioning.

[0128] In one embodiment, step S2 includes:

[0129] Based on the WGS84 ellipsoidal coordinate system, the initial installation position and nominal installation height of the gimbal camera, an initial white-box physical model is constructed. The WGS84 ellipsoidal coordinate system is used as the model input, the theoretical observation angle of the gimbal camera in the unified world coordinate system is used as the model output, and the initial installation position and nominal installation height are used as the initial parameters of the model. The initial white-box physical model is a parameterized model that maps geographic coordinates to the theoretical observation angle of the camera. Its parameters to be optimized represent the actual installation deviation of the camera relative to the installation prior.

[0130] The initial set of matching points is input into the initial white-box physical model, and the model parameters are iteratively trained using the stochastic gradient descent algorithm until the model loss converges, thus obtaining the initial localization model. The model parameters include camera latitude and longitude offset, installation height correction, and installation attitude tilt angle.

[0131] In this embodiment, the pan-tilt camera is deployed on a monitoring tower along the border, with a nominal installation height of 25 meters. Due to potential interference with GNSS signals in border areas, its initial installation location is recorded. There is an error margin of approximately ±30 meters. Additionally, the tower may experience slight deformation under strong winds, causing the camera's optical axis to be not perfectly horizontal. To address this scenario, this embodiment first constructs a transformation function from geographic coordinates to geocentric Earth-Fixed (ECEF) coordinates using the standard parameters of the WGS84 ellipsoid (major semi-axis a = 6378137.0 m, flattening f = 1 / 298.257223563). Then, using the nominal position... A local ENU (East-North-South) coordinate system is constructed with the origin as the origin. Based on this, two attitude parameters to be calibrated are introduced: pitch angle and tilt angle. (Indicates the vertical tilt of the optical axis in the vertical plane) and roll angle (This indicates the left and right tilt of the optical axis in the horizontal plane). In this way, any geographic coordinate point can be mapped to a three-dimensional vector in the camera coordinate system through this series of deterministic transformations, thereby allowing the calculation of the theoretical observation angle. This complete transformation chain is the initial white-box physical model M. P Subsequently, the initial set of matching points D obtained through cross-modal matching is input into M. P Use a nonlinear least squares optimizer (such as least_squares in the SciPy library) to optimize the parameter vector. The solution is obtained through iterative iteration. The optimization objective is to minimize the angle prediction error of all matching points. The model is considered converged when the loss decreases by less than 0.1% over 10 consecutive iterations; this model is then considered the initial localization model P0, with its parameters... It accurately reflects the camera's true pose deviation relative to the prior installation.

[0132] The core of this embodiment lies in transforming the ambiguous engineering installation state into a set of precisely solvable physical parameters. By constructing a white-box model based on WGS84 and explicit physical transformations, this solution avoids the uninterpretability of black-box neural networks. Using matching point data to train the model's parameters not only yields a usable initial positioning model P0, but more importantly, its output parameters (such as latitude and longitude offsets and attitude tilt angles) have clear physical meanings and can be directly used to guide the calibration and maintenance of on-site equipment, significantly improving the system's maintainability and reliability.

[0133] In one embodiment, the two-stage training mechanism includes:

[0134] The first stage of coarse training involves calculating and filtering the regional loss of the initial matching point set using the initial localization model to obtain the first set of filtered matching points. The initial localization model is then retrained using the first set of filtered matching points until the first convergence condition is met, thus obtaining the first stage localization model.

[0135] The second stage of fine training involves using the first-stage localization model to perform regional loss calculation and filtering on the initial set of matching points again, obtaining a second subset of filtered matching points. The first-stage localization model is then retrained using this second subset of filtered matching points until the second convergence condition is met, resulting in an optimized localization model.

[0136] In this embodiment, the initial matching point set D contains a large number of noise points generated by cross-modal matching, especially in remote mountainous areas where the mismatch rate is high. Directly training the model using D would introduce significant bias. Therefore, this embodiment designs a two-stage training mechanism. In the first stage (coarse training), the loss is calculated for all points in D using the newly obtained initial localization model P0. Since P0 itself has limited accuracy, the filtering criteria in this stage are relatively lenient, aiming to remove obviously erroneous outliers and retain a larger but relatively clean subset E1. The model is retrained using E1 to obtain a more accurate first-stage localization model P1. In the second stage (fine training), the loss is calculated and filtered again on the original set D (instead of E1) using the more accurate P1. At this point, P1 can more accurately distinguish between correct and incorrect matching points, thus allowing for a stricter filtering standard, resulting in a smaller but high-quality subset E2. Finally, P1 is fine-tuned using E2 to obtain the final optimized localization model P2. This strategy of re-examining the original data with a better model effectively avoids the loss of correct points caused by a single filtering step.

[0137] This embodiment cleverly resolves the contradiction between inaccurate initial models and high noise levels in initial data through a two-stage, progressive model-data co-optimization approach. The first stage, coarse training, aims to reduce noise, providing a reliable starting point for the second stage; the second stage, fine training, aims to refine the data, fully utilizing all potentially correct information in the original data. This architecture is significantly superior to the traditional "single-filter-single-train" model, and is particularly suitable for large-scale natural scenes with low signal-to-noise ratios at matching points. It can stably converge to a high-precision model, which is a key step in ensuring the final positioning performance.

[0138] In one embodiment, the regional loss calculation and filtering includes:

[0139] The field of view of the gimbal camera is divided into multiple horizontal sub-regions at equal intervals in the horizontal direction, and into multiple vertical sub-regions in the vertical direction according to the pitch angle distribution characteristics using a non-linear transformation method.

[0140] For each matching point, calculate the loss between its actual observed angle and the angle predicted by the current positioning model;

[0141] Calculate the mean loss for each sub-region based on the loss of the matching points within each sub-region;

[0142] Matching points whose losses are less than the average loss of their respective sub-regions and less than a preset multiple of the global average loss are retained, forming a filtered subset of matching points.

[0143] Specifically, the preset multiple is preferably a value between 1.5 and 3.0, and in this embodiment, the preset multiple is 2 times.

[0144] In this embodiment, the gimbal camera has a horizontal field of view of 360° and a vertical field of view of -60° (looking down) to +30° (looking up). To accommodate the matching characteristics of different regions, this embodiment divides the horizontal direction into 72 equally spaced sub-regions (one every 5°). In the vertical direction, considering that the human eye and camera have higher positioning accuracy requirements for near-range (high pitch angle absolute value) targets, and that matching points in this region are usually denser, a nonlinear transformation method (such as quadratic curve transformation, exponential transformation, or logarithmic transformation) is used for nonlinear division, dividing the -60° to +30° region into 10 sub-regions, making the high absolute value regions such as -60° to -30° and +20° to +30° denser. For the current positioning model (such as P0 or P1), its loss for each point in the initial matching point set D is calculated. Next, the average loss within each sub-region is calculated. and the global loss mean of all points. For a matching point to be preserved, two conditions must be met simultaneously: and The first condition guarantees that a point is good within its local region, while the second condition prevents a subregion with poor overall quality (its...). (The inherently high) release of high-loss points.

[0145] The proposed region-based filtering algorithm in this embodiment fully considers the differences in observation characteristics between near / far and upper / lower regions in natural scenes through fine-grained region division with equal horizontal intervals and vertical nonlinearity. Its innovative dual threshold mechanism of local mean + global multiple notch not only achieves adaptive evaluation of matching quality in different regions but also ensures absolute reliability of filtering through a global benchmark. This algorithm effectively preserves high-quality matching points while strongly suppressing noise, providing a solid data foundation for subsequent model training and is one of the core technologies for improving the robustness of the system in complex global scenes.

[0146] It should be noted that the embodiments of this application provide a filtering approach for matching points through physical models and adaptive small regions. This is not limited to the specific physical model, small region division method, or threshold setting method used. Equivalent methods include, but are not limited to, the following:

[0147] Region partitioning can be achieved using clustering algorithms (such as K-Means);

[0148] Instead of dividing the model into regions, a weight such as distance is added to the loss function during model training to achieve different loss thresholds for different regions.

[0149] The design of the regional threshold function is not limited to using the mean; it can also incorporate statistics such as variance and median to stabilize the data.

[0150] Before filtering the physical model, preprocessing operations such as pre-filtering can be performed by introducing fundamental matrices and homography matrices.

[0151] The physical model can be any parametric model that can map geographic coordinates to observation angles.

[0152] All the above equivalent replacement schemes can achieve the regional threshold physical model filtering effect of this application, and all serve the core architecture of "global spatial coordinate system construction + bidirectional alternating training + visual physical model fusion", so they all fall within the protection scope of this application.

[0153] In one embodiment, in the step of dividing the vertical direction into multiple vertical sub-regions using a nonlinear transformation based on the pitch angle distribution characteristics, a quadratic curve transformation is used for the division, so that the division density is higher in regions with larger absolute pitch angle values ​​than in regions with smaller absolute pitch angle values.

[0154] In this embodiment, for wildlife monitoring cameras deployed in plains areas, the main observation targets (such as deer and wild boar) are mostly located at medium to close range (corresponding to elevation angles between -10° and -40°). To achieve more refined loss assessment and filtering in these key areas, this embodiment employs a quadratic curve in the vertical direction. The system performs region division. Specifically, the pitch angle range [-60°, +30°] is mapped to the interval [0, 1], and then non-uniformly divided into N segments using a quadratic function. The segmentation density can be flexibly controlled by adjusting coefficients a, b, and c. For example, setting a > 0 results in more segmentation boundaries at the ends with larger absolute pitch angle values ​​(-60° and +30°), while the segmentation is sparser near the horizontal 0° area. This segmentation strategy perfectly meets the application requirements for higher positioning accuracy for nearby targets.

[0155] This embodiment introduces a specific nonlinear partitioning method—a quadratic curve transformation—to translate the general idea of ​​densely partitioning high-elevation-angle regions into an executable algorithm. This partitioning method is computationally simple and highly effective, ensuring that valuable computational resources and model attention are focused on the regions with the highest application value. Thus, without increasing overall computational complexity, it significantly improves the system's positioning accuracy and reliability for key targets.

[0156] In one embodiment, fine-tuning the elevation perturbation of the subset of matching points finally obtained through iterative filtering includes:

[0157] Fix all parameters of the optimized positioning model except for elevation;

[0158] For each matching point in the subset of matching points, the optimal elevation disturbance that minimizes the model prediction loss is solved in reverse.

[0159] The original elevation data of each matching point is corrected by using the optimal elevation disturbance, resulting in a subset of matching points with fine-tuned elevation.

[0160] In this embodiment, the application scenario is the forest-building boundary area on the edge of a city. Remote sensing DEM data in this area has serious problems: for buildings, the DEM records the roof elevation; for forests, it records the canopy elevation. However, the actual elevation of the targets observed by the camera (such as vehicles on the road and pedestrians in the forest) is much lower than the DEM value. After two stages of training, the resulting subset of matching points E2, although of high quality, has a systematically high elevation data skewness. Therefore, this embodiment fixes all parameters (position, attitude, etc.) of the converged optimized localization model P2, and only records the elevation data of each matching point... elevation Treated as free variables, an optimal elevation disturbance is independently found for each point using one-dimensional numerical optimization methods (such as the golden section search). This causes the model prediction loss at that point to be... To achieve a local minimum, the original elevation... Updated to This yields the corrected set of elevation fine-tuning matching points, E2'. This operation is equivalent to finding the most reasonable ground elevation for each matching point under the current physical model.

[0161] The elevation perturbation fine-tuning mechanism in this embodiment is an intelligent compensation for the limitations of existing geographic information data. It does not rely on additional high-precision elevation data sources, but cleverly utilizes existing physical imaging models and high-quality matching points to correct elevations through reverse optimization. This method specifically addresses the problem of artificially inflated elevations prevalent in open natural spaces, significantly reducing the resulting positioning errors and greatly improving the model's practicality and accuracy in complex mixed scenarios such as suburban areas and forest regions.

[0162] In one embodiment, the step of generating a static mapping file based on the target localization model includes:

[0163] Based on the target localization model, the mapping relationship between the geographic coordinates within the target area and the optimal observation angle of the camera is discretized and calculated to generate a reverse mapping file;

[0164] Based on the target localization model, the mapping relationship between camera observation angle and geographic coordinates is discretized and calculated to generate a forward mapping file;

[0165] The reverse mapping file and the forward mapping file are stored as two static files that support fast loading and indexing, respectively, in an array.

[0166] In this embodiment, the final target localization model P3 has been deployed on a server in a forest fire command center. To meet the needs of frontline firefighters for real-time queries via mobile terminals, the system needs to transform the complex model inference process into an instant response. This embodiment first defines the target area as a circular area with a radius of 15 kilometers centered on a lookout tower. Within this area, a regular grid is constructed with a step size of 10 meters, containing approximately 700,000 grid points. For each grid point, the P3 model is called to calculate its corresponding optimal horizontal and vertical angles, and the results are stored in a two-dimensional array, forming a reverse mapping file (e.g., Geo2PT.npy stored in .npy format). Simultaneously, within the camera's PT angle space (horizontal 0-360°, vertical -60° to +30°), sampling is performed with a step size of 0.1° to calculate the geographic coordinates corresponding to each angle combination, generating a forward mapping file (PT2Geo.npy). Both files are pre-loaded into memory. When the command center receives a fire report (such as a PT angle of (125.3°, -18.7°)), the system can instantly return its geographic coordinates through a simple array index; conversely, when it is necessary to turn the camera to view a certain coordinate point, the required PT command can be obtained immediately.

[0167] This embodiment successfully transforms a complex, physical model-based nonlinear positioning problem into an efficient table lookup operation through discretization pre-computation and static file storage. This not only reduces the positioning response time from hundreds of milliseconds to sub-milliseconds, meeting the real-time requirements of demanding scenarios such as emergency command, but also significantly reduces the computing power requirements of terminal devices, enabling ordinary mobile phones or tablets to use the positioning function smoothly. This design philosophy of being complex during training but simple during inference is the key to the large-scale application of this application.

[0168] To better demonstrate the advantages of the bidirectional precise localization method for large-scale natural scenes that integrates visual-physical model constraints proposed in this application, such as... Figures 4-6As shown, the following example illustrates a complex application scenario such as a mountainous forestry monitoring area. This area is characterized by steep slopes and dense forests, with buildings and water features. This scenario presents challenges such as limited image features, making matching difficult, uneven feature point distribution, and unavailable building elevation data. Existing matching and localization algorithms are unable to model and locate such complex scenarios.

[0169] The method proposed in this application eliminates the need for camera calibration parameters and limitations on camera arm length and installation precision. Modeling can be completed simply by providing the approximate latitude and longitude of the camera (within a 100-meter error range) and the tower height. The model automatically corrects for position and installation errors. In this case, the pan-tilt camera is fixedly installed on a monitoring tower approximately 28 meters above the ground. Precise positioning of targets such as trees and fire points within the area is required. The positioning accuracy requirement employs a distance-dependent linear error standard, where the error increases linearly with distance. The fitting relationship is: Allowable average error = 0.05 × distance (unit: m).

[0170] Table 1 compares the positioning accuracy of target point localization using existing technologies (i.e., the conventional methods in the table, using matching algorithms + basic matrix filtering) and the positioning method of this application (using matching algorithms + regional filtering + bidirectional training optimization) at different target distances.

[0171] Table 1 - Comparison of Positioning Accuracy

[0172]

[0173] Based on the comparison results of positioning errors and error percentages in the table above, it can be seen that the method of this application is significantly better than the existing technology in terms of positioning accuracy, overfitting control, and matching point retention rate, and is fully adaptable to the precise positioning needs of complex land space scenarios.

[0174] like Figure 2 As shown, this application embodiment provides a global natural large-scale scene bidirectional precise positioning device that integrates visual-physical model constraints. The device may include:

[0175] The image acquisition module 201 is used to acquire panoramic images captured by the pan-tilt camera to be positioned, and to acquire remote sensing images that at least partially cover the geographical area corresponding to the panoramic image.

[0176] Image matching module 202 is used to establish an initial correspondence between panoramic images and remote sensing images and obtain an initial matching point set, wherein each matching point in the initial matching point set is associated with observation angle information determined based on the panoramic image and geographic coordinate information determined based on the remote sensing image.

[0177] The model building module 203 is used to build an initial white-box physical model under a unified world coordinate system. This model is used to characterize the physical constraint relationship of global spatial positioning mapping between geographic coordinates and camera observation position, and is initialized based on camera installation priors. The module also uses the initial matching point set to train the parameters of the initial white-box physical model to obtain an initial positioning model containing gimbal camera installation deviation parameters.

[0178] The iterative optimization module 204 is used to adopt a two-stage training mechanism, iteratively filter matching points through a regional matching point filtering algorithm, and perform alternating optimization training in combination with the initial positioning model to obtain the optimized positioning model.

[0179] The elevation disturbance fine-tuning module 205 is used to perform elevation disturbance fine-tuning on the subset of matching points finally obtained by iterative screening, so as to compensate for the low elevation error of the digital elevation model caused by the occlusion of ground objects, and to perform final fine-tuning on the optimized positioning model based on the fine-tuned subset of matching points to obtain the target positioning model for accurate positioning of the gimbal camera.

[0180] The positioning execution module 206 generates a static mapping file based on the target positioning model, and realizes bidirectional positioning of camera observation angle and geographic coordinates through the static mapping file.

[0181] In one embodiment, the model building module 203 is specifically used for:

[0182] Based on the WGS84 ellipsoidal coordinate system, the initial installation position and nominal installation height of the gimbal camera, an initial white-box physical model is constructed. The WGS84 ellipsoidal coordinate system is used as the model input, the theoretical observation angle of the gimbal camera in the unified world coordinate system is used as the model output, and the initial installation position and nominal installation height are used as the initial parameters of the model. The initial white-box physical model is a parameterized model that maps geographic coordinates to the theoretical observation angle of the camera. Its parameters to be optimized represent the actual installation deviation of the camera relative to the installation prior.

[0183] The initial set of matching points is input into the initial white-box physical model, and the model parameters are iteratively trained using the stochastic gradient descent algorithm until the model loss converges, thus obtaining the initial localization model. The model parameters include camera latitude and longitude offset, installation height correction, and installation attitude tilt angle.

[0184] In one embodiment, the two-stage training mechanism includes:

[0185] The first stage of coarse training involves calculating and filtering the regional loss of the initial matching point set using the initial localization model to obtain the first set of filtered matching points. The initial localization model is then retrained using the first set of filtered matching points until the first convergence condition is met, thus obtaining the first stage localization model.

[0186] The second stage of fine training involves using the first-stage localization model to perform regional loss calculation and filtering on the initial set of matching points again, obtaining a second subset of filtered matching points. The first-stage localization model is then retrained using this second subset of filtered matching points until the second convergence condition is met, resulting in an optimized localization model.

[0187] In one embodiment, the regional loss calculation and filtering includes:

[0188] The field of view of the gimbal camera is divided into multiple horizontal sub-regions at equal intervals in the horizontal direction, and into multiple vertical sub-regions in the vertical direction according to the pitch angle distribution characteristics using a non-linear transformation method.

[0189] For each matching point, calculate the loss between its actual observed angle and the angle predicted by the current positioning model;

[0190] Calculate the mean loss for each sub-region based on the loss of the matching points within each sub-region;

[0191] Matching points whose losses are less than the average loss of their respective sub-regions and less than a preset multiple of the global average loss are retained, forming a filtered subset of matching points.

[0192] In one embodiment, in the step of dividing the vertical direction into multiple vertical sub-regions using a nonlinear transformation based on the pitch angle distribution characteristics, a quadratic curve transformation is used for the division, so that the division density is higher in regions with larger absolute pitch angle values ​​than in regions with smaller absolute pitch angle values.

[0193] In one embodiment, when the elevation perturbation fine-tuning module 205 performs elevation perturbation fine-tuning on the subset of matching points finally obtained from iterative screening, it is specifically used for:

[0194] Fix all parameters of the optimized positioning model except for elevation;

[0195] For each matching point in the subset of matching points, the optimal elevation disturbance that minimizes the model prediction loss is solved in reverse.

[0196] The original elevation data of each matching point is corrected by using the optimal elevation disturbance, resulting in a subset of matching points with fine-tuned elevation.

[0197] In one embodiment, when the positioning execution module 206 generates a static mapping file based on the target positioning model, it specifically performs the following:

[0198] Based on the target localization model, the mapping relationship between the geographic coordinates within the target area and the optimal observation angle of the camera is discretized and calculated to generate a reverse mapping file;

[0199] Based on the target localization model, the mapping relationship between camera observation angle and geographic coordinates is discretized and calculated to generate a forward mapping file;

[0200] The reverse mapping file and the forward mapping file are stored as two static files that support fast loading and indexing, respectively, in an array.

[0201] like Figure 3 As shown, this application provides an electronic device 3, which includes a memory 301, a processor 302, and a computer program 303 stored in the memory 301 and executable on the processor 302. The memory 301 and the processor 302 are connected via a bus 304. When the processor 302 executes the computer program 303, it implements the global natural large scene bidirectional precise positioning method with fusion of visual physical model constraints as described in the above method embodiment of this application.

[0202] Specifically, the electronic device 3 can be an intelligent device with memory and processor, such as an industrial control computer, PC, or smart mobile terminal, or a computer component with memory and processor, such as a CPU or GPU.

[0203] In this embodiment, electronic device 3 is a remote monitoring terminal such as a PC or smartphone.

[0204] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the bidirectional precise localization method for a large-scale natural scene with fusion of visual-physical model constraints as described in the above-described method embodiments of this application.

[0205] It should be noted that the global natural large scene bidirectional precise positioning device, electronic device and computer-readable storage medium that integrate visual physical model constraints in the above embodiments have the same working principle and technical effect as the global natural large scene bidirectional precise positioning method that integrates visual physical model constraints in the above embodiments, and will not be repeated here.

[0206] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0207] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0208] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly using hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0209] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for bidirectional precise localization of a large-scale natural scene across the entire domain, incorporating visual-physical model constraints, characterized in that: The method includes: S1, acquire a panoramic image captured by the camera of the pan-tilt unit to be positioned, and acquire a remote sensing image that at least partially covers the geographical area corresponding to the panoramic image; S2, by cross-modal feature matching, an initial correspondence between the panoramic image and the remote sensing image is established to obtain an initial matching point set, wherein each matching point in the initial matching point set is associated with observation angle information determined based on the panoramic image and geographic coordinate information determined based on the remote sensing image; S3. Construct an initial white-box physical model under a unified world coordinate system. This model is used to characterize the physical constraint relationship of global spatial positioning mapping between geographic coordinates and camera observation position, and is initialized based on camera installation priors. Then, use the initial matching point set to train the parameters of the initial white-box physical model to obtain an initial positioning model containing gimbal camera installation deviation parameters. S4. A two-stage training mechanism is adopted. The matching points are iteratively filtered through a regional matching point filtering algorithm, and the initial positioning model is combined with the initial positioning model for alternating optimization training to obtain the optimized positioning model. S5, the elevation perturbation of the matching point subset finally obtained by iterative screening is fine-tuned to compensate for the low elevation error of the digital elevation model caused by the obstruction of ground objects, and the optimized positioning model is finally fine-tuned based on the fine-tuned matching point subset to obtain the target positioning model for accurate positioning of the gimbal camera. S6. Generate a static mapping file based on the target positioning model, and realize bidirectional positioning of camera observation angle and geographic coordinates through the static mapping file; In step S4, the two-stage training mechanism includes: The first stage of coarse training involves using the initial localization model to perform regional loss calculation and filtering on the initial matching point set to obtain a first filtered matching point subset. The initial localization model is then retrained using the first filtered matching point subset until the first convergence condition is met, thus obtaining the first stage localization model. The second stage of fine training involves using the first stage localization model to perform regional loss calculation and filtering on the initial matching point set again, obtaining a second set of filtered matching points, and then using the second set of filtered matching points to retrain the first stage localization model until the second convergence condition is met, thus obtaining the optimized localization model.

2. The method for bidirectional precise localization of a large-scale natural scene with integrated visual-physical model constraints as described in claim 1, characterized in that, Step S3 includes: Based on the WGS84 ellipsoidal coordinate system, the initial installation position and nominal installation height of the gimbal camera, an initial white-box physical model is constructed. The WGS84 ellipsoidal coordinate system serves as the model input, the theoretical observation angle of the gimbal camera in a unified world coordinate system serves as the model output, and the initial installation position and nominal installation height serve as the model's initial parameters. The initial white-box physical model is a parameterized model that maps geographic coordinates to the camera's theoretical observation angle, and its parameters to be optimized characterize the actual installation deviation of the camera relative to the installation prior. The initial set of matching points is input into the initial white-box physical model, and the model parameters are iteratively trained using the stochastic gradient descent algorithm until the model loss converges to obtain the initial positioning model. The model parameters include camera latitude and longitude offset, installation height correction, and installation attitude tilt angle.

3. The method for bidirectional precise localization of a large-scale natural scene with integrated visual-physical model constraints as described in claim 1, characterized in that, The regional loss calculation and filtering includes: The field of view of the gimbal camera is divided into multiple horizontal sub-regions at equal intervals in the horizontal direction, and into multiple vertical sub-regions in the vertical direction according to the pitch angle distribution characteristics using a nonlinear transformation method. For each matching point, calculate the loss between its actual observed angle and the angle predicted by the current positioning model; Calculate the mean loss for each sub-region based on the loss of the matching points within each sub-region; Matching points whose losses are less than the average loss of their respective sub-regions and less than a preset multiple of the global average loss are retained, forming a filtered subset of matching points.

4. The method for bidirectional precise localization of a large-scale natural scene with integrated visual-physical model constraints as described in claim 3, characterized in that, In the step of dividing the vertical direction into multiple vertical sub-regions using a nonlinear transformation method based on the pitch angle distribution characteristics, a quadratic curve transformation is used for the division, so that the division density is higher in regions with larger absolute pitch angle values ​​than in regions with smaller absolute pitch angle values.

5. The method for bidirectional precise localization of a large-scale natural scene with integrated visual-physical model constraints as described in claim 1, characterized in that, In step S5, the elevation perturbation fine-tuning of the subset of matching points finally obtained by iterative screening includes: Fix all parameters of the optimized positioning model except for elevation; For each matching point in the subset of matching points, the optimal elevation disturbance amount that minimizes the model prediction loss is solved in reverse. The original elevation data of each matching point is corrected using the optimal elevation disturbance to obtain a subset of matching points with fine-tuned elevation.

6. The method for bidirectional precise localization of a large-scale natural scene with integrated visual-physical model constraints as described in claim 1, characterized in that, In step S6, generating a static mapping file based on the target localization model includes: Based on the target localization model, the mapping relationship between the geographic coordinates within the target area and the optimal observation angle of the camera is discretized and calculated to generate a reverse mapping file; Based on the target positioning model, the mapping relationship between the camera observation angle and geographic coordinates is discretized and calculated to generate a forward mapping file; The reverse mapping file and the forward mapping file are stored in an array as two structured static files that support fast loading and indexing.

7. A bidirectional precise positioning device for a large-scale natural scene integrating visual-physical model constraints, characterized in that, The device includes: The image acquisition module is used to acquire panoramic images captured by the camera of the pan-tilt unit to be positioned, and to acquire remote sensing images that at least partially cover the geographical area corresponding to the panoramic image. The image matching module is used to establish an initial correspondence between the panoramic image and the remote sensing image and obtain an initial matching point set, wherein each matching point in the initial matching point set is associated with observation angle information determined based on the panoramic image and geographic coordinate information determined based on the remote sensing image. The model building module is used to construct an initial white-box physical model under a unified world coordinate system. This model is used to characterize the global spatial positioning mapping physical constraint relationship between geographic coordinates and camera observation positions, and is initialized based on camera installation priors. The module also uses the initial matching point set to train the parameters of the initial white-box physical model to obtain an initial positioning model containing gimbal camera installation deviation parameters. The iterative optimization module is used to employ a two-stage training mechanism, iteratively filtering matching points through a regional matching point filtering algorithm, and alternately optimizing and training the initial positioning model to obtain the optimized positioning model. The elevation perturbation fine-tuning module is used to perform elevation perturbation fine-tuning on the subset of matching points finally obtained by iterative screening, and to perform final fine-tuning on the optimized positioning model based on the fine-tuned subset of matching points to obtain a target positioning model for precise positioning of the gimbal camera. The positioning execution module generates a static mapping file based on the target positioning model, and uses the static mapping file to achieve bidirectional positioning of camera observation angle and geographic coordinates; The two-stage training mechanism includes: The first stage of coarse training involves using the initial localization model to perform regional loss calculation and filtering on the initial matching point set to obtain a first filtered matching point subset. The initial localization model is then retrained using the first filtered matching point subset until the first convergence condition is met, thus obtaining the first stage localization model. The second stage of fine training involves using the first stage localization model to perform regional loss calculation and filtering on the initial matching point set again, obtaining a second set of filtered matching points, and then using the second set of filtered matching points to retrain the first stage localization model until the second convergence condition is met, thus obtaining the optimized localization model.

8. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the global natural large-scale scene bidirectional precise localization method that integrates visual physical model constraints as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the bidirectional precise localization method for large-scale natural scenes with fusion of visual-physical model constraints as described in any one of claims 1-6.

Images