A method for fusing a real photo with a survey digital twin scene

Abstract

The present application relates to the field of engineering surveying technology, and particularly relates to a method for fusing a real scene photo with a surveying digital twin scene, comprising: inputting a photo and extracting content and EXIF information, and taking a blurred GPS or a collected point to estimate a position as an initial reference when there is no effective position; constructing a sphere, an ellipsoid and a cylinder standard projection container in a twin model; rendering a panoramic projection graph and extracting an edge graph and a depth graph; preliminarily determining a camera position through a total error matrix formed by a structural similarity and a depth residual mean square deviation; selecting an optimal container to precisely determine the position through a nonlinear optimization, and determining a final fusion model by means of an interactive interface fine adjustment; realizing bidirectional mapping of photo notes and model objects; and the present application can directly process discrete surveying photos taken by non-measuring equipment such as mobile phones and ordinary cameras without complete camera internal participation and distortion parameters, and activates the value of historical surveying data through bidirectional labeling mapping, thereby providing support for multi-source data fusion and intelligent analysis of an engineering surveying digital twin scene.

Description

Technical Field

[0001] This invention belongs to the field of engineering surveying technology, specifically relating to a method for fusing real-scene photographs with surveying digital twin scenes. Background Technology

[0002] In fields such as autonomous driving, photogrammetry, and robot vision, image modeling technology has become relatively mature. It typically uses algorithms such as Structure from Motion (SfM), SLAM, and Multi-View Stereo (MVS) to reconstruct the 3D structure of a scene and calculate camera pose based on continuously captured image sequences, video streams, or highly overlapping photographs. However, such technologies have inherent dependencies: they require continuous shooting paths or dense image acquisition, shooting equipment with accurate pose / trajectory / depth perception capabilities, and complete imaging parameters (such as focal length, aperture, and orientation), making them difficult to adapt to unstructured shooting scenarios.

[0003] In the field of engineering surveying, although methods such as laser scanning and UAV oblique photography can quickly construct regional terrain models or 3D digital twin scenes, the photo data taken by professionals using mobile phones, ordinary cameras and other non-surveying equipment in daily field surveys has significant unique characteristics: the variety of shooting equipment leads to inconsistent data standards, and most photos lack pose records; the shooting points are scattered, and the overlap between images is low, which cannot meet the requirements of traditional modeling techniques for image sequences; the shooting content is mostly professional objects such as geological outcrops, profiles and structural features, which lack structured labels and are difficult to directly locate in the 3D twin scene.

[0004] Existing methods for fusing real-world photographs with digital twin scenes (such as panoramic image projection, video texture mapping, and GIS layer overlay) generally require the input image to have complete pose information or pre-set structured labels, which cannot be adapted to the aforementioned disordered survey photographs. Furthermore, the lack of a precise geometric alignment mechanism between image content and the twin model results in low fusion accuracy and an inability to achieve information linkage between the image and the model. This severely limits the usability and interactivity of survey photographs in digital twin scenes and makes it difficult to reuse a large number of historical survey photographs without location information, thus hindering the expansion of data sources for the construction of digital twins in engineering projects.

[0005] In view of this, the present invention is hereby proposed. Summary of the Invention

[0006] To address the aforementioned technical problems in the existing technology, this invention provides a method for fusing real-scene photographs with survey digital twin scenes, solving the problem that existing image modeling and fusion methods are unable to handle discrete, discontinuous, and weakly localized photographs.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] A method for fusing real-world photographs with surveyed digital twin scenes includes:

[0009] S1. Read the image content of the real scene photo and the available information in the EXIF. If there is no valid pose information in the EXIF, extract the fuzzy GPS information of the photo or the estimated position of the acquisition point as the initial position reference.

[0010] S2. In the digital twin 3D model, based on the initial position reference, a set of standard image projection containers are constructed with the initial position as the center. The standard image projection containers include spheres, ellipsoids and cylinders, which are used to carry the geometric projection of the image.

[0011] S3. Based on the standard image projection container and the initial pose, render a panoramic projection map of the corresponding viewpoint from the digital twin 3D model. The panoramic projection map includes an edge map and a depth map.

[0012] S4. Extract the edge map and depth map of the real scene photo, and calculate the error between the extraction result and the edge map and depth map of the panoramic projection image; the error calculation adopts an error function including structural similarity and depth residual mean square error, the two error functions are fused to form a total error matrix, and the camera pose of the real scene photo is initially determined based on the total error matrix;

[0013] S5. Based on the initially determined camera pose and the set of standard image projection containers, select the best-fitting standard image projection container; adjust the camera pose through nonlinear optimization, and under the condition of the selected best-fitting standard image projection container, further accurately determine the camera pose near the initially determined camera pose and evaluate it.

[0014] S6. Based on the further precise camera pose, the system provides an interactive interface to allow users to adjust the position, angle and transparency of the real-world photo in three-dimensional space in order to determine the final pose and fusion model.

[0015] S7. Based on the final pose and fusion model, the annotation information in the real-world photo is synchronously mapped to the digital twin 3D model, or the object is mapped from the digital twin 3D model to the real-world photo and highlighted, thus constructing a two-way interaction mechanism between the real-world photo and the digital twin 3D model.

[0016] Furthermore, the process of reading the image content of the real-world photograph and the available information in the EXIF ​​includes:

[0017] Parse the EXIF ​​information of the photo to extract usable shooting parameters, including: focal length, principal point coordinates, camera model, shooting time, and resolution;

[0018] If the EXIF ​​contains GPS coordinates, use them as the initial fuzzy positioning reference; if there is no GPS information, look up the estimated location in the table according to the photo naming rules, the field collection point number, or the collection records in the imported table.

[0019] If the device provides an orientation angle, it is extracted as well, and used to constrain the range of subsequent candidate viewpoint generation.

[0020] Furthermore, in step S2, constructing a set of standard image projection containers includes:

[0021] In the digital twin 3D scene, the system constructs a variety of standard image mapping containers centered on the position obtained in step S1, including a unit sphere, an ellipsoid with different aspect ratios, and a cylinder with a fixed viewing direction.

[0022] Each container shape is used to simulate different degrees of camera distortion or image projection characteristics, and its spatial parameters are recorded as the basic structure for subsequent error fitting.

[0023] Further, in step S3, the rendering of the panoramic projection map corresponding to the viewpoint includes: the system renders and generates a projection map from the digital twin 3D model with the same position and orientation based on the current candidate camera pose parameters;

[0024] The panoramic projection map includes an edge map and a depth map. The edge map is generated by the Canny algorithm or the Sobel / Laplacian edge detection method, and the depth map is rendered using the vertex depth information in the model.

[0025] Both the edge map and the depth map are aligned with the resolution of the real-world photo in S1 to facilitate subsequent pixel-level error comparison.

[0026] Furthermore, step S4 specifically includes:

[0027] S41. Extract two structural feature maps, edge map and depth map, from the input real-world photo;

[0028] S42. By controlling the scaling factor and rotation angle, the extracted edge map is coarsely matched with the panoramic edge map in step S3, and the candidate pose parameter set is determined by the intersection-union error function.

[0029] S43. Use the depth map residual to further optimize the candidate pose parameter set and select the best error parameter;

[0030] S44. The candidate pose matrix is ​​obtained by back projection calculation, and the camera position of the real scene photo is determined.

[0031] Furthermore, the formula for the depth map residual is:

[0032]

[0033] in, Total number of pixels Scaling factor This is the result of an affine transformation of the depth map from a real-world photograph. This is a segment corresponding to the depth map of the panoramic projection.

[0034] Furthermore, step S5 specifically includes:

[0035] S51. The system constructs an error matrix using edge map error and depth map error among multiple standard image projection containers, and selects the container with the smallest total error as the optimal projection container.

[0036] S52. Under this optimal projection container, the optimal spatial pose parameters are calculated through nonlinear optimization.

[0037] S53. Provides prompts on the type of fitting container and the trend of error changes to help judge the fitting effect.

[0038] Furthermore, the formula for the total error is:

[0039]

[0040] in, To minimize the error of the edge map, To minimize the error of the depth map, .

[0041] Furthermore, step S6 specifically includes:

[0042] The system provides users with a 3D visualization interface that displays a twin fusion model of a real-world photo fitted to the best-fit container. Users can adjust the position, orientation, attachment method, and transparency of the photo. The final confirmed fusion pose and projection state will be saved as the binding parameters of the photo in the digital twin system.

[0043] Furthermore, step S7 specifically includes:

[0044] Users can add structural annotations in the form of points, lines, and surfaces to real-world photos. These structural annotations include geological structure lines, fracture markers, and numbered labels.

[0045] Based on the final pose of the photo, the system maps the structural annotations to the digital twin 3D model space through back projection, generating corresponding 3D labeled objects; it supports clicking on any spatial object in the digital twin 3D model to trace back to the relevant real-world photo, and automatically highlights the corresponding area in the photo, realizing two-way interaction;

[0046] All annotations can be exported as structured annotation data for use in collaborative work platforms or reporting systems.

[0047] Compared with existing technologies, the present invention provides a method for fusing real-world photographs with digital twin scenes of surveying, comprising: inputting photographs and extracting content and EXIF ​​information; using fuzzy GPS or estimated location from acquisition points as an initial reference when no effective pose is available; constructing standard projection containers of spheres, ellipsoids, and cylinders in the twin model; rendering panoramic projection maps and extracting edge maps and depth maps; fusing total error matrices using structural similarity and depth residual mean square error to initially determine camera pose; selecting the optimal container and refining the pose through nonlinear optimization, supplemented by fine-tuning with an interactive interface to determine the final fusion model; and realizing bidirectional mapping between photograph annotations and model objects. This invention does not require complete intrinsic parameters and distortion parameters of the camera, and can directly process discrete survey photographs taken by non-measuring devices such as mobile phones and ordinary cameras. It breaks through the dependence of traditional technologies on continuous shooting and pose integrity, while activating the value of historical survey data through bidirectional annotation mapping, adapting to lightweight operations in the field, and providing support for multi-source data fusion and intelligent analysis of digital twin scenes in engineering surveying. Attached Figure Description

[0048] Figure 1 A flowchart illustrating the method for fusing real-world photographs with surveyed digital twin scenes provided in this embodiment of the invention;

[0049] Figure 2 This is a schematic diagram of constructing a spherical projection container based on an initial pose in a twin model, provided by an embodiment of the present invention.

[0050] Figure 3 Schematic diagrams of three optional standard image projection containers provided for embodiments of the present invention;

[0051] Figure 4 This is a schematic diagram illustrating the projection of a twin scene container and the generation of its edge and depth maps, provided in an embodiment of the present invention.

[0052] Figure 5 This is a schematic diagram illustrating the generation of edge and depth maps from real-world photographs provided in an embodiment of the present invention.

[0053] Figure 6 This is a schematic diagram illustrating the edge map and depth map registration and pose calculation optimization provided in an embodiment of the present invention. Detailed Implementation

[0054] The technical solution of the present invention will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are not all embodiments of the present invention. All other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0055] It should be noted that, unless otherwise specifically stated, the relative arrangement and numerical expressions of the components and steps described in these embodiments should not be construed as limiting the scope of the invention.

[0056] The following description of exemplary embodiments is merely illustrative and is not intended to limit the invention or its application or use in any way. Techniques, methods, and apparatus known to those skilled in the art may not be discussed in detail herein, but where applicable, such techniques, methods, and apparatus should be considered part of this specification.

[0057] Example 1

[0058] See Figure 1 , Figure 1 This is a flowchart of a method for fusing real-world photographs with surveyed digital twin scenes, as proposed in this invention. Specific steps may include:

[0059] S1. Real-world photo input and information acquisition: Read the image content and available information from the EXIF ​​data of the real-world photo. If there is no valid pose information in the EXIF ​​data, extract the fuzzy GPS information or estimated location of the acquisition point from the photo as the initial location reference. The process of reading the image content and available information from the EXIF ​​data of the real-world photo includes:

[0060] S11. Parse the EXIF ​​information of the photo to extract usable shooting parameters, including: focal length, principal point coordinates, camera model, shooting time and resolution;

[0061] S12. If the EXIF ​​contains GPS coordinates, use them as the initial fuzzy positioning reference; if there is no GPS information, look up the estimated location in the table according to the photo naming rules, the field collection point number, or the collection records in the imported table.

[0062] S13. If the device provides an orientation angle, extract it as well, and use it to constrain the range of subsequent candidate viewpoint generation.

[0063] Specifically, the photos can be taken from mobile phones, portable cameras, panoramic cameras, or other field acquisition devices, and their content typically includes geological outcrops, profiles, and structural features. The system first parses the photo files and extracts their EXIF ​​metadata, including but not limited to: shooting time, shooting device model, focal length information, image size, GPS location information, and orientation information (such as pitch and yaw angles recorded by a gyroscope).

[0064] If the photo contains GPS coordinates (e.g., taken with a mobile phone), it is used as a fuzzy location point to form an initial position estimate. If the photo has no location information but has a number and naming rules, the system can find the corresponding collection point coordinates by matching them with the field point database, map sampling point table, etc., to form the estimated position. If no available location information is available, the user can manually specify an initial reference position in the 3D scene.

[0065] In addition, if the camera orientation (such as tilt angle or azimuth angle) is provided in the EXIF ​​or auxiliary sensor data, it is used for subsequent candidate view range constraints to reduce computational space.

[0066] S2. Establishment of a standard projection body for a twin scene: See [link / reference] Figure 2 In the digital twin 3D model, based on the initial position reference, a set of standard image projection containers are constructed with the initial position as the center; the standard image projection containers include spheres, ellipsoids, and cylinders, used to carry the geometric projection of the image; the process of reading the image content of the real-world photograph and the available information in the EXIF ​​includes:

[0067] S21. Parse the EXIF ​​information of the photo to extract usable shooting parameters, including: focal length, principal point coordinates, camera model, shooting time and resolution;

[0068] S22. If the EXIF ​​contains GPS coordinates, use them as the initial fuzzy positioning reference; if there is no GPS information, look up the estimated location in the table according to the photo naming rules, the field collection point number, or the collection records in the imported table.

[0069] S23. If the device provides an orientation angle, extract it as well, and use it to constrain the range of subsequent candidate viewpoint generation.

[0070] Specifically, for each photograph to be processed, multiple standard geometric projection bodies are pre-set at its estimated location (obtained in step S1) to serve as projection containers for the photograph image. See [link / reference] Figure 3 Based on the initial position coordinates of each photograph, the system constructs the following three types of standard projection bodies:

[0071] Sphere projection: With the estimated position as the center of the sphere, a fixed radius is set. It is suitable for image models with no or minimal distortion.

[0072] Ellipsoid projection: Based on a sphere, a major-minor axis ratio (e.g., 1.2 to 1.5) is introduced. It is suitable for photographs with moderate radial distortion. By changing the axial length, it simulates the effect of outward radial stretching or compression of the image.

[0073] Cylindrical projection: A cylindrical projection surface with a fixed radius is established in the horizontal direction with the estimated position as the center. It is suitable for image models with a wide horizontal field of view but small vertical distortion (such as local parts of drone scan images or panoramic images).

[0074] In actual engineering deployments, if the system does not depend on the shooting equipment parameters, the following default container parameters can be used uniformly:

[0075] Sphere container: The center of the sphere is located at the initial estimated position of the photo, and the radius is... ;

[0076] Ellipsoidal container: with the same sphere center as the major axis. short axis The vertical axis is the default. The short axis direction is consistent with the shooting direction by default;

[0077] Cylindrical containers: with the same position as the axis, the radius... The height is The axis is perpendicular to the ground.

[0078] The above parameters are empirically set values, balancing the visible range of image content with computational stability, and are suitable for most field survey scenarios. The system will automatically select the optimal container type and correct the container parameters during the subsequent error comparison stage.

[0079] The three types of projectors mentioned above will be used in subsequent steps to perform parallel projection mapping on the same photographic image and to conduct error comparison analysis with the projection map of the digital twin model, so as to select the "best container" that is most suitable for the structural characteristics of the image as the basis for its spatial geometric fitting.

[0080] It should be noted that the goal of this step is not to reproduce the actual optical distortion model at the time of photograph capture, but rather to construct a set of standardized spatial envelopes that can be used to simulate the geometric shape of images, facilitating error-driven fitting and structural alignment. This invention, by pre-setting multiple standard projection body shapes, can significantly improve the system's adaptability and fitting accuracy for photographs taken by different devices, and is particularly suitable for non-measurement photographs.

[0081] In practice, the three projection bodies mentioned above can be constructed and cached simultaneously for use in subsequent steps; or they can be dynamically called and generated when candidate pose samples are generated.

[0082] S3. Generation and structural diagram extraction of the twin scene panoramic projection map: based on the standard image projection container and initial pose; see reference Figure 4 A panoramic projection map corresponding to the viewpoint is generated by rendering from a digital twin 3D model. The panoramic projection map includes an edge map and a depth map.

[0083] S31. Panoramic image generation driven by projection container: Rendering and generating a panoramic projection image of the corresponding viewpoint includes the system rendering and generating a projection image from the digital twin 3D model with the same position and orientation based on the current candidate camera pose parameters.

[0084] Specifically, in step S2, the system constructs multiple standard projection containers for each real-scene photograph. , representing spheres, ellipsoids, and cylinders, respectively. In this step, the system renders the corresponding panoramic image based on each type of projection container c. And establish the mapping relationship between image pixel coordinates and three-dimensional direction vectors.

[0085] Unfolded image resolution settings (default):

[0086] Sphere, ellipsoid: ,cover ;

[0087] Cylinder: Covering the horizontal direction The height is set according to the container's proportions.

[0088] Definition of coordinates for the unfolded graph:

[0089] : Horizontal pixel position;

[0090] Vertical pixel position;

[0091] coordinate With three-dimensional direction vector There is an invertible mapping relationship between them.

[0092] To support subsequent structural diagram registration and pose inversion, it is necessary to explain the unfolded coordinate definition and orientation mapping method for each type of container:

[0093] S311, sphere container:

[0094] Unfolding method: equirectangular projection; pixel coordinate definition:

[0095] : Corresponding yaw angle (horizontal view) ;

[0096] : Corresponding pitch angle (vertical view) ;

[0097] Angle calculation formula:

[0098]

[0099] Unit spherical direction vector:

[0100]

[0101] Applications: Suitable for wide-angle images with almost no distortion.

[0102] S312, Ellipsoidal Container: Based on a sphere, the ratio of major and minor axes is varied to simulate distortion caused by stretching or compression;

[0103] Unfolding method and angle calculation: same as for spherical containers;

[0104] Weighted direction vector (before normalization):

[0105]

[0106] Where a, b, and c are the lengths of the three axes of the ellipsoid, respectively;

[0107] Final unit direction vector:

[0108]

[0109] Scope of application: Suitable for photos with a certain degree of radial distortion, such as edge stretching and compression.

[0110] S313, cylindrical container:

[0111] Development method: Generate the unfolded diagram using the circumference and height of the cylinder as coordinate axes;

[0112] Coordinate definition:

[0113] : Corresponding circumferential angle ;

[0114] The linear mapping corresponding to the height direction of the cylinder (along the vertical axis) is as follows:

[0115]

[0116] Direction vector expression:

[0117]

[0118] Where r is the radius of the cylinder, used to convert the vertical height into a unit directional quantity.

[0119] Scope of application: Suitable for images with a wide horizontal field of view but minimal vertical distortion, such as those captured by drones in horizontal shooting mode or by mobile phone in horizontal scanning mode.

[0120] S32. Mapping Relationship Caching and Structure Graph Management: The panoramic projection map includes an edge map and a depth map. The edge map is generated by the Canny algorithm or the Sobel / Laplacian edge detection method, and the depth map is rendered using the vertex depth information in the model. Both the edge map and the depth map are aligned with the resolution of the real-world photos in S1 to facilitate subsequent pixel-level error comparison.

[0121] Specifically, the system supports image structure graph matching in step S4 and multi-container error optimization in step S6. It caches the unfolded and structure graph results generated for each container type and records the following data: unfolded graph size and resolution, unfolded graph and orientation vector mapping function, and images for each channel. The projection container type and initial parameters (such as radius, axis length, etc.) are required. This information will be repeatedly used in subsequent steps to avoid redundant calculations.

[0122] S4. Image Registration and Error Calculation: See [link / reference] Figure 5 The edge map and depth map of the real-scene photo are extracted, and the error between the extraction results and the edge map and depth map of the panoramic projection image is calculated; see reference. Figure 6 Error calculation adopts an error function that includes structural similarity and depth residual mean square error. The two error functions are fused to form a total error matrix, and the camera pose of the real scene photo is initially determined based on the total error matrix.

[0123] Specifically, by performing pixel-level registration between the structural diagram of the real-world photograph and the structural diagram in the twin scene projection unfolded diagram, the optimal matching position, rotation angle, and scaling factor of the photograph in the unfolded diagram are determined, and its camera pose in three-dimensional space (including rotation matrix and position vector) is further derived. For this purpose, homogeneous coordinate transformation in Euclidean space is uniformly used to represent the spatial pose of the camera.

[0124] The system uniformly adopts the homogeneous coordinate transformation form in Euclidean space to represent the spatial pose P of the camera:

[0125] Rotation matrix : Indicates the camera orientation, generated by Euler angles in the order ZYX (roll-pitch-yaw);

[0126] Translation vector : Indicates the camera's position in the world coordinate system;

[0127] The two are combined to form an extrinsic parameter matrix. This is used for spatial point projection; if homogeneous coordinates are used, the complete pose matrix is:

[0128]

[0129] To improve the stability and accuracy of registration, this step adopts a "two-stage structure map collaborative registration strategy", which utilizes the sensitive characteristics of the edge map and the depth map to complete coarse matching and fine optimization in sequence.

[0130] S41. Input Image Structure Map Extraction: Extract two types of structural feature maps, edge map and depth map, from the input real-world photo; for the input image... The system extracts its structural feature map, including:

[0131] S411, Edge Map Extract salient structural lines from an image using edge detection algorithms (such as Canny, Sobel, or deep learning methods);

[0132] S412, Depth Map Generate relative depth maps using monocular depth estimation networks (such as MiDaS).

[0133] The above images are uniformly processed into single-channel grayscale images, and can be normalized or sized as needed.

[0134] S42. Image Registration: By controlling the scaling factor and rotation angle, the extracted edge map is coarsely matched with the panoramic edge map in step S3. The candidate pose parameter set is determined by the intersection-union error function. The approximate position, orientation, and scaling factor of the image are located in the unfolded image structure map as initial values ​​for subsequent optimization. Specifically, this includes:

[0135] Parameter settings: Scaling factor set: Rotation angle set: ;

[0136] Matching process: For each candidate combination ,Will Perform scaling and rotation (affine transformation) to obtain ;

[0137] In the unfolded structural diagram Perform a sliding window comparison and crop the fragment to obtain... ;

[0138] Calculate the edge map matching error (e.g., using the IoU method):

[0139]

[0140] The error function is defined as:

[0141]

[0142] To reduce the impact of local extrema on error optimization, the system adopts a Top-K candidate mechanism, which retains the top k candidate parameters with the smallest errors. As a set of candidate pose parameters.

[0143] S43. Image Matching: The depth map residuals are used to further optimize the candidate pose parameter set, selecting the optimal error parameters. Based on the determined candidate positions and angles, the scaling factor S is refined and optimized using the depth map to improve scale consistency. Specifically, this includes:

[0144] Fixed Candidates Continuously adjust S to Affine transformation ,from Cut the corresponding segment to get Calculate the pixel-level residuals and select the one with the smallest error. The original parameters are combined to form the final matching result. The specific formula is as follows:

[0145]

[0146] in, Total number of pixels Scaling factor This is the result of an affine transformation of the depth map from a real-world photograph. This is a segment corresponding to the depth map of the panoramic projection.

[0147] S44. Pose Backprojection Calculation: The candidate pose matrix is ​​obtained through backprojection calculation to determine the camera position in the real-world photograph. The system will then expand the image to find the best matching point. Mapping back to the surface of the projected container and calculating the camera pose of the photograph in the 3D scene based on the container type, specifically including:

[0148] S441. Spatial direction vector calculation: Using the cached mapping relationship from step S3, Convert to direction vector If a spherical container is used, then:

[0149]

[0150]

[0151] S442, Rotation Matrix R Construction: Set the camera's -Z axis orientation Construct a right-handed coordinate system and form a rotation matrix. If the image has a matching angle Then, adjust around the line of sight axis in the local coordinate system.

[0152] S443, Position Vector T: Defaults to the coordinates of the center point of the current projection container; can be further adjusted in subsequent S6 optimizations. Final Camera Pose Matrix: As the initial pose of the image in the 3D twin scene.

[0153] S5. Error-driven joint optimization of pose and container: Based on the initially determined camera pose and the set of standard image projection containers, the best-fitting standard image projection container is selected; the camera pose is adjusted through nonlinear optimization, and under the condition of the selected best-fitting standard image projection container, the camera pose is further accurately determined and evaluated near the initially determined camera pose; the system obtains the initial registration parameters between each real-scene image and the twin scene through structural diagram comparison. And further backproject to obtain its corresponding initial camera pose. This step is performed on multiple standard projection containers. Select the best-fit container ; near the current position, for pose Joint optimization is performed with potential image distortion trends. Specific steps include:

[0154] S51. The system constructs an error matrix using edge map error and depth map error among multiple standard image projection containers, and selects the container with the smallest total error as the optimal projection container; specifically including:

[0155] For the current image, the system has constructed its own unfolded graph structure atlas in three containers (sphere, ellipsoid, cylinder):

[0156]

[0157] The system performs a structure diagram comparison in each container (the matching process in S4 can be reused) and records the corresponding minimum total error:

[0158]

[0159] To balance edge map errors Error with depth map The system introduces weighting coefficients to determine the influence weights across different image types. A combined solution is performed. Among them, The default configuration is:

[0160]

[0161] This weight configuration balances the localization accuracy of the edge map with the scale consistency of the depth map, making it suitable for most field photography scenarios. For scenarios lacking depth information or with blurred image edges, the system allows users to customize weights or employ the following adaptive strategy:

[0162]

[0163] in This represents the pixel standard deviation (or sparsity) of the current depth map, reflecting its reliability. Let's take:

[0164]

[0165] That is, select the projection container that has the best fitting effect in the error space.

[0166] S52. Under this optimal projection container, the optimal spatial pose parameters are calculated through nonlinear optimization; after selecting the optimal container... Then, the system calculates the spatial pose parameters of the image in the container coordinate system. Local nonlinear optimization is performed to further reduce the structure diagram error and improve the registration accuracy.

[0167] The system uses the initial estimated pose obtained in step S4. To optimize the starting point, a target error function is constructed:

[0168]

[0169] During the optimization process, the system completes the iteration through the following strategies:

[0170] S521, Fixed Scaling Factor Adjust the camera rotation matrix R within a local window;

[0171] S522. Fine-tune the position vector T near the center of the container to match the actual spatial projection effect;

[0172] S523. After each iteration, convergence is judged based on the decrease in error. The maximum number of iterations is N (default N=10).

[0173] S524. If the error decrease is lower than the set threshold (e.g.) If the contract is terminated early, then the contract will be terminated early.

[0174] The final result This refers to the optimal spatial pose parameters of the image under the best projection container.

[0175] S53. The system provides prompts regarding the type of fitted container and the trend of error changes to assist in judging the fitting effect. Based on the above error analysis and matching results, the system provides the following auxiliary prompts to the user:

[0176] S531, Fitting Container Recommendation: The system will... The container is used as the current fitting container, and it will be used by default to generate the structure diagram expansion in subsequent steps.

[0177] S532. Qualitative error trend prompts: If the error of the spherical container is significantly greater than that of the ellipsoidal or cylindrical container, the system prompts that the image may have radial distortion or warping; if the error is concentrated in the edge area of ​​the image, the system prompts that there may be lens stretching or focal length inconsistency; if the optimal container is an ellipsoid and the major axis direction is consistent with the image rotation direction, the system prompts that the image has edge stretching distortion.

[0178] S6. Manual Fine-tuning and Fusion Optimization: Based on the further precisely determined camera pose, the system provides an interactive interface that allows users to adjust the position, angle, and transparency of the real-world photo in three-dimensional space to determine the final pose and fusion model.

[0179] Specifically, an interactive visual correction mechanism is provided for photos with errors or uncertainties in automatic registration. This allows professional users to intuitively adjust the spatial position, orientation, and container fitting shape of the image within the digital twin 3D scene, thereby further improving registration accuracy and fusion results. This is based on the optimal pose matrix obtained in step S5. With the selected container type The system automatically completes the initial projection loading and provides a visual adjustment interface.

[0180] S61. User Interface Initialization: The system loads the current photo and its automatically matched pose. According to the selected container (Sphere, ellipsoid, or cylinder) The photo texture is attached to the inner surface of the projector and displayed in a semi-transparent overlay. The 3D twin model displays auxiliary visual information such as the spatial location of the photo, shooting direction, and field of view; it can also overlay structural reference layers such as model edge maps and depth shadow maps to help users judge alignment accuracy.

[0181] S62. Manual Interactive Fine-Tuning: Users can fine-tune the following parameters using mouse operations, sliders, knobs, and other controls in the graphical interface:

[0182]

[0183] Table 1

[0184] S63. Fine-tuning Confirmation and Fusion Processing: After the user confirms the fine-tuning operation, the system records the adjusted pose matrix. and corresponding containers The image is embedded into the digital twin scene in its final pose and container form, forming a stable and reusable image-model fusion node; it supports subsequent image annotation, structure recognition, attribute association and other functional modules to call the current alignment information; if the user does not perform manual fine-tuning, the system's automatic result is used by default. As the final parameter.

[0185] The system provides users with a 3D visualization interface that displays a twin fusion model of a real-world photo fitted to the best-fit container. Users can adjust the position, orientation, attachment method, and transparency of the photo. The final confirmed fusion pose and projection state will be saved as the binding parameters of the photo in the digital twin system.

[0186] S7. Interactive annotation and synchronous mapping: After the real-world photos have been accurately embedded into the digital twin 3D scene, a two-way mapping relationship between the image and the model is established, allowing users to perform annotations on either side and synchronize them to the other space in real time, realizing the linkage and sharing of image and model data and geometric collaboration.

[0187] Specifically, based on the final pose and fusion model, annotation information in the real-world photograph is synchronously mapped to the digital twin 3D model, or objects are mapped from the digital twin 3D model to the real-world photograph and highlighted, thus constructing a two-way interaction mechanism between the real-world photograph and the digital twin 3D model. This includes:

[0188] S71. Photo annotations are synchronized with 3D scenes: Users can add structural annotations in the form of points, lines, and surfaces to real-world photos. These structural annotations include geological structural lines, fissure markers, and numbered labels.

[0189] Specifically, users can directly edit photo images. Clicking, selecting, drawing points / lines / regions, etc., on the surface allows the system to perform back projection based on the current projection container and pose matrix, calculating its corresponding 3D spatial position. The calculation process is as follows:

[0190]

[0191] This method is suitable for application scenarios where exploration results, strata boundaries, joint traces, and structural surface locations are reflected from images to models.

[0192] S72. Highlighting projection of 3D model objects onto images: Based on the final pose of the photo, the system back-projects the structural annotations onto the digital twin 3D model space to generate corresponding 3D labeled objects; it supports clicking on any spatial object in the digital twin 3D model to trace back to the relevant real-world photo, and automatically highlights the corresponding area in the photo, realizing two-way interaction.

[0193] Specifically, users can also click and select any spatial object (such as borehole number, structural surface, boundary line, or measuring point) in the digital twin 3D model, and the system will use the aforementioned camera projection model to map it to the image pixel position. First, the model point coordinates... Transform to camera coordinate system :

[0194]

[0195] in It is the projection function of the corresponding container, which has been defined separately in S3.

[0196] This pixel location will be used to display the highlighted outline, text label, or spatial path of the corresponding element in the model on the original image, thereby enhancing the ability to understand the spatial structure from the image perspective.

[0197] Typical applications include: clicking "ZK18 borehole" in the model will automatically highlight the borehole opening in the image; clicking on a structural surface in the model will select the outline of the structural surface in the image and project relevant annotation information such as "Fault F2".

[0198] S73. Annotation Data Organization and Output Interface: All annotations can be exported as structured annotation data for use in collaborative work platforms or reporting systems.

[0199] Specifically, all interactive annotation information between images and models can form structured data, including annotation type (point, line, surface, text); bidirectional mapping coordinates (image position, 3D position); annotation content and attributes (type, level, remarks, time); and the ID of the associated image and model object.

[0200] The labeled data can be exported to various standard formats, including but not limited to: GeoJSON and Shapefile (for geographic information systems); IFC and Revit nodes (for BIM platforms); HTML reports and data packages (for results display and archiving); and API interfaces (for WebGIS platforms or business systems to call).

[0201] This step establishes a real-time linkage and information mapping mechanism between real-world photos and digital twin models, significantly enhancing the analytical, interactive, and data value of image usage. It is applicable to multi-source image fusion and analysis in fields such as geological exploration, construction inspection, and engineering evaluation.

[0202] In summary, the present invention has the following advantages:

[0203] 1. By using a processing mechanism that does not rely on continuous image sequences and complete pose information, it can directly adapt to discrete and weakly localized survey photos taken by non-measurement equipment. It gets rid of the limitations of traditional technologies such as SfM and SLAM on the continuity and overlap of image sequences, significantly expands the applicability of images in engineering survey scenarios, and enables photos taken by non-professional equipment such as mobile phones and ordinary cameras to effectively participate in twin scene fusion.

[0204] 2. By constructing various standard projection bodies such as spheres, ellipsoids, and cylinders, and combining the two-dimensional error comparison of edge structure similarity and depth residual mean square error, automatic correction and accurate attachment of images in twin models can be achieved without traditional target calibration or camera distortion parameters. This solves the problem of low registration accuracy caused by diverse equipment and missing parameters in survey scenarios, and improves the reliability of fusion alignment.

[0205] 3. By using image back projection mechanism and 3D object mapping algorithm, the spatial mapping of annotation information in real scene photos to digital twin models is realized, as well as the backtracking highlighting of 3D model objects to the pixel positions of photos. This constructs a structured linkage relationship between images and models, providing basic support for cross-dimensional analysis and intelligent association of multi-source data.

[0206] 4. By designing a 3D visualization interactive interface that can be embedded in a web platform or mobile terminal, it supports manual fine-tuning operations such as position translation, angle adjustment, and transparency setting, which makes it easy for engineers to quickly complete image reset and confirmation in the field. While improving the convenience of operation, manual intervention can make up for automatic registration errors and ensure the final fusion quality.

[0207] 5. By associating the shapes of different standard projectors with the results of error functions, an image distortion trend model is adaptively generated, which enhances the system's understanding and reasoning ability regarding image distortion features. This model can serve as a basis for evaluating fitting quality and as a parameter reference for subsequent batch photo correction, thereby improving the system's adaptive performance.

[0208] 6. By re-embedding historical survey photos lacking location information into a modern digital twin system and supporting the export of labeled data to standard formats such as GeoJSON and IFC, the reuse value of historical data is activated, enabling comparative analysis between historical and current scenarios, and providing a key data bridge for engineering projects to transform from traditional records to digital twins.

[0209] The above specific embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to examples, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for fusing real-scene photographs with surveyed digital twin scenes, characterized in that, include: S1. Read the image content of the real scene photo and the available information in the EXIF. If there is no valid pose information in the EXIF, extract the fuzzy GPS information of the photo or the estimated position of the acquisition point as the initial position reference. S2. In the digital twin 3D model, based on the initial position reference, a standard image projection container is constructed with the initial position as the center. The standard image projection container includes a sphere, an ellipsoid, and a cylinder, which are used to carry the geometric projection of the image. S3. Based on the standard image projection container and the initial pose, render a panoramic projection map of the corresponding viewpoint from the digital twin 3D model. The panoramic projection map includes an edge map and a depth map. S4. Extract the edge map and depth map of the real-scene photo, and calculate the error between the extraction results and the edge map and depth map of the panoramic projection image; the error calculation adopts an error function including structural similarity and depth residual mean square error, the two error functions are fused to form a total error matrix, and the camera pose of the real-scene photo is initially determined based on the total error matrix; specifically including: S41. Extract two structural feature maps, edge map and depth map, from the input real-world photo; S42. By controlling the scaling factor and rotation angle, the extracted edge map is coarsely matched with the panoramic edge map in step S3, and the candidate pose parameter set is determined by the intersection-union error function. S43. Further optimize the candidate pose parameter set using the depth map residuals, and select the optimal error parameters; the formula for the depth map residuals is: in, Total number of pixels Scaling factor This is the result of an affine transformation of the depth map from a real-world photograph. This is a segment corresponding to the depth map of the panoramic projection. For container type, These are candidate pose parameters; S44. The candidate pose matrix is ​​obtained by back projection calculation, and the camera position of the real scene photo is determined. S5. Based on the initially determined camera pose and standard image projection container, select the best-fitting standard image projection container; adjust the camera pose through nonlinear optimization, and under the condition of the selected best-fitting standard image projection container, further determine the camera pose near the initially determined camera pose and evaluate it. S6. Based on the further determined camera pose, the system provides an interactive interface that allows users to adjust the position, angle, and transparency of the real-world photo in three-dimensional space in order to determine the final pose and fusion model. S7. Based on the final pose and fusion model, the annotation information in the real-world photo is synchronously mapped to the digital twin 3D model, or the object is mapped from the digital twin 3D model to the real-world photo and highlighted, thus constructing a two-way interaction mechanism between the real-world photo and the digital twin 3D model.

2. The method for fusing real-scene photographs with surveyed digital twin scenes according to claim 1, characterized in that, The process of reading the image content and available information from the EXIF ​​data of a real-world photograph includes: Parse the EXIF ​​information of the photo to extract usable shooting parameters, including: focal length, principal point coordinates, camera model, shooting time, and resolution; If the EXIF ​​contains GPS coordinates, use them as the initial fuzzy positioning reference; if there is no GPS information, look up the estimated location in the table according to the photo naming rules, the field collection point number, or the collection records in the imported table. If the device provides an orientation angle, it is extracted as well, and used to constrain the range of subsequent candidate viewpoint generation.

3. The method for fusing real-scene photographs with surveyed digital twin scenes according to claim 1, characterized in that, In step S2, constructing a set of standard image projection containers includes: In the digital twin 3D scene, the system constructs a variety of standard image mapping containers centered on the position obtained in step S1, including a unit sphere, an ellipsoid with different aspect ratios, and a cylinder with a fixed viewing direction. Each container shape is used to simulate different degrees of camera distortion or image projection characteristics, and its spatial parameters are recorded as the basic structure for subsequent error fitting.

4. The method for fusing real-scene photographs with surveyed digital twin scenes according to claim 1, characterized in that, In step S3, the rendering of the panoramic projection map corresponding to the viewpoint includes: the system renders and generates a projection map from the digital twin 3D model with the same position and orientation based on the current candidate camera pose parameters; The edge map is generated using the Canny algorithm or the Sobel / Laplacian edge detection method, and the depth map is rendered using the vertex depth information in the model. Both the edge map and the depth map are aligned with the resolution of the real-world photo in S1 to facilitate subsequent pixel-level error comparison.

5. The method for fusing real-scene photographs with surveyed digital twin scenes according to claim 1, characterized in that, Step S5 specifically includes: S51. The system constructs an error matrix using edge map error and depth map error among multiple standard image projection containers, and selects the container with the smallest total error as the optimal projection container. S52. Under this optimal projection container, the optimal spatial pose parameters are calculated through nonlinear optimization. S53. Provides prompts on the type of fitting container and the trend of error changes to help judge the fitting effect.

6. The method for fusing real-scene photographs with surveyed digital twin scenes according to claim 5, characterized in that, The formula for the total error is: in, To minimize the error of the edge map, To minimize the error of the depth map, .

7. The method for fusing real-scene photographs with surveyed digital twin scenes according to claim 1, characterized in that, Step S6 specifically includes: The system provides users with a 3D visualization interface that displays a twin fusion model of a real-world photo fitted to the best-fit container. Users can adjust the position, orientation, attachment method, and transparency of the photo. The final confirmed fusion pose and projection state will be saved as the binding parameters of the photo in the digital twin system.

8. The method for fusing real-scene photographs with surveyed digital twin scenes according to claim 1, characterized in that, Step S7 specifically includes: Users can add structural annotations in the form of points, lines, and surfaces to real-world photos. These structural annotations include geological structure lines, fracture markers, and numbered labels. Based on the final pose of the photo, the system maps the structural annotations to the digital twin 3D model space through back projection, generating corresponding 3D labeled objects; it supports clicking on any spatial object in the digital twin 3D model to trace back to the relevant real-world photo, and automatically highlights the corresponding area in the photo, realizing two-way interaction; All annotations can be exported as structured annotation data for use in collaborative work platforms or reporting systems.

Images