Method, system, medium and equipment for panoramic display of underground facilities

Abstract

The invention discloses a method, a system, a medium and equipment for panoramic display of underground facilities, and belongs to the technical field of augmented reality visualization. The method comprises the following steps: real picture data of a real camera in a target area is acquired in real time, and the real picture data of the real camera are acquired based on pose data of the real camera; synchronously adjusting the pose state of a virtual camera configured in the fused three-dimensional model; wherein the fused three-dimensional model is obtained by performing space alignment based on an underground facility three-dimensional model and a ground three-dimensional model; according to the pose state, performing perspective projection rendering and image matting processing on the fused three-dimensional model to obtain an underground facility image only containing a view angle corresponding to the current pose state; and carrying out picture fusion on the real picture data and the underground facility image to obtain a virtual-real fused perspective image, and displaying the virtual-real fused perspective image. By implementing the method and the device, the problem that the underground facility shielded by the ground cannot be accurately and stably displayed in a real-time picture of a common camera in the prior art can be solved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of augmented reality visualization, and relates to a method, a system, a medium and equipment for panoramic display of underground facilities. BACKGROUND

[0002] With the increasing complexity of urban underground pipe networks, various pipelines such as power, communication and water supply are densely arranged in limited space, which puts forward an urgent demand for accurate positioning of underground facilities for urban operation, construction excavation and emergency repair. The traditional operation highly depends on the interpretation ability of on-site personnel for two-dimensional drawings or GIS systems, and requires restoring the plane data to a three-dimensional spatial relationship in the brain, which is tedious and prone to errors.

[0003] The current mainstream technology mainly displays the plane layout of the pipeline through a two-dimensional GIS system or combines ground penetrating radar for physical detection. However, the former lacks real three-dimensional space sense and field environment fusion, and the user needs to restore the spatial relationship in the brain by experience, which is tedious and prone to errors. The latter is high in cost, low in efficiency and does not have continuous visualization capability. Even if some applications try to introduce AR technology, it is difficult to realize dynamic and high-precision spatial alignment of virtual models and real camera pictures in complex urban environments due to the dependence on low-precision maps or simple visual markers, resulting in virtual-real fusion picture drift and jitter, and unable to form a stable and reliable "perspective" effect, which seriously restricts the engineering landing and large-scale promotion of the technology. SUMMARY

[0004] The application provides a method, a system, a medium and equipment for panoramic display of underground facilities, which can solve the problem that the prior art cannot accurately and stably display underground facilities obscured by the ground in real-time pictures of ordinary cameras.

[0005] To achieve the above-mentioned purpose, in a first aspect, the application provides a method for panoramic display of underground facilities, comprising:

[0006] Real-time acquisition of real picture data of a real camera in a target area, and synchronous adjustment of the pose state of a virtual camera configured in a fusion three-dimensional model based on the pose data of the real camera; wherein the fusion three-dimensional model is obtained by spatial alignment based on an underground facility three-dimensional model and a ground three-dimensional model; the underground facility three-dimensional model and the ground three-dimensional model are constructed by underground facility data and ground remote sensing surveying data of the target area, respectively;

[0007] Perspective projection rendering and matte processing of the fusion three-dimensional model according to the pose state to obtain an underground facility image only containing the view angle corresponding to the current pose state;

[0008] The real picture data is fused with the underground facility image to obtain a virtual-real fused perspective image and display the virtual-real fused perspective image.

[0009] Compared with the prior art, the embodiments of the present application have the following beneficial effects: by constructing the fusion three-dimensional model through spatial alignment of the underground facility three-dimensional model and the ground three-dimensional model, the geometric consistency of the underground and aboveground facilities in the unified coordinate system is ensured, and perspective distortion caused by model mispositioning is avoided; by real-time acquisition of real picture data of a real camera and synchronous adjustment of the pose state of a virtual camera, dynamic alignment of the virtual view and the real view in the spatial pose is realized, thereby solving the picture drift and spatial mispositioning problem caused by asynchronization of the virtual and real cameras in the traditional AR system; at the same time, by performing perspective projection rendering and matting processing on the fusion three-dimensional model based on the pose state, only the visible underground facility pixels under the current view angle are extracted and irrelevant backgrounds are removed, visual penetration of the occlusion and accurate separation of the target object are realized; by fusing and displaying the real picture and the underground facility image, the invisible underground structure is intuitively presented in the real scene in a virtual-real superimposed manner, and the ordinary camera is endowed with the ability of "perspective underground"; the steps are coordinated as a whole, and stable, continuous and high-fidelity AR perspective display of the underground facilities under the wide-area city monitoring camera system is realized, thereby breaking through the core bottleneck that the existing technology cannot reliably present the occluded underground facilities in the real dynamic environment due to the dependence on markers, low-precision maps or manual deduction.

[0010] In some embodiments of the first aspect of the present application, the underground facility three-dimensional model and the ground three-dimensional model are respectively constructed through underground facility data and ground remote sensing surveying data of the target area, and the underground facility data and the ground remote sensing surveying data comprise:

[0011] The original underground facility data and the original ground remote sensing surveying data of the target area are acquired; wherein the original underground facility data comprises any one or more of the following data combinations: CAD drawing data, BIM data and GIS data;

[0012] The original underground facility data and the original ground remote sensing surveying data are subjected to coordinate conversion and spatial registration processing, and the data with inconsistent resolution or scale are subjected to interpolation processing and normalization processing to obtain corresponding underground facility data and ground remote sensing surveying data;

[0013] The underground facility three-dimensional model and the ground three-dimensional model are respectively constructed based on the underground facility data and the ground remote sensing surveying data.

[0014] Compared with existing technologies, the above embodiments have the following beneficial effects: By acquiring original underground facility data and original ground remote sensing mapping data and clarifying the data sources such as CAD drawings, BIM data, or GIS data, a structured and standardized data foundation is provided for 3D modeling, improving the standardization and compatibility of model construction; furthermore, through coordinate transformation and spatial registration processing, the spatial reference system of multi-source heterogeneous data is unified, eliminating model misalignment caused by inconsistencies in coordinate systems; at the same time, interpolation and normalization processing is performed on data with inconsistent resolutions or scales, ensuring the scale consistency of data of different precisions during fusion, avoiding model distortion or texture breakage caused by differences in data granularity, thereby significantly improving the geometric accuracy and topological stability of underground and ground 3D models in the initial construction stage.

[0015] In some embodiments of the first aspect of this application, the fused 3D model is obtained by spatial alignment of an underground facility 3D model and a surface 3D model, including:

[0016] Calculate the spatial deviation between the preset test points in the target area and the corresponding positions in the underground facility 3D model or the ground 3D model. If the spatial deviation exceeds the preset threshold, the vertex coordinates of the underground facility 3D model or the ground 3D model are finely adjusted until the accuracy requirements are met, and the original fused 3D model is obtained.

[0017] The original fused 3D model is compressed to obtain the fused 3D model.

[0018] Compared with existing technologies, the above embodiments have the following beneficial effects: By calculating the spatial deviation between the preset test points and the corresponding positions of the model and setting a threshold judgment mechanism, the spatial accuracy of the fused model is quantitatively evaluated and actively corrected, avoiding the misalignment of virtual and real scenes caused by the accumulation of modeling errors; furthermore, by fine-tuning the position of the model vertex coordinates when the deviation exceeds the limit, the model approximates the real spatial relationship through local deformation optimization, thereby improving the absolute positioning accuracy of the model in engineering applications; at the same time, by performing model compression processing on the original fused 3D model, the mesh complexity and texture load are reduced while maintaining spatial accuracy, making the model more adaptable to the performance constraints of the real-time rendering engine, thus achieving a balance between accuracy and efficiency and ensuring the smooth operation of the system on ordinary hardware platforms.

[0019] In some embodiments of the first aspect of this application, the step of synchronously adjusting the pose state of the virtual camera configured in the fused 3D model based on the pose data of the real camera includes:

[0020] Based on the pose data corresponding to the current frame of the real camera, extract the installation position coordinates, horizontal rotation angle, pitch angle, roll angle and lens field parameters of the real camera corresponding to the current frame.

[0021] The horizontal rotation angle, pitch angle, and roll angle are converted into rotation matrices and combined with the installation position coordinates to generate a three-dimensional transformation matrix, which is then converted into a quaternion representation.

[0022] Based on the lens field parameters, the near clipping plane distance, far clipping plane distance, and view frustum angle are calculated, and the projection matrix of the virtual camera is constructed.

[0023] Based on the quaternion representation and projection matrix, the virtual camera is adjusted to complete the synchronous adjustment of the pose state of the current frame.

[0024] Compared with existing technologies, the above embodiments have the following beneficial effects: By extracting the installation position coordinates, horizontal rotation angle, pitch angle, roll angle, and lens field parameters from the pose data of real cameras, the six degrees of freedom state of the camera in three-dimensional space is fully captured, providing complete input for the accurate reproduction of virtual cameras; furthermore, the rotation angle is converted into a rotation matrix and combined with the position coordinates to generate a three-dimensional transformation matrix, and then converted into a quaternion representation, which not only preserves the continuity of rotation but also avoids the gimbal lock problem, improving the numerical stability of the attitude expression; at the same time, the projection matrix of the virtual camera is constructed according to the lens field parameters, so that the virtual view frustum and the perspective relationship of the real lens are strictly matched, thereby ensuring that the geometric projection of the rendered image is completely consistent with the real image, solving the problem of fusion tearing caused by perspective distortion or scale mismatch between virtual and real images.

[0025] In some embodiments of the first aspect of this application, the step of calculating the near clipping plane distance, the far clipping plane distance, and the view frustum angle based on the lens field of view parameters, and constructing the projection matrix of the virtual camera, includes:

[0026] Based on the lens field parameters, calculate the near clipping plane distance, far clipping plane distance, and view frustum angle corresponding to the current frame;

[0027] Calculate the difference between the view frustum angle of the current frame and the view frustum angle of the corresponding frame in the previous frame;

[0028] If the difference is greater than a preset threshold, linear interpolation is performed on the view frustum angles of the current frame and the previous frame to obtain a transition angle value, and the projection matrix of the virtual camera for the current frame is constructed based on the transition angle value; otherwise, the projection matrix of the virtual camera for the current frame is directly constructed based on the view frustum angles of the current frame.

[0029] Compared with existing technologies, the above embodiments have the following beneficial effects: by calculating the difference between the view frustum angle of the current frame and the previous frame and setting a threshold judgment mechanism, the sudden change behavior of the camera's viewpoint is identified, avoiding the image jump caused by rapid zooming or rotation; furthermore, when the difference exceeds the limit, linear interpolation is performed on the view frustum angle to generate a transition angle value, replacing the sudden switching with a smooth interpolation method, so that the change of the virtual camera's projection matrix has temporal continuity; at the same time, the projection matrix is ​​dynamically constructed according to the transition angle value or the original angle value, ensuring that the perspective transformation of the virtual image is smoothly connected with the behavior of the real camera in any motion state, thereby significantly improving the smoothness and naturalness of the user's visual experience during dynamic observation, and effectively suppressing the image jitter or discomfort caused by step changes in parameters.

[0030] In some embodiments of the first aspect of this application, adjusting the virtual camera based on the quaternion representation and projection matrix to synchronize the pose state of the current frame includes:

[0031] The quaternion representation is filtered to obtain a smoothed quaternion;

[0032] Based on the preset gimbal motion model and combined with the smoothed quaternion, predict the rotation quaternion of the virtual camera in the next frame.

[0033] Based on the rotation quaternion and projection matrix, the pose state of the virtual camera in the current frame is adjusted.

[0034] Compared with existing technologies, the above embodiments have the following beneficial effects: by performing filtering processing on the quaternion representation to obtain smoothed rotation data, the attitude jitter caused by sensor noise or network latency is suppressed, and the smoothness of the virtual camera's motion trajectory is improved; further, by combining the preset gimbal motion model with the smoothed quaternion to predict the rotation state of the next frame, and using kinematic priors to compensate for transmission delay, the virtual camera's attitude adjustment is forward-looking; at the same time, based on the predicted quaternion and the projection matrix, the current frame pose is adjusted synchronously, so that the virtual viewpoint not only follows the current real attitude, but also predicts the motion trend of the next moment. Thus, under the real constraints of communication latency or sampling interval, the high synchronization and low lag of the virtual and real images are still maintained, which greatly enhances the robustness and practicality of the system in the engineering deployment environment.

[0035] In some embodiments of the first aspect of this application, the step of performing perspective projection rendering and keying on the fused 3D model based on the pose state to obtain an image of the underground facility containing only the viewpoint corresponding to the current pose state includes:

[0036] Based on the pose state of the virtual camera, a model-view matrix is ​​constructed and multiplied with the projection matrix to calculate the transformation matrix;

[0037] Based on the transformation matrix, each vertex in the fused 3D model is transformed, and its two-dimensional coordinates and corresponding spatial depth values ​​on the image plane are calculated.

[0038] Determine whether the model object to which each vertex belongs is an underground facility and whether the corresponding spatial depth value is within a preset depth threshold; if the determination condition is met, retain the color value of the pixel corresponding to that vertex; otherwise, set the pixel to be completely transparent.

[0039] Output the color values ​​of the pixels corresponding to all vertices to generate an image of the underground facility.

[0040] Compared with existing technologies, the above embodiments have the following beneficial effects: By constructing a model-view matrix and multiplying it with the projection matrix to obtain a complete transformation matrix, the accurate projection of 3D vertices from world coordinates to planar coordinates is achieved, providing a mathematical basis for pixel-level rendering; further, the two-dimensional coordinates and spatial depth values ​​of each vertex on the image plane are calculated to establish a depth mapping relationship between pixels and three-dimensional space, providing a basis for subsequent depth-based occlusion judgment; at the same time, by judging whether the object to which the vertex belongs is an underground facility and the depth value is within a preset threshold, only valid target pixels are retained and the remaining areas are set to transparent, completing accurate chroma keying based on both semantic and spatial conditions; finally, the color values ​​of all pixels that meet the conditions are output to generate an image of underground facilities, so that the rendering result is strictly limited to the range that is visible from the current viewpoint and belongs to underground facilities, thereby achieving high-fidelity, anti-interference, and background-free visual extraction of target objects in complex urban scenes, providing a clean and accurate layer input for subsequent virtual-real fusion.

[0041] Secondly, the present invention also provides a system for panoramic display of underground facilities, comprising: a data acquisition and synchronization module, a rendering and keying module, and a fusion output module;

[0042] The data acquisition and synchronization module is used to acquire real-time image data of real cameras in the target area, and synchronously adjust the pose state of virtual cameras configured in the fused 3D model based on the pose data of the real cameras. The fused 3D model is obtained by spatial alignment of underground facility 3D models and ground 3D models. The underground facility 3D model and the ground 3D model are constructed respectively using underground facility data and ground remote sensing mapping data of the target area.

[0043] The rendering and chroma keying module is used to perform perspective projection rendering and chroma keying on the fused 3D model according to the pose state, so as to obtain an image of underground facilities that only contains the viewpoint corresponding to the current pose state.

[0044] The fusion output module is used to fuse the real image data with the underground facility image to obtain a perspective image that blends reality and virtuality and then displays it.

[0045] Compared to existing technologies, the above embodiments of this application have the following beneficial effects: By constructing a fused 3D model from underground facility 3D models and ground 3D models through spatial alignment, the geometric consistency of underground and above-ground facilities under a unified coordinate system is ensured, avoiding perspective distortion caused by model misalignment; by acquiring real-time image data from real cameras and synchronously adjusting the pose of virtual cameras, the virtual and real viewpoints are dynamically aligned in spatial pose, thus solving the problem of image drift and spatial misalignment caused by the asynchrony of virtual and real cameras in traditional AR systems; and by performing perspective projection rendering and keying on the fused 3D model based on the pose state. The process extracts only the pixels of underground facilities visible from the current viewpoint and removes irrelevant backgrounds, achieving visual penetration of obstructions and precise separation of target objects. Then, by merging the real image with the image of underground facilities, the invisible underground structure is presented intuitively in the real scene through a virtual-real overlay, giving ordinary cameras the ability to "see through the underground." Overall, the various steps work together to achieve stable, continuous, and high-fidelity AR perspective display of underground facilities in a wide-area urban surveillance camera system. This breaks through the core bottleneck of existing technologies that rely on markers, low-precision maps, or manual deduction, which makes it impossible to reliably present obstructed underground facilities in a real dynamic environment.

[0046] Thirdly, the present invention also provides a device for panoramic display of underground facilities, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when loaded onto the processor, implements the steps of any of the methods for panoramic display of underground facilities of the present invention.

[0047] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the methods for panoramic display of underground facilities according to the present invention. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating a method for panoramic display of underground facilities provided in some embodiments of the present invention.

[0049] Figure 2 This is a structural schematic diagram of a system for panoramic display of underground facilities provided in some embodiments of the present invention.

[0050] Figure 3 : This is a structural diagram of a device for panoramic display of underground facilities provided in some embodiments of the present invention.

[0051] Figure 4 This is a flowchart illustrating the acquisition and processing of real-world image data in some embodiments of the present invention.

[0052] Figure 5 This is a flowchart illustrating the pose synchronization adjustment of a virtual camera as provided in some embodiments of the present invention. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] Example 1:

[0055] Please refer to Figure 1 To address the problem that existing technologies cannot accurately and stably display underground facilities obscured by the ground in real-time footage from ordinary cameras, an embodiment of the present invention provides a method for panoramic display of underground facilities, comprising steps S1 to S3:

[0056] Step S1: Real-time acquisition of real-world image data from real cameras in the target area, and synchronous adjustment of the pose state of virtual cameras configured in the fused 3D model based on the pose data of the real cameras; wherein, the fused 3D model is obtained by spatial alignment of underground facility 3D model and ground 3D model; the underground facility 3D model and ground 3D model are constructed respectively through underground facility data and ground remote sensing mapping data of the target area.

[0057] Furthermore, the underground facility 3D model and the ground 3D model can be constructed through the following preferred embodiments, including steps S11-S13, as follows:

[0058] S11: Obtain the original underground facility data and original ground remote sensing mapping data of the target area; wherein, the original underground facility data includes any one or more of the following data combinations: CAD drawing data, BIM data, and GIS data;

[0059] S12: Perform coordinate transformation and spatial registration on the original underground facility data and the original ground remote sensing mapping data, and perform interpolation and normalization on the data with inconsistent resolution or scale to obtain the corresponding underground facility data and ground remote sensing mapping data.

[0060] S13: Based on the underground facility data and the ground remote sensing mapping data, respectively, construct the underground facility three-dimensional model and the ground three-dimensional model.

[0061] In practical implementation, to realistically recreate the underground cables and surrounding facilities of the target area in the digital twin environment, CAD drawings, BIM (Building Information Modeling) data, or GIS (Geographic Information System) data of existing underground cables, communication pipelines, water supply pipes, gas pipes, and other facilities are obtained from the urban pipeline management department. This data is then supplemented or updated using methods such as ground-penetrating radar (GPR), electromagnetic detection, and underground cameras. Ground remote sensing mapping data is obtained using aerial surveying, UAV oblique photography, LiDAR scanning, and panoramic camera photography to acquire high-precision three-dimensional information on roads, buildings, green spaces, and terrain.

[0062] After data collection, all collected data is converted and registered according to a unified geographic coordinate system (such as WGS84, CGCS2000) to ensure precise spatial alignment between underground facilities and the surface scene. For data with inconsistent resolution or scale, interpolation resampling and scale normalization methods are used for unified processing.

[0063] Next, model building will be carried out, including:

[0064] Underground facility modeling: Construct a 3D solid model based on parameters such as pipe diameter, material, and laying method, and record information such as facility type, burial depth, and service life in the model attributes to facilitate subsequent classification rendering and interactive query.

[0065] Ground scene modeling: High-precision terrain meshes are generated based on laser point clouds and photogrammetric data, and texture mapping is used to give them realistic materials, ensuring visual consistency with the real environment.

[0066] In this preferred embodiment, by acquiring original underground facility data and original ground remote sensing mapping data and clarifying the data sources such as CAD drawings, BIM data, or GIS data, a structured and standardized data foundation is provided for 3D modeling, improving the standardization and compatibility of model construction. Furthermore, through coordinate transformation and spatial registration processing, the spatial reference system of multi-source heterogeneous data is unified, eliminating model misalignment caused by inconsistencies in coordinate systems. At the same time, interpolation and normalization processing are performed on data with inconsistent resolutions or scales, ensuring the scale consistency of data with different precisions during fusion, avoiding model distortion or texture breakage caused by differences in data granularity, thereby significantly improving the geometric accuracy and topological stability of underground and ground 3D models in the initial construction stage.

[0067] Furthermore, the fused 3D model can be obtained through the following preferred implementation method, including steps S14-S15, as follows:

[0068] S14: Calculate the spatial deviation between the preset test points in the target area and the corresponding positions in the underground facility 3D model or the ground 3D model. If the spatial deviation exceeds the preset threshold, fine-tune the position coordinates of the vertices of the underground facility 3D model or the ground 3D model until the accuracy requirements are met, and obtain the original fused 3D model.

[0069] S15: Perform model compression processing on the original fused 3D model to obtain the fused 3D model.

[0070] In practical implementation, after the two models are constructed, the spatial deviation between the actual measured points and the corresponding positions in the models is compared. If the error exceeds a set threshold, the model is fine-tuned in position and optimized in deformation to ensure the model's spatial accuracy. In addition, to ensure subsequent real-time rendering performance, this invention performs mesh simplification, normal merging, and texture compression on the model while maintaining accuracy, enabling it to be efficiently loaded and rendered on the GPU.

[0071] In this preferred embodiment, by calculating the spatial deviation between the preset test points and the corresponding positions of the model and setting a threshold judgment mechanism, the spatial accuracy of the fused model is quantitatively evaluated and actively corrected, avoiding the misalignment of virtual and real scenes caused by the accumulation of modeling errors. Furthermore, by fine-tuning the position of the model vertex coordinates when the deviation exceeds the limit, the model approximates the real spatial relationship through local deformation optimization, thereby improving the absolute positioning accuracy of the model in engineering applications. At the same time, by performing model compression processing on the original fused 3D model, the mesh complexity and texture load are reduced while maintaining spatial accuracy, making the model more adaptable to the performance constraints of the real-time rendering engine, thus achieving a balance between accuracy and efficiency and ensuring the smooth operation of the system on ordinary hardware platforms.

[0072] After obtaining the fused 3D model, it is necessary to acquire real-time image data from actual cameras in the target area. During implementation, refer to... Figure 4 The flowchart shown illustrates the acquisition and processing of real-time video data. First, network access configuration is performed on the surveillance cameras deployed on-site. These cameras can be of various types, including fixed, PTZ, and fisheye wide-angle cameras, all of which must support Real-Time Streaming Protocol (RTSP) or other transmission methods capable of acquiring continuous video frames (such as HTTP-FLV, WebRTC, etc.). By specifying the camera's IP address, port number, and access credentials, a video stream connection is established with the camera, and high-definition video frames are continuously received.

[0073] In addition, the video stream access module is used to preprocess the received raw video data, including but not limited to:

[0074] Video decoding: Converts compressed video streams (H.264, H.265, etc.) into image frame data that can be directly rendered using a GPU or CPU decoder.

[0075] Resolution and frame rate adjustment: Based on the needs of the virtual rendering scene, the video frames are scaled or interpolated to ensure smooth visuals under different hardware performance conditions.

[0076] Color space conversion: Convert the color space commonly used by cameras such as YUV to RGB format to meet the texture input requirements of the rendering engine.

[0077] Preprocessed video frames are passed to the 3D rendering engine via video memory mapping or memory buffering. In the rendering engine, the video frame data is bound to a texture object and dynamically updated as input parameters for the material. This texture can be directly mapped onto planar, cubic, or spherical objects in the virtual scene, enabling the spatial representation of the video footage.

[0078] To further ensure the temporal consistency between the video stream and the virtual scene, this invention introduces a frame synchronization mechanism during texture updates:

[0079] Before the virtual rendering engine renders each frame, it checks if the latest frame exists in the video stream buffer. If it does, it updates the texture data.

[0080] If there is a delay or frame loss in the video stream, the texture content of the previous frame is preserved to avoid screen flickering or distortion.

[0081] In practical implementations, texture mapping of video streams can be divided into the following modes:

[0082] Plane mapping mode: Directly maps video frames onto a two-dimensional plane, suitable for use as a reference display of monitoring screens in UI interfaces or virtual scenes.

[0083] Perspective mapping mode: Maps video frames onto the view frustum of a virtual camera according to the projection matrix of the real camera, achieving geometric consistency with the real image.

[0084] Environment Mapping Mode: In fisheye or panoramic camera applications, video frames are spherically unfolded and mapped for panoramic environment reconstruction.

[0085] Furthermore, based on the pose data of the real camera, the pose state of the virtual camera is adjusted synchronously. This can be achieved through the following preferred implementation method, including steps S16-S19:

[0086] S16: Based on the pose data corresponding to the current frame of the real camera, extract the installation position coordinates, horizontal rotation angle, pitch angle, roll angle and lens field parameters of the real camera corresponding to the current frame.

[0087] S17: Convert the horizontal rotation angle, pitch angle and roll angle into a rotation matrix, and combine it with the installation position coordinates to generate a three-dimensional transformation matrix and convert it into a quaternion representation;

[0088] S18: Based on the lens field parameters, calculate the near clipping plane distance, the far clipping plane distance, and the viewing frustum angle, and construct the projection matrix of the virtual camera;

[0089] S19: Based on the quaternion representation and projection matrix, adjust the virtual camera to complete the synchronous adjustment of the pose state of the current frame.

[0090] In this preferred embodiment, by extracting the installation position coordinates, horizontal rotation angle, pitch angle, roll angle, and lens field parameters from the pose data of the real camera, the six degrees of freedom state of the camera in three-dimensional space is fully captured, providing complete input for the accurate reproduction of the virtual camera. Furthermore, the rotation angle is converted into a rotation matrix and combined with the position coordinates to generate a three-dimensional transformation matrix, which is then converted into a quaternion representation. This preserves the continuity of rotation and avoids the gimbal lock problem, improving the numerical stability of the attitude expression. At the same time, the projection matrix of the virtual camera is constructed according to the lens field parameters, so that the virtual view frustum and the perspective relationship of the real lens are strictly matched, thereby ensuring that the geometric projection of the rendered image is completely consistent with the real image, solving the problem of fusion tearing caused by perspective distortion or scale mismatch between virtual and real images.

[0091] Furthermore, step S18 can be implemented through the following preferred embodiments, including steps S181-S183, as follows:

[0092] S181: Calculate the near clipping plane distance, far clipping plane distance, and view frustum angle corresponding to the current frame based on the lens field parameters;

[0093] S182: Calculate the difference between the view frustum angle of the current frame and the view frustum angle of the previous frame;

[0094] S183: If the difference is greater than a preset threshold, then perform linear interpolation on the view frustum angle between the current frame and the previous frame to obtain a transition angle value, and construct the projection matrix of the virtual camera for the current frame based on the transition angle value; otherwise, directly construct the projection matrix of the virtual camera for the current frame based on the view frustum angle of the current frame.

[0095] In this preferred embodiment, by calculating the difference between the view frustum angle of the current frame and the previous frame and setting a threshold judgment mechanism, abrupt changes in the camera's viewpoint are identified, avoiding image jumps caused by rapid zooming or rotation. Furthermore, when the difference exceeds the limit, linear interpolation is performed on the view frustum angle to generate a transition angle value, replacing abrupt switching with smooth interpolation, making the changes in the virtual camera's projection matrix temporally continuous. At the same time, the projection matrix is ​​dynamically constructed based on the transition angle value or the original angle value, ensuring that the perspective transformation of the virtual image is smoothly connected with the behavior of the real camera under any motion state, thereby significantly improving the smoothness and naturalness of the user's visual experience during dynamic observation and effectively suppressing image jitter or discomfort caused by step changes in parameters.

[0096] Furthermore, step S19 can be implemented through the following preferred embodiments, including steps S191-S193, as follows:

[0097] S191: Perform filtering on the quaternion representation to obtain a smoothed quaternion;

[0098] S192: Based on the preset gimbal motion model and combined with the smoothed quaternion, predict the rotation quaternion of the virtual camera in the next frame.

[0099] S193: Based on the rotation quaternion and projection matrix, adjust the pose state of the virtual camera in the current frame.

[0100] In this preferred embodiment, smoothed rotation data is obtained by filtering the quaternion representation, which suppresses attitude jitter caused by sensor noise or network latency and improves the smoothness of the virtual camera's motion trajectory. Furthermore, by combining the preset gimbal motion model with the smoothed quaternion to predict the rotation state of the next frame, kinematic priors are used to compensate for transmission delay, making the virtual camera's attitude adjustment forward-looking. At the same time, the pose of the current frame is adjusted synchronously based on the predicted quaternion and the projection matrix, so that the virtual viewpoint not only follows the current real attitude, but can also predict the motion trend of the next moment. Thus, even under the real constraints of communication delay or sampling interval, the high synchronization and low lag of the virtual and real images are maintained, which greatly enhances the robustness and practicality of the system in the engineering deployment environment.

[0101] In practical implementation, in order to match and calibrate the positions of virtual cameras deployed in the digital twin scenario, refer to Figure 5 The diagram illustrates a flowchart of pose synchronization adjustment for a virtual camera. Precise matching is performed based on the installation coordinates, height, orientation, and lens parameters (field of view, focal length, etc.) of the real surveillance camera. A one-to-one correspondence between position, rotation, and field of view is achieved through calibration algorithms or sensor data (such as IMU, GPS, rangefinder), minimizing the parallax between the image captured by the virtual camera and the image captured by the real camera.

[0102] In this process, to ensure that the viewpoint of the image captured by the virtual camera is consistent with that of the video image from the real surveillance camera, this invention establishes a real-time camera pose synchronization mechanism to dynamically adjust the rotation angle and field parameters of the virtual camera in the digital twin scenario:

[0103] First, the mounting orientation of the actual camera needs to be calibrated, including the camera's fixed position coordinates (X, Y, Z), horizontal rotation angle (Yaw), pitch angle (Pitch), roll angle (Roll), and field of view (FOV). Calibration can be obtained through the following methods:

[0104] Installation parameters can be read directly: For controllable PTZ cameras, the current PTZ rotation angle and zoom parameters can be obtained directly from the camera control interface.

[0105] Visual reverse calibration: Calculates the camera's rotation and viewing angle information in the world coordinate system by identifying known landmarks (such as AR markers and feature points) in the image.

[0106] When the synchronization mechanism is running, it acquires the pose data of the real camera at fixed time intervals and transforms it into a unified coordinate system used by the virtual scene. The transformation process includes coordinate system transformation, rotation matrix or quaternion calculations to ensure that the virtual camera's orientation is consistent with that of the real camera.

[0107] Regarding field-of-view synchronization, the projection matrix parameters of the virtual camera are adjusted in real time based on the zoom operation of the real camera, ensuring that the perspective relationship captured by the virtual camera is consistent with that of the real camera. For lenses with a large zoom range, interpolation transitions can be used to smooth the changes and avoid sudden changes in the image that could cause discomfort to the user.

[0108] To reduce the impact of network latency or data jitter on synchronization accuracy, this invention introduces attitude prediction and filtering algorithms:

[0109] Smooth changes in camera rotation angle and reduce jitter by using methods such as Kalman filtering and complementary filtering;

[0110] A motion prediction model (i.e., a PTZ motion model) is established by combining the motion characteristics of the camera PTZ. When network latency exists, the virtual camera's posture is corrected in advance to ensure that the dynamic changes of the virtual image and the real image are consistent.

[0111] Step S2: Based on the pose state, perform perspective projection rendering and keying on the fused 3D model to obtain an image of the underground facility that only contains the viewpoint corresponding to the current pose state.

[0112] Furthermore, step S2 can be implemented through the following preferred embodiments, including steps S21-S24, as follows:

[0113] S21: Based on the pose state of the virtual camera, construct the model view matrix and multiply it with the projection matrix to calculate the transformation matrix;

[0114] S22: Based on the transformation matrix, transform each vertex in the fused 3D model, and calculate its 2D coordinates and corresponding spatial depth values ​​on the image plane;

[0115] S23: Determine whether the model object to which each vertex belongs is an underground facility and whether the corresponding spatial depth value is within the preset depth threshold; if the determination condition is met, retain the color value of the pixel corresponding to the vertex, otherwise set the pixel to be completely transparent;

[0116] S24: Output the color values ​​of all pixels corresponding to the vertices to generate an image of the underground facility.

[0117] When rendering and keying, traditional methods often rely on a single background color, which cannot accurately handle semi-transparent or transparent objects. Color overflow often occurs at the edges, and the effect deteriorates significantly in complex lighting or multi-colored background environments. In addition, conventional depth buffer mechanisms also have shortcomings: the main depth buffer has limited precision, making it difficult to distinguish pixels of the same depth but belonging to different objects; it lacks independent control over individual objects; and obvious jagged edges often appear in edge areas.

[0118] In order to achieve perspective display of facilities such as underground pipelines covered by ground obstructions in video footage, this invention introduces a custom depth rendering and keying mechanism in the digital twin rendering environment, so that the underground facility model can be separated and processed independently of the scene depth information during the rendering process.

[0119] First, in the digital twin scene, a custom depth channel or separate rendering layer is set for key model objects such as underground pipelines, power substations, and communication wells. This depth channel is independent of the standard scene depth buffer, recording only the spatial relationship between specific target objects and the virtual camera, while ignoring the depth information of other non-target objects. This prevents pipeline models from being eliminated in depth comparisons with objects such as the ground and buildings, thus providing a data foundation for subsequent perspective rendering.

[0120] During the rendering process, the system generates two streams of images:

[0121] Normal scene capture footage: includes a complete 3D environment such as ground, buildings, and vegetation from the perspective of a virtual camera;

[0122] Custom depth chroma keying: Includes only underground pipeline models within a custom depth channel, and generates an image with a transparent background based on the depth data for later overlay with real-world video.

[0123] The keying process includes the following steps:

[0124] Depth threshold filtering: Based on the spatial distribution range of underground facilities, a depth threshold is set to retain only model pixels within the effective range and discard the parts that do not need to be displayed.

[0125] Pixel mask generation: Generate single-channel or multi-channel mask images using a custom depth channel, and combine alpha blending technology to make pixels in non-target areas completely transparent;

[0126] Edge anti-aliasing: To prevent jagged edges or transparency defects at the pipeline edges, multi-sampling anti-aliasing (MSAA), supersampling (SSAA), or depth-based blurring algorithms are used to smooth the matting edges.

[0127] Color and material adjustments: Dynamically change the model's color, brightness, and lighting effects according to the importance or type of underground facilities to improve its recognizability in real-world scenes.

[0128] In some scenarios requiring higher precision, the present invention can also employ a pixel-by-pixel depth comparison matting method: the depth information obtained from the video stream of a real camera through structured light, binocular stereo vision, etc., is compared with the custom depth of the virtual scene at the pixel level, retaining only the pixels of underground pipelines behind the real terrain, thereby obtaining a more accurate perspective effect.

[0129] In this preferred embodiment, by constructing a model-view matrix and multiplying it with the projection matrix to obtain a complete transformation matrix, the accurate projection of 3D vertices from world coordinates to planar coordinates is achieved, providing a mathematical basis for pixel-level rendering. Furthermore, the 2D coordinates and spatial depth values ​​of each vertex on the image plane are calculated to establish a depth mapping relationship between pixels and 3D space, providing a basis for subsequent depth-based occlusion judgment. Simultaneously, by determining whether the object to which the vertex belongs is an underground facility and whether the depth value is within a preset threshold, only valid target pixels are retained, and the remaining areas are set to transparent, completing accurate chroma keying based on both semantic and spatial conditions. Finally, the color values ​​of all pixels meeting the conditions are output to generate an image of the underground facility, strictly limiting the rendering result to the range visible from the current viewpoint and belonging to the underground facility. This achieves high-fidelity, interference-resistant, and background-free visual extraction of target objects in complex urban scenes, providing clean and accurate layer input for subsequent virtual-real fusion.

[0130] Step S3: The real image data is fused with the underground facility image to obtain a perspective image that blends reality and virtuality, and then displayed.

[0131] After obtaining the custom depth matting results of key models such as underground pipelines, it is necessary to perform virtual-real fusion and render the output for display. Here, the present invention adopts a perspective-first rendering strategy to ensure that underground facilities are always in the visual foreground position in the virtual-real fusion screen. Even if they are completely obscured by the ground, roads or buildings in the real scene, they can still be directly displayed in the output screen, thereby achieving the "perspective display" effect of the obscured target.

[0132] The core idea of ​​perspective-first rendering is to skip the standard depth detection process in the rendering pipeline, allowing pixels of underground facilities to be directly written to the frame buffer without having to compare their depth with the ground or other objects. Specific implementation methods include:

[0133] 1. Disable depth testing: When rendering underground pipeline models, disable the depth testing function of the rendering engine so that its pixel data directly covers the corresponding position in the current frame buffer, thus visually placing it above other objects.

[0134] Second: Adjust the rendering order. By modifying the rendering queue or drawing batch order, the rendering process of underground facilities is scheduled to be executed after all scene objects to ensure that its final pixel values ​​cover the existing image data.

[0135] 3. Custom depth writing strategy: In some rendering modes, it is allowed to write only to the color buffer without updating the depth buffer, thereby avoiding affecting the depth ordering of other objects, while ensuring that the pixels of underground facilities are always "front".

[0136] Fourth: Semi-transparent perspective mode. To increase the amount of information that can be displayed on the screen, semi-transparent shading can be used in perspective priority rendering to blend underground facilities with the real scene. This preserves the texture information of the ground while allowing users to clearly see the location and direction of the underground structure.

[0137] 5. Dynamic highlighting and edge outlining: To enhance the visual recognizability of underground facilities in perspective mode, this invention can add effects such as glowing and outline outlining to the model during the rendering stage, or dynamically change the color according to the type of facility. For example, high-voltage cables use red glowing outlines, while communication optical cables use blue glowing outlines.

[0138] Six: Deep integration optimization. For underground facilities that display multiple different depth levels at the same time, pseudo-depth layering can be introduced in perspective-first rendering. The sense of spatial layering can be conveyed by using color gradients, transparency changes and other methods to avoid the difficulty of recognition caused by overlapping content.

[0139] Finally, the digital twin captured footage obtained through custom depth chroma keying is overlaid in real time with the video texture from a real camera, and an image blending algorithm is used to generate a fused image. This fused image includes both ground information from the real scene and intuitively displays the location and shape of underground facilities, achieving a seamless integration of virtual and reality.

[0140] In addition, a transparency adjustment slider can be set on the system where this solution is deployed. Users can dynamically adjust the transparency of the virtual and real images according to their needs, so as to achieve a continuous switch from a pure real image to a pure virtual image, or maintain an arbitrary ratio of superposition between the two to adapt to the visualization needs of different scenarios.

[0141] In this solution, through the above-described implementation method, the present invention constructs a high-precision 3D scene model, embeds underground facility data into it, and combines it with a virtual rendering system synchronized with the real camera's perspective to achieve a visual overlay display of invisible underground facilities in the real scene. Compared with traditional GIS drawings or static model display methods, users do not need professional software or paper materials; they can "see" the direction and distribution of underground cables simply through the camera's view, truly achieving a WYSIWYG effect. Furthermore, by comprehensively calibrating the position, posture, focal length, and other parameters of the real camera and constructing an equivalent virtual camera in the 3D virtual environment, a dynamic one-to-one correspondence between virtual and real perspectives is achieved. Combined with a camera posture tracking and synchronization mechanism, the virtual image always changes synchronously with the real camera, ensuring that the overlay image maintains a precise spatial registration relationship with the real scene. This breaks through the low-precision limitations of traditional AR systems that rely solely on planar markers or GPS, achieving centimeter-level 3D matching accuracy and effectively guaranteeing the accuracy and practicality of the perspective effect. Furthermore, this invention employs a depth-information-assisted matting and rendering algorithm during image fusion, combined with a real-time angle synchronization and screen refresh mechanism from the virtual camera, achieving highly stable visual presentation. It ensures that the perspective effect remains consistent with the camera's actual position in the real world as the camera rotates or moves, providing a natural visual experience for users. Compared to traditional AR systems prone to issues like "image ghosting," "positioning jumps," and "posture jitter," this system optimizes its algorithm by predicting and anticipating camera rotation speed, interpolating rotation behavior, and smoothing out jitter, resulting in greater user-friendliness. On the other hand, compared to traditional work methods where pipeline maintenance personnel rely on cumbersome methods such as consulting paper drawings, retrieving GIS system information, and conducting on-site surveys to determine the location of underground facilities—a process that is inefficient and prone to errors—this invention addresses these issues. This solution's deployment system allows operators to directly access underground cable information via mobile terminals, tablets, or existing monitoring system feeds. Combined with real-world visuals, it enables rapid location of target facilities, achieving "zero-learning-cost" visual decision support. Economically, it can be deployed directly using existing urban camera resources without large-scale hardware replacement or reconstruction. Only software access and parameter configuration are required for it to take effect. This gives the solution advantages in urban smart infrastructure upgrades: low deployment cost, short transformation cycle, and rapid online operation, significantly improving the technology's feasibility and scalability.

[0142] In summary, compared with the prior art, the above embodiments of this application have the following beneficial effects: By constructing a fused 3D model from underground facility 3D models and ground 3D models through spatial alignment, the geometric consistency of underground and above-ground facilities under a unified coordinate system is ensured, avoiding perspective distortion caused by model misalignment; by acquiring real-time image data from real cameras and synchronously adjusting the pose state of virtual cameras, the virtual viewpoint and the real viewpoint are dynamically aligned in spatial pose, thereby solving the problem of image drift and spatial misalignment caused by the asynchrony of virtual and real cameras in traditional AR systems; and by performing perspective projection rendering and matting on the fused 3D model based on the pose state... Image processing extracts only the pixels of underground facilities visible from the current viewpoint and removes irrelevant backgrounds, achieving visual penetration of obstructions and precise separation of target objects. Then, by merging the real image with the image of underground facilities, the invisible underground structure is presented intuitively in the real scene through a virtual-real overlay, giving ordinary cameras the ability to "see through the underground". Overall, the steps work together to achieve stable, continuous, and high-fidelity AR perspective display of underground facilities in a wide-area urban surveillance camera system, breaking through the core bottleneck of existing technologies that rely on markers, low-precision maps, or manual deduction, which cannot reliably present obstructed underground facilities in a real dynamic environment.

[0143] Example 2:

[0144] Please refer to Figure 2 Based on the same inventive concept, the present invention discloses a system for panoramic display of underground facilities, comprising: a data acquisition and synchronization module M1, a rendering and keying module M2, and a fusion output module M3;

[0145] The data acquisition and synchronization module M1 is used to acquire real-time image data of real cameras in the target area, and synchronously adjust the pose state of virtual cameras configured in the fused 3D model based on the pose data of the real cameras. The fused 3D model is obtained by spatial alignment of underground facility 3D model and ground 3D model. The underground facility 3D model and ground 3D model are constructed by underground facility data and ground remote sensing mapping data of the target area, respectively.

[0146] Furthermore, the data acquisition and synchronization module M1 includes: a data acquisition unit, a preprocessing unit, and a model building unit;

[0147] The data acquisition unit is used to acquire raw underground facility data and raw ground remote sensing mapping data of the target area; wherein the raw underground facility data includes any one or more of the following data combinations: CAD drawing data, BIM data, and GIS data;

[0148] The preprocessing unit is used to perform coordinate transformation and spatial registration on the original underground facility data and the original ground remote sensing mapping data, and to perform interpolation and normalization on data with inconsistent resolution or scale to obtain the corresponding underground facility data and ground remote sensing mapping data.

[0149] The model building unit is used to construct a three-dimensional model of the underground facility and a three-dimensional model of the ground based on the underground facility data and the ground remote sensing mapping data, respectively.

[0150] In this preferred embodiment, by acquiring original underground facility data and original ground remote sensing mapping data and clarifying the data sources such as CAD drawings, BIM data, or GIS data, a structured and standardized data foundation is provided for 3D modeling, improving the standardization and compatibility of model construction. Furthermore, through coordinate transformation and spatial registration processing, the spatial reference system of multi-source heterogeneous data is unified, eliminating model misalignment caused by inconsistencies in coordinate systems. At the same time, interpolation and normalization processing are performed on data with inconsistent resolutions or scales, ensuring the scale consistency of data with different precisions during fusion, avoiding model distortion or texture breakage caused by differences in data granularity, thereby significantly improving the geometric accuracy and topological stability of underground and ground 3D models in the initial construction stage.

[0151] Furthermore, the data acquisition and synchronization module M1 also includes: a model adjustment unit and a model compression unit;

[0152] The model adjustment unit is used to calculate the spatial deviation between the preset test point in the target area and the corresponding position in the underground facility 3D model or the ground 3D model. If the spatial deviation exceeds the preset threshold, the vertex coordinates of the underground facility 3D model or the ground 3D model are finely adjusted until the accuracy requirements are met, and the original fused 3D model is obtained.

[0153] The model compression unit is used to perform model compression processing on the original fused 3D model to obtain the fused 3D model.

[0154] In this preferred embodiment, by calculating the spatial deviation between the preset test points and the corresponding positions of the model and setting a threshold judgment mechanism, the spatial accuracy of the fused model is quantitatively evaluated and actively corrected, avoiding the misalignment of virtual and real scenes caused by the accumulation of modeling errors. Furthermore, by fine-tuning the position of the model vertex coordinates when the deviation exceeds the limit, the model approximates the real spatial relationship through local deformation optimization, thereby improving the absolute positioning accuracy of the model in engineering applications. At the same time, by performing model compression processing on the original fused 3D model, the mesh complexity and texture load are reduced while maintaining spatial accuracy, making the model more adaptable to the performance constraints of the real-time rendering engine, thus achieving a balance between accuracy and efficiency and ensuring the smooth operation of the system on ordinary hardware platforms.

[0155] Furthermore, the data acquisition and synchronization module M1 also includes: a parameter extraction unit, a quaternion generation unit, a matrix generation unit, and a pose adjustment unit;

[0156] The parameter extraction unit is used to extract the installation position coordinates, horizontal rotation angle, pitch angle, roll angle and lens field parameters of the real camera corresponding to the current frame based on the pose data corresponding to the current frame of the real camera.

[0157] The quaternion generation unit is used to convert the horizontal rotation angle, pitch angle and roll angle into a rotation matrix, and combine it with the installation position coordinates to generate a three-dimensional transformation matrix and convert it into a quaternion representation.

[0158] The matrix generation unit is used to calculate the near clipping plane distance, the far clipping plane distance, and the view frustum angle based on the lens field parameters, and to construct the projection matrix of the virtual camera.

[0159] The pose adjustment unit is used to adjust the virtual camera based on the quaternion representation and the projection matrix to complete the synchronous adjustment of the pose state of the current frame.

[0160] In this preferred embodiment, by extracting the installation position coordinates, horizontal rotation angle, pitch angle, roll angle, and lens field parameters from the pose data of a real camera, the six degrees of freedom state of the camera in three-dimensional space is fully captured, providing complete input for the accurate reproduction of the virtual camera. Furthermore, the rotation angle is converted into a rotation matrix and combined with the position coordinates to generate a three-dimensional transformation matrix, which is then converted into a quaternion representation. This preserves the continuity of rotation and avoids the gimbal lock problem, improving the numerical stability of the attitude expression. At the same time, the projection matrix of the virtual camera is constructed based on the lens field parameters, so that the virtual view frustum and the perspective relationship of the real lens are strictly matched, thereby ensuring that the geometric projection of the rendered image is completely consistent with the real image, solving the problem of fusion tearing caused by perspective distortion or scale mismatch between virtual and real images.

[0161] Furthermore, the matrix generation unit includes: a first calculation subunit, a difference calculation subunit, and a matrix calculation subunit;

[0162] The first calculation subunit is used to calculate the near clipping plane distance, far clipping plane distance and view frustum angle corresponding to the current frame based on the lens field parameters.

[0163] The difference calculation subunit is used to calculate the difference between the view frustum angle of the current frame and the view frustum angle of the previous frame.

[0164] The matrix calculation subunit is used to perform linear interpolation on the view frustum angles of the current frame and the previous frame to obtain a transition angle value if the difference is greater than a preset threshold, and construct the projection matrix of the virtual camera of the current frame based on the transition angle value; otherwise, it directly constructs the projection matrix of the virtual camera of the current frame based on the view frustum angle of the current frame.

[0165] In this preferred embodiment, by calculating the difference between the view frustum angle of the current frame and the previous frame and setting a threshold judgment mechanism, the sudden change behavior of the camera's viewpoint is identified, avoiding image jumps caused by rapid zooming or rotation. Furthermore, when the difference exceeds the limit, linear interpolation is performed on the view frustum angle to generate a transition angle value, replacing the sudden switching with a smooth interpolation method, so that the changes in the projection matrix of the virtual camera have temporal continuity. At the same time, the projection matrix is ​​dynamically constructed based on the transition angle value or the original angle value, ensuring that the perspective transformation of the virtual image is smoothly connected with the behavior of the real camera in any motion state, thereby significantly improving the smoothness and naturalness of the user's visual experience during dynamic observation and effectively suppressing image jitter or discomfort caused by step changes in parameters.

[0166] Furthermore, the pose adjustment unit includes: a filtering subunit, a prediction subunit, and an adjustment subunit;

[0167] The filtering subunit is used to perform filtering on the quaternion representation to obtain a smoothed quaternion.

[0168] The prediction subunit is used to predict the rotation quaternion of the virtual camera in the next frame based on the preset gimbal motion model and the smoothed quaternion.

[0169] The adjustment subunit is used to adjust the pose state of the virtual camera in the current frame based on the rotation quaternion and the projection matrix.

[0170] In this preferred embodiment, smoothed rotation data is obtained by filtering the quaternion representation, which suppresses attitude jitter caused by sensor noise or network latency and improves the smoothness of the virtual camera's motion trajectory. Furthermore, by combining the preset gimbal motion model with the smoothed quaternion to predict the rotation state of the next frame, kinematic priors are used to compensate for transmission delay, making the virtual camera's attitude adjustment forward-looking. At the same time, the current frame pose is adjusted synchronously based on the predicted quaternion and the projection matrix, so that the virtual viewpoint not only follows the current real pose but can also predict the motion trend at the next moment. Thus, even under the real constraints of communication delay or sampling interval, the high synchronization and low lag of the virtual and real images are maintained, greatly enhancing the robustness and practicality of the system in engineering deployment environments.

[0171] The rendering and keying module M2 is used to perform perspective projection rendering and keying processing on the fused 3D model according to the pose state, so as to obtain an image of underground facilities that only contains the viewpoint corresponding to the current pose state.

[0172] Furthermore, the rendering matting module M2 includes: a transformation matrix calculation unit, a depth calculation unit, a judgment unit, and a pixel output unit;

[0173] The transformation matrix calculation unit is used to construct a model-view matrix based on the pose state of the virtual camera, and multiply it with the projection matrix to calculate the transformation matrix.

[0174] The depth calculation unit is used to transform each vertex in the fused 3D model based on the transformation matrix, and calculate its two-dimensional coordinates and corresponding spatial depth value on the image plane.

[0175] The judgment unit is used to determine whether the model object to which each vertex belongs is an underground facility and whether the corresponding spatial depth value is within a preset depth threshold; if the judgment condition is met, the color value of the pixel corresponding to the vertex is retained, otherwise the pixel is set to completely transparent.

[0176] The pixel output unit is used to output the color values ​​of the pixels corresponding to all vertices to generate an image of the underground facility.

[0177] In this preferred embodiment, a complete transformation matrix is ​​obtained by constructing a model-view matrix and multiplying it with the projection matrix, thus achieving accurate projection of 3D vertices from world coordinates to planar coordinates, providing a mathematical basis for pixel-level rendering. Furthermore, the 2D coordinates and spatial depth values ​​of each vertex on the image plane are calculated to establish a depth mapping relationship between pixels and 3D space, providing a basis for subsequent depth-based occlusion judgment. Simultaneously, by determining whether the object to which the vertex belongs is an underground facility and whether the depth value is within a preset threshold, only valid target pixels are retained, and the remaining areas are set to transparent, completing accurate chroma keying based on both semantic and spatial conditions. Finally, the color values ​​of all pixels meeting the conditions are output to generate an image of the underground facility, strictly limiting the rendering result to the area visible from the current viewpoint and belonging to the underground facility. This achieves high-fidelity, interference-resistant, and background-free visual extraction of target objects in complex urban scenes, providing clean and accurate layer input for subsequent virtual-real fusion.

[0178] The fusion output module M3 is used to fuse the real image data with the underground facility image to obtain a perspective image that blends reality and virtuality and then displays it.

[0179] In summary, compared with existing technologies, the embodiments of this application have the following beneficial effects: By constructing a fused 3D model from underground facility 3D models and ground 3D models through spatial alignment, the geometric consistency of underground and above-ground facilities under a unified coordinate system is ensured, avoiding perspective distortion caused by model misalignment; by acquiring real-time image data from real cameras and synchronously adjusting the pose state of virtual cameras, the virtual and real viewpoints are dynamically aligned in spatial pose, thereby solving the problem of image drift and spatial misalignment caused by the asynchrony of virtual and real cameras in traditional AR systems; and by performing perspective projection rendering and keying on the fused 3D model based on the pose state. The process extracts only the pixels of underground facilities visible from the current viewpoint and removes irrelevant backgrounds, achieving visual penetration of obstructions and precise separation of target objects. Then, by merging the real image with the image of underground facilities, the invisible underground structure is presented intuitively in the real scene through a virtual-real overlay, giving ordinary cameras the ability to "see through the underground." Overall, the various steps work together to achieve stable, continuous, and high-fidelity AR perspective display of underground facilities in a wide-area urban surveillance camera system. This breaks through the core bottleneck of existing technologies that rely on markers, low-precision maps, or manual deduction, which makes it impossible to reliably present obstructed underground facilities in a real dynamic environment.

[0180] Example 3:

[0181] Figure 3 A structural diagram of a device for panoramic display of underground facilities, as described in this application, is presented. For example... Figure 3As shown, the equipment for panoramic display of underground facilities may include: processor N1, memory N2, data interface N3, and communication bus N4.

[0182] Wherein: processor N1, memory N2, and data interface N3 communicate with each other through communication bus N4; data interface N3 is used for data communication with other devices such as input devices or output devices; processor N1 is used to execute program N5, which can specifically execute the relevant steps in any of the above-mentioned methods for panoramic display of underground facilities.

[0183] Specifically, program N5 may include program code, which includes computer-executable instructions.

[0184] The processor N1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The device for panoramic viewing of underground facilities includes one or more processors, which may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0185] Memory N2 is used to store program N5. Memory N2 may include high-speed RAM memory and may also include non-volatile memory, such as at least one disk storage device.

[0186] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Furthermore, the embodiments in this application are not directed to any particular programming language.

[0187] Example 4:

[0188] This invention also provides a computer-readable storage medium storing at least one executable instruction that, when executed on a device / system for panoramic display of underground facilities, causes the device / system to perform a method for panoramic display of underground facilities as described in any of the above method embodiments.

[0189] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. Similarly, for the purpose of simplification and aiding understanding of one or more aspects of the invention, in the above description of exemplary embodiments of this application, various features of the embodiments are sometimes grouped together in a single embodiment, figure, or description thereof. The claims, which follow the detailed description, are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0190] Those skilled in the art will understand that the modules in the device of the embodiment can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiment can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components, except that at least some of such features and / or processes or units are mutually exclusive.

Claims

1. A method for panoramic display of underground facilities, characterized in that, include: The system acquires real-time image data from real cameras in the target area and synchronously adjusts the pose state of virtual cameras configured in the fused 3D model based on the pose data of the real cameras. The fused 3D model is obtained by spatial alignment of underground facility 3D models and ground 3D models. The underground facility 3D model and the ground 3D model are constructed using underground facility data and ground remote sensing mapping data of the target area, respectively. Based on the pose state, the fused 3D model is subjected to perspective projection rendering and keying to obtain an image of the underground facility that only contains the viewpoint corresponding to the current pose state. The real-world image data is fused with images of underground facilities to obtain a perspective image that blends reality and virtuality, which is then displayed.

2. The method for panoramic display of underground facilities as described in claim 1, characterized in that, The underground facility 3D model and the surface 3D model are constructed using underground facility data and surface remote sensing mapping data of the target area, respectively, including: Acquire raw underground infrastructure data and raw ground remote sensing mapping data of the target area; wherein, the raw underground infrastructure data includes any one or more of the following data combinations: CAD drawing data, BIM data, and GIS data; The original underground facility data and the original ground remote sensing mapping data are subjected to coordinate transformation and spatial registration processing, and the data with inconsistent resolution or scale are subjected to interpolation and normalization processing to obtain the corresponding underground facility data and ground remote sensing mapping data. Based on the underground facility data and the ground remote sensing mapping data, a three-dimensional model of the underground facility and a three-dimensional model of the ground are constructed respectively.

3. The method for panoramic display of underground facilities as described in claim 1, characterized in that, The fused 3D model is obtained by spatial alignment of the underground facility 3D model and the ground 3D model, including: Calculate the spatial deviation between the preset test points in the target area and the corresponding positions in the underground facility 3D model or the ground 3D model. If the spatial deviation exceeds the preset threshold, the vertex coordinates of the underground facility 3D model or the ground 3D model are finely adjusted until the accuracy requirements are met, and the original fused 3D model is obtained. The original fused 3D model is compressed to obtain the fused 3D model.

4. The method for panoramic display of underground facilities as described in claim 1, characterized in that, The step of synchronously adjusting the pose state of the virtual camera configured in the fused 3D model based on the pose data of the real camera includes: Based on the pose data corresponding to the current frame of the real camera, extract the installation position coordinates, horizontal rotation angle, pitch angle, roll angle and lens field parameters of the real camera corresponding to the current frame. The horizontal rotation angle, pitch angle, and roll angle are converted into rotation matrices and combined with the installation position coordinates to generate a three-dimensional transformation matrix, which is then converted into a quaternion representation. Based on the lens field parameters, the near clipping plane distance, far clipping plane distance, and view frustum angle are calculated, and the projection matrix of the virtual camera is constructed. Based on the quaternion representation and projection matrix, the virtual camera is adjusted to complete the synchronous adjustment of the pose state of the current frame.

5. The method for panoramic display of underground facilities as described in claim 4, characterized in that, The step of calculating the near clipping plane distance, far clipping plane distance, and view frustum angle based on the lens field parameters, and constructing the projection matrix of the virtual camera, includes: Based on the lens field parameters, calculate the near clipping plane distance, far clipping plane distance, and view frustum angle corresponding to the current frame; Calculate the difference between the view frustum angle of the current frame and the view frustum angle of the corresponding frame in the previous frame; If the difference is greater than a preset threshold, linear interpolation is performed on the view frustum angles of the current frame and the previous frame to obtain a transition angle value, and the projection matrix of the virtual camera for the current frame is constructed based on the transition angle value; otherwise, the projection matrix of the virtual camera for the current frame is directly constructed based on the view frustum angles of the current frame.

6. The method for panoramic display of underground facilities as described in claim 5, characterized in that, The step of adjusting the virtual camera based on the quaternion representation and projection matrix to synchronize the pose state of the current frame includes: The quaternion representation is filtered to obtain a smoothed quaternion; Based on the preset gimbal motion model and combined with the smoothed quaternion, predict the rotation quaternion of the virtual camera in the next frame. Based on the rotation quaternion and projection matrix, the pose state of the virtual camera in the current frame is adjusted.

7. The method for panoramic display of underground facilities as described in claim 4, characterized in that, The step of performing perspective projection rendering and keying on the fused 3D model based on the pose state to obtain an image of the underground facility containing only the viewpoint corresponding to the current pose state includes: Based on the pose state of the virtual camera, a model-view matrix is ​​constructed and multiplied with the projection matrix to calculate the transformation matrix; Based on the transformation matrix, each vertex in the fused 3D model is transformed, and its two-dimensional coordinates and corresponding spatial depth values ​​on the image plane are calculated. Determine whether the model object to which each vertex belongs is an underground facility and whether the corresponding spatial depth value is within a preset depth threshold; if the determination condition is met, retain the color value of the pixel corresponding to that vertex; otherwise, set the pixel to be completely transparent. Output the color values ​​of the pixels corresponding to all vertices to generate an image of the underground facility.

8. A system for panoramic display of underground facilities, characterized in that, include: Data acquisition and synchronization module, rendering and keying module, and fusion output module; The data acquisition and synchronization module is used to acquire real-time image data of real cameras in the target area, and synchronously adjust the pose state of virtual cameras configured in the fused 3D model based on the pose data of the real cameras. The fused 3D model is obtained by spatial alignment of underground facility 3D models and ground 3D models. The underground facility 3D model and the ground 3D model are constructed respectively using underground facility data and ground remote sensing mapping data of the target area. The rendering and chroma keying module is used to perform perspective projection rendering and chroma keying on the fused 3D model according to the pose state, so as to obtain an image of underground facilities that only contains the viewpoint corresponding to the current pose state. The fusion output module is used to fuse the real image data with the underground facility image to obtain a perspective image that blends reality and virtuality and then displays it.

9. A surround-view calibration device supporting multiple types of templates, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is loaded into the processor, it implements the steps of a look-around calibration method supporting multiple template types according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of a look-around calibration method supporting multiple types of templates according to any one of claims 1-7.

Images