Mobile platform digital twin based on dynamic video projection and AR fusion method and system

Abstract

The application discloses a mobile platform digital twin and AR fusion method and system based on dynamic video projection, and the method comprises the following steps: acquiring real-time video stream collected by a mobile platform and real-time pose data of the mobile platform, and dynamically solving a projection matrix of the mobile platform based on the real-time pose data; constructing a three-dimensional real scene model of a target area, projecting video images in the real-time video stream frame by frame to corresponding surface areas of the three-dimensional real scene model through the projection matrix, and obtaining a dynamic three-dimensional real scene scene with real-time video texture; performing depth pre-rendering based on the dynamic three-dimensional real scene scene, acquiring depth buffer information corresponding to a real scene, performing depth testing and occlusion judgment on virtual augmented information by using the depth buffer information, and obtaining an augmented reality fusion picture with correct virtual-actual occlusion relationship. Through the above steps, the application solves the problems of real-time accurate fusion and physically correct occlusion of dynamic video of a mobile platform and a three-dimensional real scene model.

Description

Technical Field

[0001] This invention relates to the field of digital twin and augmented reality technology, and in particular to a method and system for fusing digital twin and AR on a mobile platform based on dynamic video projection. Background Technology

[0002] Against the backdrop of deepening digital transformation, digital twin technology, by constructing a virtual mapping of the physical world, provides novel digital tools for various fields such as urban management, cultural heritage protection, emergency response, industrial inspection, and virtual film production. It has become one of the core technologies driving industrial digital upgrading and a current research hotspot in related fields. With the widespread adoption of mobile platforms such as drones and inspection robots, utilizing these platforms to collect real-time video and deeply integrating it with digital twin models has become an important development direction for improving the timeliness, interactivity, and practicality of digital twin systems. This effectively breaks the traditional disconnect between digital twin models and the real environment, further expanding the application boundaries of digital twin technology. However, current mainstream digital twin systems still rely on static or quasi-static models as their core, and their technological applications face many common challenges, making the limitations of existing technical solutions increasingly apparent. Specifically, the 3D model texture data of current digital twin systems typically comes from periodic aerial photography or manual surveying, with update cycles lasting months or even years. This results in a significant time lag between the digital twin model and the real environment, making it unable to reflect dynamic information such as road construction, temporary congestion, site weathering, and equipment status changes in a timely manner. This inherent defect of static twins makes digital twin technology difficult to play a practical role in scenarios requiring immediate response, such as emergency command and real-time inspection, often relegating it to a visually appealing but impractical visualization tool. In recent years, while some manufacturers have attempted to integrate real-time monitoring video into digital twin systems—for example, companies like Zhihui Yunzhou have proposed video twin solutions based on fixed monitoring cameras—by inverting the pixel coordinates of fixed-view monitoring images into 3D geographic coordinates to achieve target positioning and trajectory tracking, suitable for fixed camera scenarios such as urban security, traffic monitoring, and smart parks, such solutions cannot be directly applied to mobile applications represented by drones and ground robots. Video fusion scenarios on mobile platforms: Due to the large and rapid changes in camera pose, dynamic expansion of the screen coverage, and complex conditions such as lens zoom, high-speed rotation, and signal obstruction in videos captured during mobile platform movement, the shortcomings of existing fixed camera solutions are further highlighted: They lack the ability to dynamically calculate the real-time pose of the mobile platform, and cannot integrate multiple sensors such as RTK, IMU, and visual odometry to estimate pose in real time; they cannot maintain pixel-level alignment between video and model under high-speed motion, and the projection matrix cannot meet the dynamic update requirement of at least 30Hz, easily resulting in projection drift; virtual information overlay lacks physically correct occlusion based on model depth, often using simple screen coordinate overlay, causing virtual content to float, severely weakening immersion and decision reliability; at the same time, they cannot form a business closed loop from perception to processing, only stopping at visualization display, lacking linkage with AI recognition, rule engines, and execution mechanisms, making it difficult to fully realize the practical application value of digital twin technology.

[0003] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method for the fusion of digital twin and AR of mobile platform based on dynamic video projection, which aims to solve the technical problems in the prior art such as the inability of dynamic video of mobile platform to be accurately fused with three-dimensional real scene model in real time, the lack of physical correct occlusion of virtual information, and the inability to form closed-loop business processing.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: In a first aspect, the present invention provides a method for fusing digital twins and AR on a mobile platform based on dynamic video projection, wherein the method includes: The system acquires real-time video streams of the target area collected by the mobile platform and real-time pose data of the mobile platform, and dynamically calculates the projection matrix of the mobile platform based on the real-time pose data. A three-dimensional real-scene model of the target area is constructed, and video images from the real-time video stream are projected frame by frame onto the corresponding surface area of ​​the three-dimensional real-scene model through a projection matrix to obtain a dynamic three-dimensional real-scene scene with real-time video texture. Based on dynamic 3D real scene, depth pre-rendering is performed to obtain the depth buffer information corresponding to the real scene of the target area. The depth buffer information is used to perform depth testing and occlusion judgment on virtual augmented information to obtain augmented reality fusion image with correct virtual and real occlusion relationship.

[0006] In one implementation, the step of dynamically calculating the projection matrix of the mobile platform based on real-time pose data includes: The real-time pose data is converted into a camera extrinsic matrix in the world coordinate system, and then combined with the camera intrinsic matrix to generate a projection matrix.

[0007] In one implementation, the step of projecting video images from the real-time video stream frame by frame onto the corresponding surface area of ​​the 3D real-scene model using a projection matrix to obtain a dynamic 3D real-scene scene with real-time video textures includes: Obtain the projection matrix of the current rendering frame and the world coordinates of the fragment to be rendered on the surface of the 3D real scene model, and perform coordinate transformation on the fragment to be rendered based on the projection matrix and the world coordinates to generate the corresponding homogeneous clipping coordinates. A perspective division is performed on the homogeneous clipping coordinates to obtain normalized device coordinates, and the texture sampling coordinates of the fragment to be rendered in the real-time video stream are generated based on the normalized device coordinates. Based on the mapping relationship between the texture sampling coordinates and the position of the real-time video stream, the fragment to be rendered is subjected to texture rendering processing to generate a dynamic 3D real-world scene with real-time video texture.

[0008] The process of performing texture rendering on the fragment to be rendered based on the position mapping relationship between the texture sampling coordinates and the real-time video stream includes: If the texture sampling coordinates are within the effective texture mapping range of the real-time video stream, then the target color value is sampled from the real-time video stream based on bilinear interpolation or anisotropic filtering algorithm, and the target color value is used to replace the original preset texture of the fragment to be rendered for shading rendering.

[0009] In one implementation, the step of performing texture rendering processing on the fragment to be rendered based on the position mapping relationship between the texture sampling coordinates and the real-time video stream includes: If the texture sampling coordinates exceed the effective texture mapping range of the real-time video stream, the original preset texture of the fragment to be rendered is maintained for rendering.

[0010] In one implementation, the step of performing depth pre-rendering based on a dynamic 3D real-world scene to obtain depth buffer information corresponding to the real scene of the target area includes: Acquire the geometric model data of the dynamic 3D real scene and the corresponding camera frustum parameters; Configure the rendering pipeline according to the camera's frustum parameters, and configure a depth pre-write mode with depth writing enabled and color writing disabled. In depth pre-write mode, geometric model data is input into the rendering pipeline for rasterization processing, and the depth value of each fragment in the real scene in camera space is calculated. The calculated depth value is written into the depth buffer of the graphics processor to generate depth buffer information that represents the geometric distance distribution of the real scene in the target area.

[0011] In one implementation, the step of using depth buffer information to perform depth testing and occlusion judgment on virtual augmented information to obtain an augmented reality fusion image with correct virtual-real occlusion relationship includes: Obtain the vertex data of the virtual augmentation information to be rendered, and calculate the virtual depth value of each fragment of the virtual augmentation information in camera space based on the projection matrix; Read the real scene depth value stored at the corresponding pixel position in the depth buffer information; Perform a piecewise meta-depth comparison between the virtual depth value and the real scene depth value; If the virtual depth value is greater than the real scene depth value, it is determined that the fragment is occluded by the real scene and the rendering instruction of the fragment is discarded. If the virtual depth value is less than or equal to the real scene depth value, it is determined that the fragment is not occluded, and a color blending or direct overlay operation is performed. The color value of the fragment is written to the frame buffer, and the augmented reality blended image is output.

[0012] Secondly, embodiments of the present invention also provide a mobile platform digital twin and AR fusion system based on dynamic video projection, wherein the system includes: The data acquisition and projection matrix calculation module is used to acquire the real-time video stream of the target area collected by the mobile platform and the real-time pose data of the mobile platform, and dynamically calculate the projection matrix of the mobile platform based on the real-time pose data. The dynamic 3D real scene acquisition module is used to construct a 3D real scene model of the target area. The video images in the real-time video stream are projected frame by frame onto the corresponding surface area of ​​the 3D real scene model through a projection matrix to obtain a dynamic 3D real scene with real-time video texture. The augmented reality fusion image acquisition module is used to perform depth pre-rendering based on dynamic 3D real scene, obtain the depth buffer information corresponding to the real scene of the target area, and use the depth buffer information to perform depth testing and occlusion judgment on virtual augmented information to obtain augmented reality fusion image with correct virtual-real occlusion relationship.

[0013] Thirdly, embodiments of the present invention also provide a terminal, wherein the terminal includes a memory, a processor, and a mobile platform digital twin and AR fusion program based on dynamic video projection stored in the memory and executable on the processor. When the processor executes the mobile platform digital twin and AR fusion program based on dynamic video projection, it implements the steps of the mobile platform digital twin and AR fusion method based on dynamic video projection as described in any of the above schemes.

[0014] Fourthly, embodiments of the present invention also provide a computer-readable storage medium, wherein the computer-readable storage medium stores a mobile platform digital twin and AR fusion program based on dynamic video projection, wherein when the mobile platform digital twin and AR fusion program based on dynamic video projection is executed by a processor, it implements the steps of the mobile platform digital twin and AR fusion method based on dynamic video projection as described in any of the above schemes.

[0015] Beneficial Effects: This invention provides a method for fusing digital twins and AR (Augmented Reality) on a mobile platform based on dynamic video projection. Compared with existing technologies, this invention first acquires real-time video streams of the target area collected by the mobile platform and real-time pose data of the mobile platform. Based on the real-time pose data, it dynamically calculates the projection matrix of the mobile platform. The real-time pose data can accurately capture the position and posture changes of the mobile platform, and the dynamically calculated projection matrix can establish a stable correspondence between two-dimensional video images and three-dimensional space, laying a solid foundation for the accurate fusion of video and three-dimensional models, avoiding fusion deviations during the movement of the mobile platform, and ensuring the real-time performance and accuracy of the entire technical process. Next, a three-dimensional real-scene model of the target area is constructed. The video images in the real-time video stream are projected frame by frame onto the corresponding surface area of ​​the three-dimensional real-scene model through the projection matrix, resulting in a dynamic three-dimensional real-scene scene with real-time video textures. This process will... The real-time visual changes of the real scene are deeply integrated with the 3D model, breaking the limitations of traditional static 3D models. This allows the 3D scene to synchronously reflect the dynamic changes of the real environment, making it more realistic and intuitive, and providing a foundational carrier for subsequent virtual-real fusion that fits the real scene. Then, based on the dynamic 3D real scene, depth pre-rendering is performed to obtain the depth buffer information corresponding to the real scene in the target area. The depth buffer information is used to perform depth testing and occlusion judgment on the virtual augmented information, resulting in an augmented reality fusion image with correct virtual-real occlusion relationship. By simulating the depth level of the real scene through the depth buffer information, the occlusion logic between virtual information and the real scene can be accurately handled, avoiding the common problem of virtual-real occlusion confusion in existing technologies. This allows the virtual augmented information to be presented reasonably in accordance with the spatial level of the real scene, greatly improving the immersion and realism of the augmented reality experience. Through the organic connection of the above steps, this invention achieves accurate and natural fusion of real scene and virtual augmented information, effectively solving the pain points of existing augmented reality technologies such as large fusion deviation, unreasonable occlusion relationship, and insufficient scene dynamism. It significantly improves the quality of augmented reality fusion images and further expands the application space of augmented reality technology in various practical scenarios. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a specific implementation of the mobile platform digital twin and AR fusion method based on dynamic video projection provided in this embodiment of the invention.

[0017] Figure 2 This is an overall flowchart of the preferred embodiment of the mobile platform digital twin and AR fusion method based on dynamic video projection provided in the embodiments of the present invention.

[0018] Figure 3This is a schematic diagram illustrating the principle of dynamic calculation and updating of the projection matrix in the mobile platform digital twin and AR fusion method based on dynamic video projection provided in an embodiment of the present invention.

[0019] Figure 4 This is a schematic diagram illustrating the principle of virtual-real occlusion processing based on a depth buffer in the mobile platform digital twin and AR fusion method based on dynamic video projection provided in an embodiment of the present invention.

[0020] Figure 5 This is a schematic diagram illustrating the application of the mobile platform digital twin and AR fusion method based on dynamic video projection provided in an embodiment of the present invention in an urban emergency command scenario.

[0021] Figure 6 This is a schematic diagram illustrating the application of the mobile platform digital twin and AR fusion method based on dynamic video projection provided in the embodiment of the present invention in the restoration of cultural and tourism heritage sites.

[0022] Figure 7 This is a schematic diagram illustrating the application of the mobile platform digital twin and AR fusion method based on dynamic video projection provided in this embodiment of the invention in a power inspection scenario.

[0023] Figure 8 This is a principle block diagram of a mobile platform digital twin and AR fusion system based on dynamic video projection provided in an embodiment of the present invention.

[0024] Figure 9 This is a block diagram of the system modules in the mobile platform digital twin and AR fusion method based on dynamic video projection provided in an embodiment of the present invention.

[0025] Figure 10 This is a diagram showing the system data flow and interface relationships in the mobile platform digital twin and AR fusion method based on dynamic video projection provided in an embodiment of the present invention.

[0026] Figure 11 This is a block diagram illustrating the internal structure of a terminal provided in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and effects of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0028] Against the backdrop of deepening digital transformation, digital twin technology, by constructing a virtual mapping of the physical world, provides novel digital tools for various fields such as urban management, cultural heritage protection, emergency response, industrial inspection, and virtual film production. It has become one of the core technologies driving industrial digital upgrading and a current research hotspot in related fields. With the widespread adoption of mobile platforms such as drones and inspection robots, utilizing these platforms to collect real-time video and deeply integrating it with digital twin models has become an important development direction for improving the timeliness, interactivity, and practicality of digital twin systems. This effectively breaks the traditional disconnect between digital twin models and the real environment, further expanding the application boundaries of digital twin technology. However, current mainstream digital twin systems still rely on static or quasi-static models as their core, and their technological applications face many common challenges, making the limitations of existing technical solutions increasingly apparent. Specifically, the 3D model texture data of current digital twin systems typically comes from periodic aerial photography or manual surveying, with update cycles lasting months or even years. This results in a significant time lag between the digital twin model and the real environment, making it unable to reflect dynamic information such as road construction, temporary congestion, site weathering, and equipment status changes in a timely manner. This inherent defect of static twins makes digital twin technology difficult to play a practical role in scenarios requiring immediate response, such as emergency command and real-time inspection, often relegating it to a visually appealing but impractical visualization tool. In recent years, while some manufacturers have attempted to integrate real-time monitoring video into digital twin systems—for example, companies like Zhihui Yunzhou have proposed video twin solutions based on fixed monitoring cameras—by inverting the pixel coordinates of fixed-view monitoring images into 3D geographic coordinates to achieve target positioning and trajectory tracking, suitable for fixed camera scenarios such as urban security, traffic monitoring, and smart parks, such solutions cannot be directly applied to mobile applications represented by drones and ground robots. Video fusion scenarios on mobile platforms: Due to the large and rapid changes in camera pose, dynamic expansion of the screen coverage, and complex conditions such as lens zoom, high-speed rotation, and signal obstruction in videos captured during mobile platform movement, the shortcomings of existing fixed camera solutions are further highlighted: They lack the ability to dynamically calculate the real-time pose of the mobile platform, and cannot integrate multiple sensors such as RTK, IMU, and visual odometry to estimate pose in real time; they cannot maintain pixel-level alignment between video and model under high-speed motion, and the projection matrix cannot meet the dynamic update requirement of at least 30Hz, easily resulting in projection drift; virtual information overlay lacks physically correct occlusion based on model depth, often using simple screen coordinate overlay, causing virtual content to float, severely weakening immersion and decision reliability; at the same time, they cannot form a business closed loop from perception to processing, only stopping at visualization display, lacking linkage with AI recognition, rule engines, and execution mechanisms, making it difficult to fully realize the practical application value of digital twin technology.

[0029] To address the aforementioned issues, this embodiment provides a method for fusing digital twins and AR on a mobile platform based on dynamic video projection. Specifically, this embodiment first acquires real-time video streams of the target area collected by the mobile platform, along with the mobile platform's real-time pose data. Based on the real-time pose data, the projection matrix of the mobile platform is dynamically calculated. The real-time pose data accurately captures the position and posture changes of the mobile platform, while the dynamically calculated projection matrix establishes a stable correspondence between the two-dimensional video image and the three-dimensional space, laying a solid foundation for the subsequent accurate fusion of the video and the 3D model. This avoids fusion deviations during the mobile platform's movement, ensuring the real-time performance and accuracy of the entire process. Next, a 3D real-world model of the target area is constructed. The video images from the real-time video stream are projected frame by frame onto the corresponding surface areas of the 3D real-world model using the projection matrix, resulting in a dynamic 3D model with real-time video textures. The process of creating a real-world scene combines real-time visual changes of the real scene with the depth of the 3D model, breaking the limitations of traditional static 3D models. This allows the 3D scene to synchronously reflect the dynamic changes of the real environment, making it more realistic and intuitive, and providing a foundational carrier for subsequent virtual-real fusion that fits the real scene. Then, based on the dynamic 3D real-world scene, depth pre-rendering is performed to obtain the depth buffer information corresponding to the real scene in the target area. The depth buffer information is used to perform depth testing and occlusion judgment on the virtual augmented information, resulting in an augmented reality fusion image with correct virtual-real occlusion relationship. By simulating the depth level of the real scene through the depth buffer information, the occlusion logic between virtual information and the real scene can be accurately handled, avoiding the common problem of virtual-real occlusion confusion in existing technologies. This allows the virtual augmented information to be presented reasonably in accordance with the spatial level of the real scene, greatly improving the immersion and realism of the augmented reality experience. By organically linking the above steps, this invention achieves accurate and natural fusion of real scenes and virtual augmented information, effectively solving the pain points of existing augmented reality technologies such as large fusion deviations, unreasonable occlusion relationships, and insufficient scene dynamism. It significantly improves the quality of augmented reality fusion images and further expands the application space of augmented reality technology in various real-world scenarios.

[0030] The mobile platform digital twin and AR fusion method based on dynamic video projection provided in this embodiment can be applied to smart terminals, such as... Figure 1 As shown, the specific steps include the following: Step S100: Obtain the real-time video stream of the target area collected by the mobile platform and the real-time pose data of the mobile platform, and dynamically calculate the projection matrix of the mobile platform based on the real-time pose data.

[0031] In this embodiment, as Figure 2As shown, firstly, S1 acquires real-time data, specifically: acquiring real-time video streams collected by the mobile platform (drone or ground robot) during its movement in the target area, as well as real-time pose data provided by the positioning sensors and inertial measurement units mounted on the mobile platform. To ensure the continuity and accuracy of positioning, this embodiment adopts a fusion positioning strategy of RTK (Real-Time Dynamic Differential) and Visual Inertial Odometry (VIO). In open areas (such as squares, farmland, and open spaces of historical sites), RTK centimeter-level positioning is the primary method, with RTK horizontal accuracy reaching 1cm+1ppm and vertical accuracy reaching 2cm+1ppm, and an output frequency of 100Hz. In areas where RTK signals are obstructed (such as urban canyons, under tree canopies, and inside buildings), it automatically switches to Visual Inertial Odometry, using image feature matching (ORB, SuperPoint, etc.) between consecutive frames and the IMU. The relative pose is calculated through integration, achieving a short-term positioning accuracy better than 20cm and a drift rate less than 1% / min. Two positioning sources are fused using Extended Kalman Filter (EKF) or Unscented Kalman Filter (UKF) to output a smooth, continuous six-DOF pose (position X, Y, Z; attitude angles Roll, Pitch, Yaw). The output frequency is synchronized with the video frame rate (typically 30Hz). All data (video frames, RTK coordinates, IMU angular velocity / acceleration) are accompanied by a unified timestamp (based on GPS time or PTP precise time protocol) to ensure frame-level synchronization with a time synchronization error of less than 1ms. Subsequently, the S2 dynamic calculation of the projection matrix is ​​performed. Specifically, based on real-time pose data and combined with a preset coordinate system definition and pose transformation rules, the projection matrix of the mobile platform is dynamically calculated. The projection matrix describes the spatial mapping relationship between the camera image coordinate system and the 3D reality model world coordinate system, serving as the mathematical basis for subsequent texture mapping. Figure 3 As shown, the coordinate systems are clearly defined: the world coordinate system (W) is consistent with the coordinate system of the 3D reality model, usually using the CGCS2000 or WGS84 geocentric-ground-fixed coordinate system (ECEF) projected onto the northeast-sky (ENU) local coordinate system, in meters; the camera coordinate system (C) has its origin at the camera's optical center, the Z-axis pointing towards the scene along the optical axis, the X-axis to the right, and the Y-axis downwards (conforming to image coordinate system conventions); the image coordinate system (I) is a two-dimensional pixel coordinate system, with its origin at the top left corner of the image, the u-axis to the right, and the v-axis downwards, in pixels. During pose transformation, the latitude and longitude provided by RTK are first converted... The elevation h is converted to three-dimensional coordinates P_w = (X_w, Y_w, Z_w) in the world coordinate system. The calculation formula is = ×π / 180, λ′=λ×π / 180, X_w = (N + h) cos cosλ′、Y_w = (N +h) cos sinλ′, Z_w = (N (1-e²) ​​+ h) sin Where N = a / √(1-e² sin²) Let be the radius of curvature of the ramusoidal circle, a be the semi-major axis of the ellipsoid (a = 6378137m in WGS84), and e be the first eccentricity. To obtain a local coordinate system suitable for rendering a digital twin scene, a reference point O in the scene is selected (e.g., the drone takeoff point or the center of the scenic area), and its geodetic coordinates are ( , , ), calculate its ECEF coordinates according to the above formula ( , , The local coordinates (Xenu, Yenu, Zenu) of any point P in the northeastern sky (ENU) are calculated by the following formula:

[0032] The rotation matrix R transforms the ECEF difference vector into the ENU direction:

[0033] The calculated (Xenu, Yenu, Zenu) is the world coordinate system (ENU) used in this invention, where Xenu points east, Yenu points north, and Zenu points to the sky. The attitude angles (Roll φ_r, Pitch θ, Yaw ψ) provided by the IMU are then converted into rotation matrices R_cw = R_z (ψ) R_y (θ) R_x (φ_r) in ZYX order (from world coordinate system to camera coordinate system), and the translation vector t_cw = -R_cw * C_w is calculated, completing the conversion from real-time pose to camera extrinsic parameter matrix. Simultaneously, the camera intrinsic parameter matrix K = [[ fx, 0, cx, 0 ], [ 0, fy, cy, 0 ], [ 0, 0,1, 0 ]] is determined, where fx = f / dx and fy = f / dy are the focal lengths in pixels, f is the physical focal length (mm), dx and dy are the pixel sizes (mm / pixel), and (cx, cy) are the principal point coordinates (usually close to the image center). For zoom lenses, fx and fy change with the focal length. The K matrix is ​​updated in real time by reading the lens encoder or based on image autofocus information. To improve projection accuracy, this embodiment also models and corrects camera lens distortion using the Brown-Conrady distortion model, which includes radial distortion (k1,k2,k3) and tangential distortion (p1,p2). The world coordinate system point P_w is transformed to the camera coordinate system: P_c = [R_cw | t_cw]* Let P_w be a coordinate system, and P_c be an array of coordinates (X_c, Y_c, Z_c)^T. The normalized plane projection is: x = X_c / Z_c, y = Y_c / Z_c. For any point in the world coordinate system, first transform it to the camera coordinate system and project it onto the normalized plane to obtain ideal normalized coordinates (x, y). Then apply the distortion correction formula x_corr = x (1 + k1r² + k2r)^T. 4 + k3 r 6 ) + 2 p1 xy + p2 (r² + 2x²), y_corr = y (1 + k1 r² + k2 r 4 + k3 r 6The correction is completed by adding p1 (r² + 2y²) + 2 p2 xy, where r² = x² + y². The corrected coordinates (x_corr, y_corr) are the normalized coordinates after considering distortion. These are then converted to pixel coordinates using the camera intrinsic matrix K: u = fx * x_corr + cx, v = fy * y_corr + cy. The above (u, v) are the pixel coordinates of point P_w on the original distorted video frame. It should be noted that if distortion is ignored (or the image has been corrected beforehand), the projection matrix P = K [R_cw| t_cw] can be used directly for calculation, which is a 3×4 matrix. For any point P_w = (X, Y, Z, 1)^T in the world coordinate system, its homogeneous coordinates in the image are p = P * P_w. Then, the pixel coordinates (u, v) = (p_x / p_w, p_y / p_w) are obtained through perspective division. The projection matrix is ​​updated frame by frame at a frequency no less than the video frame rate (typically 30Hz). When the camera zooms, the focal length value after zooming is obtained in real time, the K matrix is ​​updated, and the projection matrix is ​​recalculated. The update delay of the projection matrix meets the real-time synchronization requirements, ensuring strict synchronization with the video frames. The entire process achieves accurate spatial mapping between the real-time video stream of the mobile platform and the coordinate system of the 3D real scene, providing reliable mathematical support and data guarantee for subsequent applications such as texture mapping.

[0034] Step S200: Construct a three-dimensional real-scene model of the target area, and project the video images in the real-time video stream frame by frame onto the corresponding surface area of ​​the three-dimensional real-scene model through a projection matrix to obtain a dynamic three-dimensional real-scene scene with real-time video texture.

[0035] In this embodiment, firstly, as Figure 2As shown, step S3 constructs a 3D real-world model of the target area. This step establishes a 3D geometric base consistent with the real physical space, restoring the actual terrain, building outlines, and spatial structure of the target area, providing a stable and accurate 3D carrier for subsequent precise mapping of video textures. Next, step S4 performs texture mapping projection, which involves projecting video images from the real-time video stream frame by frame onto the corresponding surface area of ​​the 3D real-world model using a projection matrix. The projection matrix establishes the coordinate transformation relationship between the 2D video images and the 3D model surface, achieving spatial alignment and geometric correction between the video image and the model surface. Frame-by-frame projection ensures the real-time performance of the video image. The continuity allows video information to be precisely aligned with the corresponding positions of the model, ultimately resulting in a dynamic 3D real-world scene with real-time video textures. This step integrates the static 3D model with dynamic video information, giving the 3D scene a real-time visualization effect. Overall, this process achieves digital replication of physical space and real-time visualization of dynamic information through the fusion of 3D real-world modeling and real-time video projection. It preserves the true 3D structure of the target area while simultaneously overlaying real-time video footage, providing an intuitive and dynamic display of the scene. This offers highly realistic and real-time 3D dynamic presentation capabilities for applications such as scene monitoring, real-world visualization, and dynamic monitoring.

[0036] Specifically, step S200 includes the following steps: Step S201: Construct a 3D reality model of the target area; Step S202: Obtain the projection matrix of the current rendering frame and the world coordinates of the fragment to be rendered on the surface of the 3D real scene model, and perform coordinate transformation on the fragment to be rendered based on the projection matrix and the world coordinates to generate the corresponding homogeneous clipping coordinates. Step S203: Perform perspective division on the homogeneous clipping coordinates to obtain normalized device coordinates, and generate the texture sampling coordinates of the fragment to be rendered in the real-time video stream based on the normalized device coordinates; Step S204: Based on the position mapping relationship between the texture sampling coordinates and the real-time video stream, perform texture rendering processing on the fragment to be rendered to generate a dynamic three-dimensional real-world scene with real-time video texture.

[0037] In one implementation, firstly, as Figure 2As shown, S3 pre-constructs a high-precision 3D real-world model of the target area based on oblique photogrammetry or LiDAR technology. This model includes a complete geometric mesh (triangular facets or point clouds) and realistic textures. When using oblique photogrammetry for modeling, high overlap images with a forward overlap of ≥80% and a lateral overlap of ≥70% are acquired by UAV flying in five directions (orthogonal + four tilt angles: forward, backward, left, and right). Then, aerial triangulation is performed using Structure of Motion (SfM) and Multi-View Stereo Matching (MVS) algorithms to generate dense point clouds. After point cloud filtering, mesh reconstruction, and texture mapping, a 3D model in OSGB, OBJ, or 3D Tiles format is output, ensuring that the geometric accuracy of the model is better than 5 cm and the texture resolution is better than 2 cm / pixel. When using LiDAR for modeling, point cloud data with a point density of ≥100 points / square meter is acquired by UAV or ground-based laser scanner. After point cloud registration, filtering, classification, and triangulation, a mesh model is generated. Texture mapping is then completed by combining synchronously acquired images. In addition, to support real-time rendering of large-scale scenes, 4-6 layers are built for the model. The system employs a multi-level hierarchy of details (LOD) structure, with LOD0 preserving original precision for close-up viewing and LOD4-5 simplifying the model to a fraction of a triangle for long-distance viewing. It dynamically loads geometric data of varying precision based on viewing distance, significantly reducing rendering load. Subsequently, it acquires the projection matrix of the current rendering frame and the world coordinates of the fragments to be rendered on the surface of the 3D real-world model. Based on this projection matrix and world coordinates, it performs coordinate transformation on the fragments to generate corresponding homogeneous clip coordinates. This step converts model points in 3D world space to clip space. Then, it performs perspective division on the homogeneous clip coordinates to obtain normalized device coordinates. A viewport transformation is then performed on the normalized device coordinates to convert them to screen pixel coordinates. Based on the converted screen pixel coordinates, it generates texture sampling coordinates for the fragments to be rendered in the real-time video stream, achieving precise mapping from 3D space points to 2D video texture coordinates. Finally, based on the... The text describes the mapping relationship between texture sampling coordinates and the position of the real-time video stream. Texture rendering is performed on the fragments to be rendered to generate a dynamic 3D real-world scene with real-time video textures. The specific rendering rules are as follows: If the texture sampling coordinates are within the effective texture mapping range of the real-time video stream, a target color value is sampled from the real-time video stream based on bilinear interpolation or anisotropic filtering algorithms. This target color value is then used to replace the original preset texture of the fragment to be rendered for shading. If the texture sampling coordinates exceed the effective texture mapping range of the real-time video stream, the original preset texture of the fragment to be rendered is maintained for rendering. In this embodiment, if the camera has lens distortion, after obtaining the ideal UV coordinates, (U,V) needs to be converted to pixel coordinates (u_corr, v_corr) on the real distorted image according to the distortion correction formula before texture sampling. For simplicity, the following uses a video frame with corrected distortion as an example.The texture mapping algorithm is calculated using the formula (u, v, w)^T = P * (P_w, 1)^T, where U = u / w, V = v / w, P is the projection matrix of the current frame, and P_w is the world coordinate of the fragment to be rendered. If both U and V are within the range of [0,1], color values ​​are sampled from the video frame texture; otherwise, the original preset texture of the model is retained. Bilinear interpolation or anisotropic filtering is used to improve sampling quality. Each frame of the real-time video stream is then used as a dynamic texture. Using the projection matrix calculated above, the video image is projected frame by frame onto the corresponding surface area of ​​the 3D real-world model. This mapping process is executed during each frame rendering. Relying on the characteristic that the projection matrix updates frame by frame with the pose of the mobile platform, the coverage area of ​​the video texture on the model surface dynamically changes, simulating the visual effect of "video images being projected onto the model surface like light." This achieves dynamic updates where the corresponding area of ​​the model surface is covered by real-time video when the mobile platform flies over an area, and the area recovers its original texture after the platform flies away. To avoid GPU issues caused by calculating the projection matrix pixel by pixel... When the load is too high, a performance optimization strategy is implemented simultaneously. The UV coordinates of the model vertices are pre-calculated in the vertex shader, interpolated, and then passed to the fragment shader. The GPU's texture caching and asynchronous transmission mechanism are used to reduce CPU-GPU data exchange. View frustum culling is performed on model areas far from the camera's field of view to avoid invalid calculations. The overall process involves high-precision 3D real-scene model construction, coordinate transformation, dynamic texture mapping, real-time rendering, and performance optimization. Ultimately, it achieves 3D real-scene scene rendering that combines high-precision static base and dynamic real-time video textures. This ensures the geometric and texture accuracy of the 3D model in the target area, achieves real-time fitting and rendering of dynamic video textures, and takes into account the rendering efficiency and smoothness of large-scale scenes.

[0038] Step S300: Perform depth pre-rendering based on the dynamic 3D real scene, obtain the depth buffer information corresponding to the real scene of the target area, and use the depth buffer information to perform depth testing and occlusion judgment on the virtual augmented information to obtain an augmented reality fusion image with correct virtual-real occlusion relationship.

[0039] In this embodiment, as Figure 2As shown, S5 performs virtual-real occlusion processing. First, it performs depth pre-rendering based on the dynamic 3D real scene, presenting the real scene in a dynamic 3D form in advance, accurately capturing the spatial details and dynamic features of the scene, providing a clear and accurate foundation for subsequent depth information acquisition, and avoiding the impact of scene blur or missing features on subsequent processing effects. Then, it obtains the depth buffer information corresponding to the real scene in the target area, extracts the spatial depth data of each object in the real scene, clearly defines the front and back hierarchy of different objects, and provides a reliable basis for the spatial comparison between virtual augmented information and the real scene. Afterwards, it uses the depth buffer information to perform depth testing and occlusion judgment on the virtual augmented information. By comparing the depth parameters of the virtual information and the real scene, it naturally distinguishes the parts of the virtual information that need to be displayed and those that are occluded by real objects, avoiding problems such as virtual information penetrating real objects and chaotic occlusion logic. Finally, it obtains an augmented reality blended image with the correct virtual-real occlusion relationship. Through this series of coherent operations, it achieves seamless connection between virtual augmented information and the real scene, making the blended image more in line with human visual habits, greatly improving the realism and immersion of the augmented reality experience, and ensuring that the virtual content blends into the real scene naturally and without abruptness.

[0040] Specifically, step S300 includes the following steps: Step S301: Obtain the geometric model data of the dynamic 3D real scene and the corresponding camera frustum parameters; configure the rendering pipeline according to the camera frustum parameters, and configure the depth pre-write mode with depth writing enabled and color writing disabled; in the depth pre-write mode, input the geometric model data into the rendering pipeline for rasterization processing, and calculate the depth value of each fragment of the real scene in the camera space; write the calculated depth value into the depth buffer of the graphics processor to generate depth buffer information representing the geometric distance distribution of the real scene; Step S302: Obtain vertex data of the virtual augmented information to be rendered, and calculate the virtual depth value of each fragment of the virtual augmented information in camera space according to the projection matrix; read the real scene depth value stored at the corresponding pixel position in the depth buffer information; compare the virtual depth value with the real scene depth value on a fragment-by-fragment basis; if the virtual depth value is greater than the real scene depth value, it is determined that the fragment is occluded by the real scene and the rendering instruction of the fragment is discarded; if the virtual depth value is less than or equal to the real scene depth value, it is determined that the fragment is not occluded, perform color blending or direct overlay operation, write the color value of the fragment into the frame buffer, and output the augmented reality blended image.

[0041] In one implementation, such as Figure 4As shown, to achieve a natural fusion of virtual augmented information and real 3D scenes, eliminate the visual defect of "floating" virtual content in traditional AR, and enhance the immersiveness and realism of virtual-real fusion, the geometric model data and corresponding camera frustum parameters of the dynamic 3D scene are first acquired. The camera frustum parameters provide the foundation for subsequent configuration of the rendering pipeline and accurate calculation of fragment depth values, while the geometric model data is the core basis for reconstructing the geometric distance distribution of the real scene. Next, the rendering pipeline is configured according to the camera frustum parameters, and a depth pre-write mode is configured with depth writing enabled and color writing disabled. The core function of this mode is to record only the depth information of the real scene without displaying its color, laying the foundation for subsequent occlusion judgment of virtual content. In pre-write mode, geometric model data is input into the rendering pipeline for rasterization. Rasterization converts the 3D geometric model into pixels in screen space, thereby accurately calculating the depth values ​​of each fragment in the real scene in camera space. This depth value directly reflects the distance from the corresponding object in the real scene to the camera. The calculated depth values ​​are then written to the graphics processor's depth buffer, generating depth buffer information that represents the geometric distance distribution of the real scene. The depth buffer, as a memory area in the rendering pipeline storing the depth values ​​of each fragment, stores real scene depth information that serves as a core reference for subsequent virtual content occlusion detection. This process corresponds to... Figure 4 The first rendering pass (depth pre-writing) process: The 3D reality model (including dynamic textures) is input into the pipeline, ultimately generating a depth buffer that follows the "near → bright, far → dark" rule. This step only renders the 3D reality model (including its dynamic texture mapping) and writes it to the depth buffer, but not the color, ensuring that the depth buffer only retains the geometric depth information of the real scene. After the depth pre-writing is completed, the second rendering pass (virtual content) process begins: When rendering each virtual fragment, a depth test is performed: the depth value of the virtual fragment (i.e., the Z value of the virtual object in camera space) is compared with the depth value of the real model stored at the corresponding position in the depth buffer. Specifically, the process involves: acquiring vertex data of the virtual augmented information to be rendered (such as AR tags / models, historical restoration models, warning signs, etc.); sequentially transforming the coordinates of the virtual vertices using a projection matrix to obtain the depth information of each virtual vertex in camera space; then, through rasterization processing in the rendering pipeline, interpolating the vertex depths within primitives to obtain the virtual depth value of each fragment in camera space, which represents the distance from the corresponding position of the virtual fragment to the camera; next, reading the real scene depth value stored at the corresponding fragment position in the depth buffer information; performing a depth test on a fragment-by-fragment basis (virtual depth > depth buffer value?); comparing the virtual depth value with the real scene depth value to determine the occlusion of the virtual content: if the virtual depth value is greater than the real scene depth value (corresponding to...), the occlusion is determined. Figure 4The "Yes" branch indicates that the virtual object is behind the real model. In this case, it is determined that the fragment is occluded by the real scene and a discard (occlusion) operation is performed, i.e., "virtual depth > scene depth → discard (occluded)", to prevent virtual content from penetrating the real scene; if the virtual depth value is less than or equal to the real scene depth value (corresponding to Figure 4 If the branch indicates "No," then the fragment is determined to be unoccluded, and a display (foreground) operation is performed, i.e., "virtual depth < scene depth → display (unoccluded)." For opaque virtual objects, a direct overlay operation is performed, writing the fragment's color value to the frame buffer; for semi-transparent virtual content (such as smoke or halos), a color blending operation is performed, and the transparency is dynamically adjusted according to the depth difference to ensure a natural transition at the occlusion edges, thus completing the rendering of the virtual augmented information. After being superimposed with the pre-written real scene content, the augmented reality blended image is output. In other words, the results of the two branches are finally combined and entered into... Figure 4 The final virtual-real fusion image is generated in the fusion output stage. The core advantage of this entire processing flow lies in the fact that it achieves physically correct occlusion of each fragment through a depth buffer based on a 3D real-scene model. For example, when a drone flies over a building, the AR tag indicating that people are trapped behind the building will be occluded by the building itself. Only when the drone changes its view and the building is no longer obscuring the view will the tag reappear. This fundamentally solves the visual defect of virtual content "floating" above the screen in traditional AR, significantly improving the immersion and realism of virtual-real fusion. The core principle of the depth buffer is that when rendering opaque objects, the GPU automatically performs a depth test: if the depth value of a new fragment is less than the depth value already stored in the buffer (indicating that the fragment is closer to the camera), the buffer is updated and a color is written; otherwise, the fragment is discarded. This embodiment is based on this principle and achieves natural occlusion fusion of virtual content and real scene by rendering in two passes.

[0042] In one implementation, the method of the present invention further includes: an AI recognition and coordinated processing implementation method, such as... Figure 2As shown, S6 The system combines AI recognition, tagging, and coordinated response. Specifically, an AI model deployed on a mobile platform or cloud performs real-time target detection and recognition on the live video stream. The recognition results are converted into spatial coordinates on a 3D real-world model using a projection matrix. Virtual tags or warning signs are automatically generated at the corresponding locations. Furthermore, based on a preset rule engine, coordinated response instructions are triggered according to the recognition results, forming a complete business loop of "perception-recognition-projection-decision-response," effectively improving the real-time performance and accuracy of scene handling. The AI ​​recognition process uses lightweight deep learning models (such as YOLOv8n, YOLOv9, RT-DETR, etc.) deployed at the edge (mobile platform onboard computer) or in the cloud, enabling efficient real-time target detection. Typical targets include trapped personnel, vehicles, flooded areas, fire smoke, equipment defects (such as broken insulators), cracks in cultural relics, and abnormal tourist behavior (such as climbing or touching). The detection output includes target category, confidence level, and bounding box (image pixel coordinates), providing accurate data support for subsequent coordinate transformation. In the coordinate transformation stage, the image coordinates (u, ...) of the center point of the detection box (or key point of the target) are converted into coordinates. v) The projection matrix is ​​transformed into 3D real-world model spatial coordinates (X_w, Y_w, Z_w) through inverse transformation. Specifically, the ray equation is solved: P_w = R_cw^T * ( (K^{-1} * (u,v,1)^T) * d - t_cw ), where d is the depth (the depth d is the Z value in the camera coordinate system, which can be obtained from the depth buffer value through inverse projection transformation). This equation enables precise mapping between image coordinates and 3D real-world coordinates, ensuring the accuracy of virtual label positioning. Subsequently, AR information is generated, and virtual labels (text, icons, 3D models) are generated at the transformed spatial coordinates. The rendering of the labels also follows the occlusion handling rules to ensure the rationality and readability of the label display.In the rule engine and coordinated response phase, preset rules are defined using JSON or a graphical rule language, for example: {"condition": "Water depth > 40cm AND Personnel trapped", "action": [{"type": "drone_hail", "content": "Rescue personnel have departed, please remain calm"}, {"type": "dispatch", "target": "firefighter", "location": "Trapped coordinates"},{"type": "traffic_signal", "command": "green_wave", "route": "Rescue route"}, {"type": "pump_station", "command": "start"}]}. These preset conditions are evaluated in real time, and corresponding actions are automatically executed when the conditions are met, enabling rapid response to various emergencies and ensuring personnel safety and the orderly operation of the system.

[0043] The following specific embodiments illustrate the detailed implementation of the present invention, fully presenting the workflow, technical details, and application effects of the present invention, so that those skilled in the art can clearly understand and implement the present invention.

[0044] Example 1: Urban emergency command scenario (flood relief), such as Figure 5 As shown, this embodiment uses a drone as a mobile platform to illustrate the complete workflow of the present invention in urban emergency command. The overall process can be summarized into three core stages: "drone data collection → digital twin model fusion → large-screen virtual-real fusion display and coordinated response," corresponding to... Figure 5The system encompasses the entire chain of logic from on-site data acquisition to the closed-loop response from the command center. A city experienced continuous heavy rainfall, causing rapid flooding in low-lying areas, traffic congestion, and multiple vehicle stalls, with some people trapped on car roofs. The city command center deployed a high-precision 3D real-world model of the entire city (covering an area of ​​50 km², with an accuracy of 5 cm and LOD hierarchical storage) built using oblique photogrammetry. Drones (such as the DJI M300 RTK or similar models) were deployed in a nearby automated hangar. Upon receiving an emergency dispatch command, they automatically took off, flying to the incident area at a speed of 15 m / s and a flight altitude of 50 m. The drone was equipped with an RTK module (supporting Qianxun Location or provincial CORS services), an IMU (BMI088 or ADIS16470), and a 4K visible light camera (30fps, resolution 3840×2160). It acquired a real-time video stream at 30fps and simultaneously transmitted RTK coordinates (longitude, latitude, and elevation, with an accuracy of 1 cm + 1 ppm) and IMU data back at a frequency of 100 Hz. Attitude angles (pitch, roll, yaw, accuracy 0.1°) are measured to acquire real-time video and pose data of urban flooding scenes, including water accumulation, stalled vehicles, and trapped individuals. When flying over densely built-up areas, the RTK signal is briefly lost (lasting approximately 5 seconds). The system automatically switches to visual-inertial odometry mode, where the onboard visual sensor (global shutter, 1280×720 resolution) acquires image sequences. These images are fused with IMU data using extended Kalman filtering to calculate relative pose. Short-term positioning accuracy is better than 20cm, ensuring continuous updates to the projection matrix. Upon RTK signal recovery, the system automatically switches back to RTK mode and uses closed-loop detection to eliminate accumulated VIO errors. All data is transmitted in real-time via a 5G network (50Mbps uplink bandwidth) to the edge server (configured with an NVIDIA A10 GPU and 64GB of RAM) at the city command center. After receiving the data transmitted back from the UAV, the ground server converts the RTK / IMU fused pose into the camera pose in the world coordinate system of the 3D real scene model. The camera intrinsic parameters are obtained in advance through checkerboard calibration: fx=2850, fy=2852, cx=1920, cy=1080, distortion coefficients k1=-0.12, k2=0.05, k3=0, p1=p2=0. The projection matrix P=K [R|t] is calculated and updated frame by frame at a frequency of 30fps. Each frame of video image is projected onto the corresponding ground surface and building surface of the city digital twin model containing buildings and roads through this matrix, completing the data fusion processing of the "projection + AI recognition" stage.On the command center's large screen (4K resolution, 120-inch size), a virtual-real fusion image was generated. The waterlogged area of ​​the 3D digital twin model displayed real-time water ripples and reflections captured by drones. The video texture of the stalled vehicle was precisely overlaid on the corresponding position of the model, forming a dynamic effect of "video as texture". The projection registration error was so low that it was imperceptible to the naked eye. In the image, the video of the waterlogged area was projected as a real-time video projection texture, and at the same time, AI-generated AR tags were superimposed, marking key information such as "[AR] 2 people trapped" and "[AR] vehicle stalled", and a dashed line rescue route guide was presented in the image. It should be noted that the AI ​​recognition and coordinated response module in this embodiment is an optional feature and can be selectively enabled according to application needs in actual deployment. The following description uses a scenario where this module is enabled as an example. A deep learning model (YOLOv9-C or a similar model, input resolution 640×640, real-time inference) deployed on an edge server performs real-time analysis of the video stream, detecting trapped personnel (confidence 0.92), stalled vehicles (confidence 0.88), and water accumulation boundaries (through a semantic segmentation network). The image coordinates of the center point of the detection box are converted into 3D real-world model spatial coordinates (longitude, latitude, and elevation) through a projection matrix, and AR tags are generated at the corresponding locations with the content "2 trapped personnel", "vehicle stalled", and "water depth approximately 45cm". When rendering these AR tags, the depth buffer of the 3D real-world model is used for occlusion detection. When the tag is behind a building (such as when the trapped personnel are obscured by roadside trees), that part of the tag is not displayed. When the drone changes its perspective, the occlusion disappears, and the tag reappears. The whole process is natural and smooth, without "tag floating". Phenomenon. The rule engine's preset condition "water depth > 40cm and people trapped" triggers a Level 1 emergency response. The system automatically executes a series of operations in the coordinated response process: automatic announcement, playing the voice command "Rescue personnel have set off, please remain calm" through the drone's onboard loudspeaker, repeating 3 times; dispatching the order to the fire department, pushing the precise coordinates (longitude, latitude, altitude) of the trapped person and the optimal rescue route (based on road network analysis) to the nearest firefighter's handheld terminal (Android / iOS APP); activating the traffic signal green wave, coordinating with the city's traffic signal control system to open a green wave lane for rescue vehicles (switching traffic lights to green along the planned route), and simultaneously automatically activating the downstream drainage pumping station (30kW power, 1000m³ / h drainage capacity) through the IoT platform.The drone continuously hovers and monitors (at an altitude of 50m, with an adjustable viewing angle). The AI ​​assesses the changes in water depth and the progress of personnel rescue every 2 seconds. Once the water level drops to a safe threshold (<10cm) and all trapped personnel are rescued (AI detection confidence >0.9 and no new personnel are trapped for 30 seconds), the emergency response is automatically terminated, and the drone returns to base. The entire response time meets the real-time requirements of emergency command. Furthermore, the virtual-real fusion image generated by the rendering engine module is encoded into a video stream in real time by the image distribution module. Specifically, the rendering engine module generates the rendered image at a frame rate of 60fps, which is then compressed using H.264 encoding and pushed to the command center's large screen, web browser, and mobile terminals via the WebRTC protocol. Client terminals only need video decoding capabilities and network connectivity; no dedicated software or high-performance GPUs are required. Actual testing shows that the end-to-end image transmission latency is controlled within 200ms, meeting the real-time requirements of emergency command. When multiple users access the system concurrently, the image distribution module uses an SFU (Selective Forwarding Unit) architecture to achieve single-path streaming and multi-path distribution, significantly reducing server bandwidth pressure. This embodiment demonstrates that the present invention, through a closed-loop architecture of "real-time video projection + AI recognition + coordinated response," elevates digital twins from static displays to a real-time, intervention-enabled intelligent decision-making platform, significantly improving the speed and accuracy of urban emergency response.

[0045] Example 2: Restoration of cultural and tourism heritage sites (large-scale ancient city ruins), such as... Figure 6As shown, this embodiment uses ground robots or drones as mobile platforms for the digital restoration and immersive guided tours of large cultural heritage sites (such as ancient city ruins and grotto temples). Specifically, it takes a large ancient city ruin covering an area of ​​approximately 2 square kilometers, where some buildings only have foundations remaining and some city walls have partially collapsed, as the application object. The site has been constructed using oblique photogrammetry, resulting in a high-precision 3D real-scene model with an accuracy of 3 cm, realistic texture, and approximately 800 million triangular faces. The museum hopes that when tourists watch the drone live broadcast on a tablet, through video media, or on a large screen, they can see the original historical features such as collapsed city walls and damaged palaces superimposed on the real site in an augmented reality way. Furthermore, as the viewpoint of the tourist or drone moves, the virtual restoration part always maintains precise spatial alignment and physical occlusion with the real site. Therefore, this embodiment uses a DJI Mavic 3E or similar drone as the mobile platform, equipped with an RTK / IMU and a 20-megapixel, 30fps high-definition camera. It conducts weekly routine inspections of the site along a planned route. Simultaneously, tourists walk around the site using iPad Pro devices equipped with LiDAR and ARKit. Visual inertial odometry provides six-DOF pose data with an accuracy of approximately 5cm. Both devices serve as data acquisition terminals on the drone / handheld device side, transmitting real-time video streams and pose data, including actual city wall ruins and column bases, to the scenic area's edge server. The server then simultaneously processes this multi-source data. Figure 6 The core process of the server-side processing (in this system) involves projecting real-time video onto the surface of a 3D reality model of the archaeological site, laying a precise spatial projection foundation for subsequent virtual reconstruction overlay. Archaeologists pre-modeled virtual reconstructions of complete city walls and palaces from various historical periods within the 3D reality model, with approximately 200 million triangular faces. When visitors view the site on an iPad, the server renders virtual reconstructions such as virtual city gates (AR) onto the corresponding locations in the reality model. Simultaneously, using a depth buffer in the reality model reflecting the current state of the site, it determines the occlusion relationship between the virtual reconstruction model and the existing site. For example, existing column bases (real) naturally occlude virtual walls after depth testing, ensuring the correct spatial hierarchy between the virtual reconstruction and the real site remains, making the reconstruction effect realistic and believable. This process corresponds to the core presentation logic of the virtual reconstruction overlay display in the image. The real city wall remains serve as the reality base, and the occlusion interaction between the column bases (real) and the virtual city gates (AR) visually demonstrates the spatial physical occlusion rules in virtual-real fusion. The virtual-real fusion image generated by the rendering engine is pushed to the visitor's terminal through the image distribution module. Figure 6The system utilizes a hybrid virtual and real-world display platform (large screen / mobile device). Visitors can access the hybrid virtual and real-world guided tour page through a browser using ordinary smartphones or tablets, without installing any applications. The server employs an adaptive bitrate strategy, dynamically adjusting the video stream bitrate from 720p to 4K based on the visitor's network conditions to ensure a smooth experience under different network conditions. Simultaneously, a ResNet50-based object detection AI model automatically identifies special structures in the site, such as column bases, drainage ditches, and carved patterns, and automatically overlays AR labels such as "Tang Dynasty Palace Foundation" onto the video projection screen, synchronously linking information labels such as age and size. It also supports a visual presentation format with semi-transparent sticker overlays; visitors can click on the labels to view detailed archaeological data (text, images, and videos). For cultural relic preservation workers, the system also overlays historical crack comparisons (automatically labeled if the difference exceeds 2mm by comparing with the previous month's model) and settlement monitoring data (real-time display of settlement values) onto the inspection drone videos to assist in detecting changes in the cultural relics themselves. Overall, this embodiment relies on... Figure 6 The complete process, from drone / handheld device data collection and server processing to large-screen / mobile terminal virtual-real fusion display, enables real-time virtual-real fusion from a mobile perspective in a wide range of cultural heritage scenes. It accurately presents the spatial occlusion relationship between real city wall ruins and virtual city gates (AR), the occlusion effect of real column bases on virtual walls, and the visual interaction of AR tags such as "Tang Dynasty Palace Foundation", era, and size. It provides a brand-new technical means for site restoration and display, cultural relic monitoring, and immersive visitor tours.

[0046] Example 3: Power inspection scenario (transmission lines and towers), such as Figure 7As shown, this embodiment uses a drone as a mobile platform for the automated inspection of power transmission lines and towers. The specific implementation process is as follows: A 50km long power transmission line with 120 towers is selected as the inspection target. The traditional manual inspection cycle for this line is once a month, which is not only inefficient but also poses significant safety risks. To address this, the power company has established a high-precision three-dimensional real-scene model of the entire line using drone oblique photography. The model of towers, conductors, insulator strings, and hardware has an accuracy of 1 cm. Based on this, the present invention is used to achieve automated inspection. When performing inspection tasks, a DJI M300 RTK or similar drone (equipped with a zoom camera and thermal imaging) is used. The drone flies automatically along a preset route, accurately covering key inspection points of each real power transmission tower, including insulator string hanging points, conductor clamps, and lightning protection devices on the tower top. The flight speed is controlled at 5m / s, maintaining a distance of 15-20m from the tower. During the flight, the drone collects on-site data of the real power transmission tower / insulator string and synchronously transmits the real-time video stream and RTK / IMU data back to the ground station server (this system). The ground station server simultaneously performs projection matrix calculation and video projection operations, projecting the drone video onto the corresponding position of the 3D real-world model (tower) in real time through real-time video texture projection. Because the zoom camera's focal length changes in real time, the projection matrix is ​​also dynamically updated accordingly, ensuring accurate projection at both the telephoto end (focal length 129mm) and the wide-angle end (focal length 4.3mm), achieving precise fusion of virtual and real, and seamless alignment of the insulator string area in the model with the drone video. Meanwhile, the deep learning model (YOLOv8x, specifically trained for power defects) deployed at the ground station detects various anomalies in the video stream in real time. It can accurately identify insulator string damage (detection accuracy 97%), equipotential ring tilt (detection accuracy 94%), bird's nest (tower top, detection accuracy 98%), conductor strand breakage (detection accuracy 91%), and abnormal temperature rise in thermal imaging (detection accuracy 95%). After detecting a defect, the image coordinates of the defect location are converted into 3D real-world model spatial coordinates, and AR labels are generated at the corresponding locations in the model, such as [AR] Damage Level 4, Thermal Imaging +65℃ Anomaly, etc. The labels include the defect type, severity level (1-5), and suggested repair plan. The labels are rendered according to preset occlusion handling rules to ensure that the parts of the labels that are obscured by the tower structure are not displayed, keeping the image clean.The system automatically summarizes all identified problems and generates a defect list including insulator damage (phase A) coordinates (X,Y,Z), equipotential ring tilt (phase B), and bird's nest (tower top). After the inspection is completed, a defect report is automatically generated, which includes the three-dimensional coordinates of each defect, a defect photo (screenshot after projection onto the model), and thermal imaging temperature values. For defects with a severity level ≥4, the [Generate Report] [Dispatch Work Order] process is automatically triggered to assign the repair task to the nearest repair team. Repair personnel can view the precise location of the defect (marked in the three-dimensional model) through a handheld terminal and navigate to the site. After the repair is completed, the drone can inspect the same location again and compare the state before and after the repair through video projection to verify the repair effect. Furthermore, the ground station server pushes the fused virtual and real images during the inspection process to remote expert terminals in real time via the image distribution module. Remote experts can view the fused virtual and real images, which are identical to those on the ground station, through a web browser and annotate and mark defect locations on the images. The annotation results are transmitted back to the ground station server in real time via the WebRTC data channel, enabling remote collaborative diagnosis. This function is particularly suitable for scenarios involving complex defects requiring expert consultation, effectively reducing the need for experts to travel to the site and significantly lowering maintenance costs. The entire embodiment clearly demonstrates that this invention can achieve high-precision real-time defect detection and AR annotation in the field of industrial inspection, effectively improving inspection efficiency and accuracy while reducing manual labor intensity and operational risks, fully demonstrating its practical value in the field of industrial inspection.

[0047] In one implementation, the present invention is compared with existing fixed camera solutions from multiple dimensions, as shown in Table 1. The differences are clearly defined in terms of camera platform type, camera pose state, core technical issues, pose calculation method, projection matrix update, coordinate transformation direction, real-time requirements, number of video sources, handling of virtual and real occlusion, rendering engine, model accuracy, lighting effects, and typical application scenarios: Existing technologies use fixed-view surveillance cameras as platforms, with pose being a known constant that remains unchanged after one calibration. The core solution is the inverse calculation of 2D pixel coordinates to 3D geographic coordinates, requiring no pose calculation. The projection matrix remains unchanged after one calibration. It employs an inverse calculation method from image to geography, only supporting near real-time, second-level latency, relying on multi-channel fixed video stitching, and handling of virtual and real occlusion often depends on simple depth maps or is not processed at all. The present invention, however, uses a 3D GIS engine such as the self-developed Qianzhou. The model requires significant polygon reduction to adapt to performance limitations, supports only pre-baked lighting or simple dynamic lighting, and its application scenarios are limited to fixed scenarios such as urban security, traffic monitoring, and smart parks. In contrast, this invention uses mobile platforms such as drones and ground robots as carriers, where the camera pose is an unknown variable that changes over time and requires real-time calculation. The core implementation achieves real-time projection of dynamic video streams onto the surface of a 3D reality model. Pose calculation is performed in real-time at a frequency of 30-100Hz using a fusion of RTK, IMU, and visual inertial odometry. The projection matrix is ​​dynamically updated every frame at a frequency of no less than 30Hz to adapt to platform motion and zoom operations. It adopts a geo-to-image orthographic projection method, achieving real-time interaction with an end-to-end latency of less than 200ms. Projection can be completed based on a single source or a small amount of dynamic video. Meta-level physical occlusion is implemented based on the depth buffer of the reality model. It utilizes UE5 and Unity. HDRP and other film-grade real-time rendering engines, relying on virtual geometry technology, support original high-precision models with no less than 100 million triangle faces, achieving full dynamic global illumination and real-time light bounce. It can be applied to dynamic scenarios such as emergency inspection, cultural tourism guidance, site restoration, and power inspection. The essential difference between the two clearly demonstrates the core value of this invention in filling the gap in the fusion technology of real-time video projection and digital twin on mobile platforms. It provides a key technical path for the evolution of digital twins from "static recording" to "real-time mirroring" and also represents the future development direction of the integration of digital twins and augmented reality.Based on the aforementioned technological advantages, this invention possesses broad industrial applicability and can be applied to multiple fields: In urban emergency command scenarios, it can serve scenarios such as flood relief, fire fighting, and traffic accident handling, providing real-time on-site situational awareness and intelligent decision support. Through the image distribution module, the command center, on-site command vehicles, and mobile terminals can simultaneously view the same virtual-real fusion image, achieving multi-level collaborative command; In smart cultural tourism and cultural heritage protection scenarios, it can be used for digital restoration of sites, immersive guided tours, and monitoring of changes in cultural relics. Tourists can directly access the virtual-real fusion guided tour through the browser of their personal mobile devices (phones, tablets). It eliminates the need for dedicated application installation, significantly lowering the barriers to deployment in scenic areas and for tourists to use it. In power, energy, and transportation inspection scenarios, it can achieve automated inspection and defect detection of transmission lines, wind turbines, bridges, and tunnels. Inspection images can be pushed to remote expert terminals in real time through the image distribution module, supporting remote collaborative diagnosis, reducing the need for experts to travel to the site, and significantly reducing operation and maintenance costs. In addition, it can also be applied to environmental monitoring and agriculture (such as water pollution monitoring, forest fire prevention, and crop growth assessment), public safety and security (such as crowd control for large-scale events, patrol of key areas, and on-site command of emergencies). Leveraging existing drone and robot hardware platforms and 5G / 4G communication networks, the method and system provided by this invention are easy to deploy and promote. The system adopts an edge-cloud distributed architecture, with each module communicating through standardized APIs. It can be flexibly configured according to actual network conditions and computing power requirements: tasks with high real-time requirements (such as projection matrix calculation and texture mapping) are deployed on edge nodes, while tasks such as model training, global data fusion, and image distribution are deployed in the cloud, and data acquisition is deployed on the edge. Client terminals receive virtual and real fusion images through the WebRTC protocol, which can be accessed by any browser that supports WebRTC. No special software or high-performance GPUs are required, which significantly reduces terminal deployment costs and cross-platform adaptation difficulties. At the same time, the system supports concurrent access by multiple users, has good scalability, and is suitable for various application scenarios from single-machine deployment to large-scale cluster deployment.

[0048] Table 1

[0049] Compared with existing technologies, this invention has many significant advantages. Its core lies in breaking through the bottlenecks of traditional digital twin technology, achieving real-time virtual-real fusion in dynamic scenes on mobile platforms. Specifically, this invention overcomes the timeliness bottleneck of static digital twins by dynamically projecting real-time video streams from mobile platforms onto a 3D real-world model, ensuring millisecond-level synchronization between the model's surface texture and the real environment. For example, in urban emergency scenarios, after a drone flies over a flooded area, the command center's digital twin screen can display real-time water accumulation and the location of stranded vehicles, with end-to-end latency controlled within 200ms. In cultural and tourism heritage scenarios, tourists can see virtual reconstructions of historical buildings superimposed on the current site through drone live streams or handheld devices. This "real-time mirroring" capability transforms digital twins from static displays that are "good-looking but useless" into dynamic decision-making platforms that can reflect changes in the physical world in real time, representing the core direction of digital twin technology development. Simultaneously, this invention solves the physical correctness problem of virtual-real fusion in dynamic scenes by utilizing the depth buffer of the high-precision real-world model for piecewise occlusion judgment, ensuring that the virtual-real fusion is accurate and timely. The simulated content can be naturally occluded by real buildings, trees, terrain, etc., achieving a leading level of visual realism in the integration of virtual and real elements. In the restoration of cultural and tourism heritage sites, the virtual restored city walls can be correctly occluded by existing foundations and vegetation. In urban governance, AR tags marking dangerous areas can be naturally hidden by buildings to avoid visual interference. Quantitative evaluation shows that the occlusion accuracy of this invention achieves physically correct occlusion relationships, which is significantly better than the traditional screen coordinate overlay scheme (on the standard test set), while the traditional screen coordinate overlay scheme is only 60%-70%. In addition, this invention constructs a complete business closed loop of "perception-recognition-projection-decision-handling", organically combining AI real-time recognition with dynamic video projection and rule engine linkage. In emergency scenarios, the entire process from the mobile platform discovering trapped personnel to automatic announcement, task assignment, and traffic signal linkage can be completed automatically within seconds, with response efficiency 5-10 times higher than traditional manual dispatch. This capability upgrades digital twins from "visual display" to an "operable and interventionable" intelligent decision-making platform, providing a brand-new technical architecture for smart cities, emergency command, and other scenarios.This invention also boasts cinematic-grade rendering quality, employing a real-time rendering engine with virtual geometry technology (such as Nanite). It can directly handle the original high-precision models (hundreds of millions of triangular faces) generated by oblique photogrammetry, eliminating the need for the model reduction processing required by traditional GIS engines. This preserves the geometric details of the terrain features completely. Simultaneously, fully dynamic global illumination technology (such as Lumen) allows the lighting and shadows in the fused virtual-real image to change in real-time with the actual ambient lighting, eliminating the need for pre-baking and dynamically responding to environmental factors such as sunlight angle and weather changes. Compared to existing video fusion solutions based on 3D GIS engines, the rendering quality reaches cinematic-grade standards, significantly enhancing the visual quality of the digital twin system. The realism and immersive user experience offer significant advantages in scenarios with high visual requirements, such as the restoration of cultural and tourism heritage sites and the display of urban images. Furthermore, this invention is not limited to specific commercial engines; any real-time rendering engine that supports real-time video texture dynamic projection, virtual geometry technology, and fully dynamic global illumination can implement the technical solution of this invention. Compared with existing technologies, this invention also differs from fixed-camera solutions, filling a gap in mobile platform technology. Existing video fusion methods based on fixed surveillance cameras rely on known constant camera poses, primarily addressing the static mapping problem from two-dimensional pixel coordinates to three-dimensional geographic coordinates, which cannot be implemented on mobile platforms (such as...). This invention addresses the problem of real-time, frame-by-frame projection alignment of dynamic video streams with 3D models in high-speed, continuously changing dynamic scenarios involving drones. However, for mobile platforms such as drones and ground robots, where camera pose is an unknown variable changing over time, this invention solves the problem of "real-time, frame-by-frame projection of dynamic video streams with 3D reality models." These two approaches differ fundamentally in input data types, core technical challenges, coordinate transformation directions, real-time requirements, and scene complexity, filling a technological gap in the field of real-time video projection and digital twin fusion for mobile platforms. Furthermore, this invention differs from existing offline or static texture overlay techniques and combines with recent drone oblique photography... Compared with real-time rendering technology, this invention further achieves breakthroughs in "real-time dynamics" and "physically correct occlusion". Some existing solutions mainly address the problems of offline 3D model reconstruction and pre-rendering. The video data is still static or quasi-static texture, which cannot achieve frame-by-frame dynamic projection of the video stream and real-time synchronous mapping of the model surface during the flight of the mobile platform. Moreover, most of them use simple screen coordinate AR overlay or pre-baked lighting, lacking piece-by-piece physical correct occlusion based on the depth buffer of the real scene model. This invention solves the above-mentioned technical gaps through real-time pose-driven dynamic update of the projection matrix, frame-by-frame mapping of video as dynamic texture, and a two-pass rendering mechanism with depth buffer.This invention represents a system-level integrated innovation based on mature technology modules. The various technology modules involved, such as oblique photogrammetry modeling, RTK / IMU pose perception, and visual inertial odometry, are all well-known and mature technologies in the field. Its substantial contribution lies not in improving any single module, but in organically integrating these known modules into a complete system in a specific way. Through the collaborative work between these modules, an overall technical effect that a single module could not achieve is produced. Specifically, the mobile platform can maintain pixel-level alignment between the video and the model and physically correct virtual-real occlusion even under dynamic conditions such as high-speed movement and zoom shooting. This system-level integrated innovation... This invention represents the core technological path of digital twin evolution from static display to real-time interaction. Furthermore, it reduces system deployment and maintenance costs, eliminating the need for physical markers (such as QR codes or reflective dots) or complex motion capture equipment in the real environment. High-precision positioning and projection can be achieved solely through the RTK / IMU sensors and visual inertial odometry mounted on the mobile platform, significantly reducing system deployment time and long-term maintenance costs, making it particularly suitable for large-scale, complex terrain outdoor scenarios. Simultaneously, this invention uses mobile platforms such as drones to replace manual entry into high-risk areas for real-time data collection and virtual-real fusion analysis, reducing... This invention reduces the risk of personnel being directly exposed to hazardous environments, while simultaneously shortening emergency response time through automated and coordinated handling, effectively improving the safety of inspections and emergency operations. Technically, this invention differs significantly from WebGL-based 3D video fusion methods. The latter requires real-time rendering on a local terminal, with image quality limited by the terminal's GPU performance, and necessitates substantial polygon reduction of the 3D model. In contrast, this invention employs a "high-performance cloud rendering + real-time video projection + low-latency distribution" approach, overcoming the image quality bottlenecks and terminal performance limitations of local rendering. Furthermore, it not only covers cloud rendering solutions but also any method that achieves "dynamic..." The technical solution of "video projection + deep occlusion" ensures the unity of the width and rationality of the protection range. This invention represents an important direction for the evolution of digital twin technology towards real-time dynamics. By organically integrating technologies such as dynamic video projection, real-time pose calculation, and physically correct occlusion with high-precision real-world models, it solves the core pain point that traditional static digital twins cannot reflect changes in the physical world in real time. It conforms to the development trend of digital twin technology evolving from "static recording" to "real-time mirroring" and can be widely used in dynamic scenarios such as urban emergency response, cultural tourism restoration, and industrial inspection, providing key technical support for the real-time and intelligent development of digital twin technology.In terms of quantitative performance indicators, the embodiments of this invention can achieve the following technical effects: the projection matrix update frequency is synchronized with the video frame rate, updating frame by frame; the end-to-end system latency is controlled at the millisecond level, meeting the real-time interaction requirements (from image acquisition on the mobile platform to rendering and displaying the screen); the pose calculation accuracy is better than centimeter level in RTK mode and better than decimeter level in VIO mode; pixel-level virtual-real registration error is achieved; the accuracy of physically correct occlusion judgment is achieved; it supports 3D models with hundreds of millions of triangular faces (real-time rendering through virtual geometry technology); and when deployed in the cloud, it can support simultaneous access from multiple mobile platforms.

[0050] In summary, this embodiment first acquires the real-time video stream of the target area collected by the mobile platform and the real-time pose data of the mobile platform. Based on the real-time pose data, the projection matrix of the mobile platform is dynamically calculated. The real-time pose data can accurately capture the position and attitude changes of the mobile platform, and the dynamically calculated projection matrix can establish a stable correspondence between the two-dimensional video image and the three-dimensional space, laying a solid foundation for the accurate fusion of the subsequent video and the three-dimensional model, avoiding fusion deviations during the movement of the mobile platform, and ensuring the real-time performance and accuracy of the entire technical process. Next, a three-dimensional real-scene model of the target area is constructed. The video images in the real-time video stream are projected frame by frame onto the corresponding surface area of ​​the three-dimensional real-scene model through the projection matrix, resulting in a dynamic three-dimensional real-scene scene with real-time video textures. This process captures the real-time visual changes of the real scene. By deeply integrating with 3D models, this invention breaks through the limitations of traditional static 3D models, allowing 3D scenes to synchronously reflect the dynamic changes of the real environment, making them more realistic and intuitive. This provides a foundational carrier for subsequent virtual-real scene fusion that fits the real scene. Then, based on the dynamic 3D real-world scene, depth pre-rendering is performed to obtain the depth buffer information corresponding to the real scene in the target area. This depth buffer information is used to perform depth testing and occlusion judgment on the virtual augmented information, resulting in an augmented reality fusion image with correct virtual-real occlusion relationships. By simulating the depth layers of the real scene through the depth buffer information, the occlusion logic between virtual information and the real scene can be accurately handled, avoiding the common problem of virtual-real occlusion confusion in existing technologies. This allows virtual augmented information to be presented reasonably according to the spatial layers of the real scene, significantly improving the immersion and realism of the augmented reality experience. Through the organic connection of the above steps, this invention achieves accurate and natural fusion of the real scene and virtual augmented information, effectively solving the pain points of existing augmented reality technologies such as large fusion deviations, unreasonable occlusion relationships, and insufficient scene dynamism. It significantly improves the quality of augmented reality fusion images and further expands the application space of augmented reality technology in various practical scenarios.

[0051] like Figure 8As shown in the illustration, this embodiment also provides a mobile platform digital twin and AR fusion system based on dynamic video projection. This system includes: a data acquisition and projection matrix calculation module 10, a dynamic 3D real-scene scene acquisition module 20, and an augmented reality fusion image acquisition module 30. Specifically, the data acquisition and projection matrix calculation module 10 is used to acquire real-time video streams collected by the mobile platform and real-time pose data of the mobile platform, and dynamically calculate the projection matrix of the mobile platform based on the real-time pose data. The dynamic 3D real-scene scene acquisition module 20, deployed in the cloud or edge server, is used to construct a 3D real-scene model of the target area, and project video images from the real-time video stream frame by frame onto the corresponding surface area of ​​the 3D real-scene model through the projection matrix to obtain a dynamic 3D real-scene scene with real-time video textures. The augmented reality fusion image acquisition module 30 is used to perform depth pre-rendering based on the dynamic 3D real-scene scene, acquire depth buffer information corresponding to the real scene, and use the depth buffer information to perform depth testing and occlusion judgment on the virtual augmented information to obtain an augmented reality fusion image with correct virtual-real occlusion relationship.

[0052] Among them, such as Figure 9As shown, the data acquisition module, deployed on a mobile platform (such as a drone or ground robot), includes an image acquisition unit, an RTK positioning module, an inertial measurement unit (IMU), and an optional airborne edge computing unit. The projection matrix calculation module, deployed on an edge server or in the cloud, receives real-time pose data from the data acquisition module, performs coordinate system transformations (transformation between the world coordinate system, camera coordinate system, and image coordinate system), dynamically calculates the projection matrix P=K[R|t] of the mobile platform, and updates it frame-by-frame at a frequency no less than the video frame rate. When the mobile platform zooms, this module acquires the camera's intrinsic parameters after zooming and adjusts the projection matrix accordingly. The dynamic 3D real-scene acquisition module includes a 3D model storage module, deployed in the cloud or on an edge server, used to store a pre-built high-precision 3D real-scene model of the target area. This model is generated through oblique photogrammetry or LiDAR technology, containing a complete geometric mesh (triangular facets or point clouds) and realistic textures, with a geometric accuracy better than 5 cm, and establishes a multi-level hierarchy of detail (LOD) structure, supporting real-time streaming loading of large-scale scenes. The dynamic 3D real-scene acquisition module also includes a texture mapping module, deployed on an edge server or in the cloud. This module uses each frame of the real-time video stream as a dynamic texture, projecting the video image frame-by-frame onto the corresponding surface area of ​​the 3D real-scene model using the projection matrix output by the projection matrix solving module. This module pre-calculates the UV coordinates of the model vertices using a vertex shader and performs bilinear interpolation sampling in the fragment shader. Simultaneously, it performs frustum culling on model regions far from the camera's view frustum to reduce rendering load. The augmented reality fusion image acquisition module includes a virtual-real occlusion processing module, deployed in the rendering pipeline, used to achieve physically correct occlusion when rendering virtual augmented information. This module first performs a first rendering pass: only writing the 3D real-scene model (including dynamic textures) to the depth buffer without writing colors. Then, it performs a second rendering pass: rendering virtual augmented information (such as AR tags, restored models, and warning signs). A depth test is performed on each virtual fragment; if the virtual depth is greater than the depth value of the corresponding fragment in the real-scene model, the virtual fragment is determined to be occluded and discarded; otherwise, it is displayed normally. Through this mechanism, virtual content can be naturally occluded by real buildings, trees, terrain, etc. The augmented reality fusion image acquisition module also includes a rendering engine module, deployed in the cloud or on a high-performance edge server, for final real-time rendering output. This rendering engine features virtual geometry technology and fully dynamic global illumination capabilities, supports real-time rendering frame rates of at least 100 million triangles, and can directly handle the original high-precision models generated by oblique photogrammetry, achieving cinematic-quality image quality. The rendering engine can employ at least one of Unreal Engine 5, Unity High Definition Render Pipeline, or a self-developed real-time rendering engine.

[0053] The system also includes: an AI recognition module: deployed on an edge server or in the cloud, used for target detection and recognition of real-time video streams (such as trapped personnel, flooded areas, equipment defects, cracks in cultural relics, etc.), converting the image coordinates of the recognition results into 3D real-scene model spatial coordinates through the inverse transformation of the projection matrix, and generating virtual tags or warning signs at the corresponding locations. A coordinated response module: deployed in the cloud, with a built-in rule engine, automatically triggers coordinated response commands based on preset conditions (such as "flood depth > 40cm and trapped personnel"), including but not limited to automatic announcements from mobile platforms, rescue task assignment, traffic signal coordination, and IoT device control. A video distribution module: deployed in the cloud, used to encode the virtual-real fusion video generated by the rendering engine into a video stream in real time, push it to one or more client terminals (command center screen, web browser, mobile terminal application) via the WebRTC protocol, and receive user interaction commands returned by the clients, achieving low-latency cross-platform access. Inter-module communication and deployment: The above modules communicate with each other through standardized APIs (RESTful, gRPC, or WebSocket), supporting flexible deployment across the edge and cloud. The data acquisition module and the projection matrix calculation module transmit high-frequency pose data via the MQTT / Protobuf protocol, and transmit real-time video streams via the WebRTC / SRT protocol. The projection matrix calculation module, texture mapping module, virtual / real occlusion processing module, and rendering engine module interact via an internal high-speed bus or shared memory. The rendering engine module and the image distribution module are connected via a pixel streaming interface. The cloud module and the client terminal interact via protocols such as HTTP / 2, WebRTC, and HLS. The system can deploy some computing tasks to edge nodes to achieve low-latency response, and deploy tasks such as model training and global data fusion to the cloud for centralized management, based on actual network conditions and computing power requirements.

[0054] like Figure 10As shown in the figure, the system adopts a three - level collaborative architecture of edge - cloud - end to achieve real - time data acquisition, processing, and remote command and control of UAVs / mobile platforms: First, on the end side (mobile platform / UAV robot), the camera, RTK, and IMU form a data acquisition module to obtain video images, high - precision positioning, and inertial data. Preliminary processing can be completed through on - board edge computing, and then timestamp synchronization is completed through the data encapsulation module. Subsequently, based on the 5G / WiFi network, the video stream is transmitted through WebRTC, and the pose data is transmitted through MQTT, and the encapsulated synchronized data is sent to the edge side (field server / UAV airport); after receiving the data, the edge side performs RTSP / WebRTC access, decoding, and texture mapping processing on the video stream, performs projection matrix calculation and virtual - real occlusion processing on the pose data, and at the same time completes AI recognition and inference through a lightweight YOLO model and realizes local alarm and linkage handling. Then, the key - frame recognition results and projection matrix data are uploaded to the cloud (central cloud platform / command center) through the gRPC / HTTP2 protocol via the Internet / VPN network; the cloud provides capabilities such as three - dimensional model storage (supporting 3D Tiles / OSGB format), AI model training (based on the TensorFlow / PyTorch framework), global data fusion and long - term storage, command large - screen rendering and streaming (supporting WebRTC / HLS protocol), etc. After being processed by the screen distribution module, the finally integrated virtual - real fusion screen is pushed to the command center large - screen or Web and mobile terminals, realizing the closed - loop control and visual presentation of the entire process.

[0055] In the mobile - platform digital - twin and AR fusion system based on dynamic video projection of this embodiment, the working principles of each module are the same as those of each step in the above - mentioned method embodiment, and will not be elaborated here.

[0056] Based on the above - mentioned embodiment, the present invention also provides a terminal, and the principle block diagram of the terminal can be as Figure 11 shown. The terminal may include one or more processors 100 ( Figure 11 only one is shown in the figure), a memory 101, and a computer program 102 stored in the memory 101 and executable on one or more processors 100, for example, a mobile - platform digital - twin and AR fusion program based on dynamic video projection. When one or more processors 100 execute the computer program 102, each step in the method embodiment of the mobile - platform digital - twin and AR fusion based on dynamic video projection can be implemented. Or, when one or more processors 100 execute the computer program 102, the functions of each module / unit in the method embodiment of the mobile - platform digital - twin and AR fusion based on dynamic video projection can be implemented, and no limitation is made here.

[0057] In one embodiment, the processor 100 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0058] In one embodiment, memory 101 may be an internal storage unit of an electronic device, such as a hard drive or RAM. Memory 101 may also be an external storage device of the electronic device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc. Furthermore, memory 101 may include both internal and external storage units. Memory 101 is used to store computer programs and other programs and data required by the terminal. Memory 101 can also be used to temporarily store data that has been output or will be output.

[0059] Those skilled in the art will understand that Figure 11 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0060] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, operational databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual operating data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for fusing digital twins and AR on a mobile platform based on dynamic video projection, characterized in that, The method includes: The system acquires real-time video streams of the target area collected by the mobile platform and real-time pose data of the mobile platform, and dynamically calculates the projection matrix of the mobile platform based on the real-time pose data. A three-dimensional real-scene model of the target area is constructed, and video images from the real-time video stream are projected frame by frame onto the corresponding surface area of ​​the three-dimensional real-scene model through a projection matrix to obtain a dynamic three-dimensional real-scene scene with real-time video texture. Based on dynamic 3D real scene, depth pre-rendering is performed to obtain the depth buffer information corresponding to the real scene of the target area. The depth buffer information is used to perform depth testing and occlusion judgment on virtual augmented information to obtain augmented reality fusion image with correct virtual and real occlusion relationship.

2. The method for fusing digital twins and AR on a mobile platform based on dynamic video projection according to claim 1, characterized in that, The dynamic calculation of the projection matrix of the mobile platform based on real-time pose data includes: The real-time pose data is converted into a camera extrinsic matrix in the world coordinate system, and then combined with the camera intrinsic matrix to generate a projection matrix.

3. The method for fusing digital twins and AR on a mobile platform based on dynamic video projection according to claim 1, characterized in that, The process of projecting video images from the real-time video stream frame by frame onto the corresponding surface area of ​​the 3D real-scene model using a projection matrix to obtain a dynamic 3D real-scene scene with real-time video textures includes: Obtain the projection matrix of the current rendering frame and the world coordinates of the fragment to be rendered on the surface of the 3D real scene model, and perform coordinate transformation on the fragment to be rendered based on the projection matrix and the world coordinates to generate the corresponding homogeneous clipping coordinates. A perspective division is performed on the homogeneous clipping coordinates to obtain normalized device coordinates, and the texture sampling coordinates of the fragment to be rendered in the real-time video stream are generated based on the normalized device coordinates. Based on the mapping relationship between the texture sampling coordinates and the position of the real-time video stream, the fragment to be rendered is subjected to texture rendering processing to generate a dynamic 3D real-world scene with real-time video texture.

4. The method for fusing digital twins and AR on a mobile platform based on dynamic video projection according to claim 3, characterized in that, The process of performing texture rendering on the fragment to be rendered based on the position mapping relationship between the texture sampling coordinates and the real-time video stream includes: If the texture sampling coordinates are within the effective texture mapping range of the real-time video stream, then the target color value is sampled from the real-time video stream based on bilinear interpolation or anisotropic filtering algorithm, and the target color value is used to replace the original preset texture of the fragment to be rendered for shading rendering.

5. The method for fusing digital twins and AR on a mobile platform based on dynamic video projection according to claim 3, characterized in that, The process of performing texture rendering on the fragment to be rendered based on the position mapping relationship between the texture sampling coordinates and the real-time video stream includes: If the texture sampling coordinates exceed the effective texture mapping range of the real-time video stream, the original preset texture of the fragment to be rendered is maintained for rendering.

6. The method for fusing digital twins and AR on a mobile platform based on dynamic video projection according to claim 1, characterized in that, The process of performing depth pre-rendering based on a dynamic 3D real-world scene to obtain depth buffer information corresponding to the real scene of the target area includes: Acquire the geometric model data of the dynamic 3D real scene and the corresponding camera frustum parameters; Configure the rendering pipeline according to the camera's frustum parameters, and configure a depth pre-write mode with depth writing enabled and color writing disabled. In depth pre-write mode, geometric model data is input into the rendering pipeline for rasterization processing, and the depth value of each fragment in the real scene in camera space is calculated. The calculated depth value is written into the depth buffer of the graphics processor to generate depth buffer information that represents the geometric distance distribution of the real scene in the target area.

7. The method for fusing digital twins and AR on a mobile platform based on dynamic video projection according to claim 6, characterized in that, The process of using depth buffer information to perform depth testing and occlusion judgment on virtual augmented information to obtain an augmented reality fusion image with correct virtual-real occlusion relationship includes: Obtain the vertex data of the virtual augmentation information to be rendered, and calculate the virtual depth value of each fragment of the virtual augmentation information in camera space based on the projection matrix; Read the real scene depth value stored at the corresponding pixel position in the depth buffer information; Perform a piecewise meta-depth comparison between the virtual depth value and the real scene depth value; If the virtual depth value is greater than the real scene depth value, it is determined that the fragment is occluded by the real scene and the rendering instruction of the fragment is discarded. If the virtual depth value is less than or equal to the real scene depth value, it is determined that the fragment is not occluded, and a color blending or direct overlay operation is performed. The color value of the fragment is written to the frame buffer, and the augmented reality blended image is output.

8. A mobile platform digital twin and AR fusion system based on dynamic video projection, characterized in that, The system includes: The data acquisition and projection matrix calculation module is used to acquire the real-time video stream of the target area collected by the mobile platform and the real-time pose data of the mobile platform, and dynamically calculate the projection matrix of the mobile platform based on the real-time pose data. The dynamic 3D real scene acquisition module is used to construct a 3D real scene model of the target area. The video images in the real-time video stream are projected frame by frame onto the corresponding surface area of ​​the 3D real scene model through a projection matrix to obtain a dynamic 3D real scene with real-time video texture. The augmented reality fusion image acquisition module is used to perform depth pre-rendering based on dynamic 3D real scene, obtain the depth buffer information corresponding to the real scene of the target area, and use the depth buffer information to perform depth testing and occlusion judgment on virtual augmented information to obtain augmented reality fusion image with correct virtual-real occlusion relationship.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and a mobile platform digital twin and AR fusion program based on dynamic video projection stored in the memory and executable on the processor. When the processor executes the mobile platform digital twin and AR fusion program based on dynamic video projection, it implements the steps of the mobile platform digital twin and AR fusion method based on dynamic video projection as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a mobile platform digital twin and AR fusion program based on dynamic video projection. When the mobile platform digital twin and AR fusion program based on dynamic video projection is executed by a processor, it implements the steps of the mobile platform digital twin and AR fusion method based on dynamic video projection as described in any one of claims 1-7.

Images