A virtual-real fusion method and system based on Gaussian splash real-scene reconstruction
By generating sparse point clouds through Gaussian splash real-scene reconstruction and performing lightweight processing, and creating proxy geometry by combining physical camera parameters, texture mapping of real-time video streams and fusion with virtual scenes are achieved. This solves the problems of perspective jumps and spatial cognition interruptions in existing technologies and enables seamless switching between macroscopic real-scene models and microscopic fine models.
Patent Information
- Application Number
- CN202511685960.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-18
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2045-11-18
AI Technical Summary
In existing technologies, when users switch between different visualization systems or views, it causes a change in perspective and an interruption in spatial cognition, making it impossible to achieve seamless switching between macroscopic real-world models and local device details.
By using the Gaussian splash real-scene reconstruction method, sparse point clouds are generated and lightweighted. By combining physical camera parameters to create proxy geometry, texture mapping of real-time video streams and fusion with virtual scenes are realized. The spatial mapping relationship between real-scene models and detailed 3D models is established, and the model visibility is controlled in response to user interaction.
It enables seamless switching between macroscopic realistic models and microscopic detailed models in the same 3D scene on the same web platform, avoiding operation interruption and loss of spatial context, ensuring the continuity of observation perspective and the coherence of the exploration process, and solving the problem of perspective jump and spatial coherence interruption caused by system switching in traditional methods.
Smart Images

Figure CN121482339B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer graphics technology, and in particular to a virtual-real fusion method and system based on Gaussian splash real-scene reconstruction. Background Technology
[0002] Real-world 3D reconstruction typically generates triangular mesh models based on oblique photogrammetry or laser scanning technology to construct digital representations of large-scale scenes. Simultaneously, detailed 3D models are created using professional modeling software based on design drawings to showcase the detailed structure and internal composition of local objects. In the visualization stage, real-time video surveillance streams are usually displayed synchronously with the 3D scene in the form of independent windows or 2D panels, together forming the multi-source data foundation of the digital twin system, serving the monitoring and analysis needs of real-world scenes in fields such as urban planning and industrial operation and maintenance.
[0003] In existing technologies, when users need to focus from a macroscopic real-world model to local device details, they often need to switch between different visualization systems or independent views. This abrupt transition not only disrupts the continuous operation process but also forces users to re-establish their spatial location relationship during the switching process, resulting in the complete loss of the original observation perspective and exploration context. Summary of the Invention
[0004] To overcome the above shortcomings, this invention provides a virtual-real fusion method and system based on Gaussian splash real-scene reconstruction, aiming to improve the problems of perspective jump and spatial cognition interruption caused by system switching in the prior art.
[0005] In a first aspect, the present invention provides the following technical solution: a virtual-real fusion method based on Gaussian splash real-scene reconstruction, comprising the following steps:
[0006] S1. Perform motion reconstruction processing on the acquired target scene image sequence to generate a sparse point cloud of the target scene;
[0007] S2. Train the sparse point cloud using Gaussian splashing technology to generate a real-world 3D Gaussian model, and then perform lightweight processing on the real-world 3D Gaussian model to output a lightweight real-world model.
[0008] S3. Obtain the intrinsic and extrinsic parameter matrices of the physical camera and its real-time video stream; based on the extrinsic parameter matrix, create a proxy geometry in the virtual 3D scene that corresponds to the field of view of the physical camera;
[0009] S4. Calculate the projection matrix of the physical camera based on the intrinsic and extrinsic parameter matrices; during the rendering process, use the video frames of the real-time video stream as textures, and use the projection matrix to map the textures onto the proxy geometry for rendering;
[0010] S5. Load the lightweight real-scene model, the artificially refined 3D model, and the proxy geometry with video textures in an integrated manner on the Web platform; establish the spatial mapping relationship between the lightweight real-scene model and the artificially refined 3D model;
[0011] S6. Respond to user interaction with specific areas in the lightweight real-world model, hide or make the lightweight real-world model of the specific area transparent according to the spatial mapping relationship, and simultaneously display the artificially refined 3D model mapped to the specific area.
[0012] By adopting the above technical solution, seamless switching and integrated display of macroscopic real-world models and microscopic detailed models in the same 3D scene on the web terminal are realized. Users do not need to jump between different systems or views when focusing from the global scene to local details, which effectively avoids operation interruption and spatial context loss, and ensures the continuity of observation perspective and the coherence of exploration process.
[0013] Preferably, the motion recovery structure treatment includes:
[0014] Adaptive keyframe extraction is performed on the target scene image sequence, and a set of keyframes is selected from the image sequence based on the image similarity and information entropy change between adjacent frames;
[0015] Image feature points are extracted and matched from the set of keyframes to generate feature matching pairs;
[0016] Based on the feature matching pairs, the camera's intrinsic and extrinsic parameter matrices for each keyframe are recovered through motion recovery structure processing, and a sparse point cloud of the target scene is generated simultaneously.
[0017] Preferably, the weight reduction process includes:
[0018] The Gaussian point cloud in the real-world 3D Gaussian model is subjected to data compression processing, and the spatial position, color attribute and transparency attribute of the Gaussian points are quantized and encoded.
[0019] A hierarchical detail structure is constructed for the compressed Gaussian point cloud. By establishing a multi-level Gaussian point cloud representation from low density to high density, a lightweight real-world model suitable for multi-scale rendering on the web is generated.
[0020] Preferably, creating proxy geometry includes:
[0021] The view frustum of a physical camera in virtual 3D space is determined based on its extrinsic parameter matrix.
[0022] Calculate the geometric intersection region between the view frustum and the virtual 3D scene;
[0023] Within the intersecting region, a corresponding proxy geometry is created based on the geometric features of the scene. The proxy geometry includes one or more planes corresponding to the main plane in the scene.
[0024] Preferably, when the proxy geometry includes multiple planes, it further includes:
[0025] The video frames of the real-time video stream are divided into multiple image blocks according to the geometric structure of the virtual 3D scene;
[0026] Create a corresponding independent proxy plane for each image patch, and adjust the normal and spatial position of each proxy plane;
[0027] Each image block is treated as a texture and mapped onto the corresponding proxy plane for rendering.
[0028] Preferably, before using video frames from the real-time video stream as textures, a video frame preprocessing step is also included:
[0029] Obtain the distortion coefficients of the physical camera calibrated using a calibration board;
[0030] Geometric distortion correction is performed on each video frame of the real-time video stream using distortion coefficients.
[0031] The corrected video frames are output as texture data to be mapped.
[0032] Preferably, the step of mapping the texture onto the proxy geometry for rendering includes:
[0033] In the shader, the projection matrix and view matrix of the physical camera are used to transform the world coordinates of each vertex on the proxy geometry to the clip space of the physical camera.
[0034] The clipping space coordinates are normalized to calculate the corresponding video texture UV coordinates.
[0035] Use UV coordinates to sample the texture of the video frame;
[0036] Based on the sampling results, the proxy geometry is rendered and output using the virtual camera's view matrix. When the virtual camera's view changes, the projection matrix of the physical camera is used to calculate the UV coordinates.
[0037] Preferably, establishing spatial mapping relationships includes:
[0038] In the lightweight real-world model, determine the spatial region corresponding to the finely crafted 3D model;
[0039] Establish a three-dimensional bounding box for the spatial region and obtain the coordinate range of the bounding box in the world coordinate system;
[0040] Establish a corresponding 3D spatial bounding box for a finely crafted 3D model and obtain its coordinate range in the world coordinate system;
[0041] Establish the association between the coordinate ranges of the two bounding boxes and store this association as a spatial mapping relationship.
[0042] Preferably, the method further includes sectioning analysis, which includes:
[0043] Respond to user commands to cut a finely detailed 3D model;
[0044] Real-time calculation of cross-sections of finely crafted 3D models using shader programming;
[0045] Render and display the cross section.
[0046] Secondly, the present invention provides the following technical solution: a virtual-real fusion system based on Gaussian splash real-scene reconstruction, the system comprising the following modules:
[0047] The motion reconstruction processing module is used to perform motion reconstruction processing on the acquired target scene image sequence to generate a sparse point cloud of the target scene.
[0048] The Gaussian splashing processing module is used to train sparse point clouds using Gaussian splashing technology, generate a real-world 3D Gaussian model, and perform lightweight processing on the real-world 3D Gaussian model to output a lightweight real-world model.
[0049] The proxy geometry creation module is used to obtain the intrinsic parameter matrix, extrinsic parameter matrix and real-time video stream of the physical camera, and create a proxy geometry in the virtual 3D scene corresponding to the field of view of the physical camera based on the extrinsic parameter matrix;
[0050] The video texture mapping module is used to calculate the projection matrix of the physical camera based on the intrinsic and extrinsic parameters. During the rendering process, the video frames of the real-time video stream are used as textures, and the projection matrix is used to map the textures onto the proxy geometry for rendering.
[0051] The scene loading and mapping module is used to load lightweight real-scene models, artificially refined 3D models, and proxy geometry with video textures in an integrated manner on the Web, and to establish the spatial mapping relationship between lightweight real-scene models and artificially refined 3D models.
[0052] The interactive response module is used to respond to user interaction with specific areas in the lightweight real-world model. It hides or makes the lightweight real-world model of the specific area transparent according to the spatial mapping relationship, and simultaneously displays the artificially refined 3D model mapped to the specific area.
[0053] The present invention has the following beneficial effects:
[0054] 1. In this invention, by establishing a spatial mapping relationship between the real-scene reconstruction model and the artificial fine model, and responding to user interaction commands, the visibility state of models at different detail levels in a unified scene is dynamically controlled, thereby achieving a smooth switching from macro to micro without context loss, and solving the problem of perspective jump and spatial cognition interruption caused by system switching in traditional solutions.
[0055] 2. In this invention, the perspective projection matrix of a physical camera is calculated by acquiring its intrinsic and extrinsic parameters; a proxy geometry is created in the virtual scene; during the rendering process, the video frame texture is dynamically projected onto the proxy geometry using the projection matrix, so that the video content can maintain the correct geometric perspective relationship as the user's viewpoint changes, thus solving the problems of perspective distortion and scale misalignment caused by ignoring the camera's geometric relationship in traditional texture pasting methods.
[0056] 3. In this invention, by constructing an end-to-end system architecture that integrates neural rendering technology and lightweight Web processing, efficient transmission and smooth rendering of photorealistic reconstruction models in ordinary browsers are achieved. This solves the network transmission bottleneck and terminal rendering pressure caused by the large amount of data in traditional Mesh models, and achieves high-quality visualization and real-time interaction of large-scale scenes in the Web environment. Attached Figure Description
[0057] Figure 1 This is a flowchart of a virtual-real fusion method based on Gaussian splash real-scene reconstruction proposed in this invention;
[0058] Figure 2 This is an example image of the sparse point cloud reconstruction result of a virtual-real fusion method based on Gaussian splash real scene reconstruction proposed in this invention;
[0059] Figure 3 This is a 3D model image after Gaussian rendering of a virtual-real fusion method based on Gaussian splash real-scene reconstruction proposed in this invention.
[0060] Figure 4 This is an artificial high-precision 3D model diagram of a virtual-real fusion method based on Gaussian splash real-scene reconstruction proposed in this invention;
[0061] Figure 5 This is an example image of video fusion based on a 3D Gaussian model, which is a virtual-real fusion method based on Gaussian splash real-scene reconstruction proposed in this invention.
[0062] Figure 6 This invention presents a three-dimensional Gaussian model, an artificially refined model, and a video proxy surface simultaneously loaded image, based on a virtual-real fusion method for Gaussian splash real-scene reconstruction.
[0063] Figure 7This is a schematic diagram of the Gaussian model coordinates and refined model coordinate mapping of a virtual-real fusion method based on Gaussian splash real scene reconstruction proposed in this invention;
[0064] Figure 8 This is a diagram showing the switching between the Gaussian model and the artificially refined model of a virtual-real fusion method based on Gaussian splash real-scene reconstruction proposed in this invention.
[0065] Figure 9 This is a detailed cross-sectional analysis diagram of a virtual-real fusion method based on Gaussian splash real-scene reconstruction proposed in this invention;
[0066] Figure 10 This is a detailed internal cross-sectional view of the virtual-real fusion method based on Gaussian splash real-scene reconstruction proposed in this invention.
[0067] Figure 11 This is an architecture diagram of a virtual-real fusion system based on Gaussian splash real-scene reconstruction proposed in this invention. Detailed Implementation
[0068] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0069] Example 1:
[0070] In a first embodiment of the present invention, the present invention provides a virtual-real fusion method based on Gaussian splash real-scene reconstruction, such as... Figures 1-10 As shown, it includes the following steps:
[0071] S1. Perform motion reconstruction processing on the acquired target scene image sequence to generate a sparse point cloud of the target scene;
[0072] Furthermore, structural treatment for motion recovery includes:
[0073] Adaptive keyframe extraction is performed on the target scene image sequence, and a set of keyframes is selected from the image sequence based on the image similarity and information entropy change between adjacent frames;
[0074] Image feature points are extracted and matched from the keyframe set to generate feature matching pairs;
[0075] Based on feature matching pairs, the camera's intrinsic and extrinsic parameter matrices for each keyframe are recovered through motion reconstruction structure processing, and a sparse point cloud of the target scene is generated simultaneously.
[0076] Specifically, the first step is to adaptively extract keyframes from the acquired target scene image sequence. This process analyzes the image similarity and information entropy changes between adjacent frames, automatically selecting keyframes with the highest information content and lowest ambiguity by calculating the differences between frames, thus forming a keyframe set. Image similarity calculation can use methods such as structural similarity index or mean squared error, while information entropy changes reflect the fluctuations in the complexity of image content. This adaptive mechanism effectively eliminates a large number of redundant frames, significantly reducing the subsequent computational burden, while ensuring that the keyframes can fully cover the diverse perspectives and details of the scene.
[0077] Image feature point extraction and matching are performed on the keyframe set. Feature point extraction can use scale-invariant or affine-invariant feature detection algorithms, such as SIFT or ORB operators, which can stably extract robust feature points from each keyframe. Feature matching generates consistent feature matching pairs among all keyframes through descriptor alignment and geometric verification. This step establishes the inter-frame correspondence and provides the necessary observation data for 3D reconstruction.
[0078] Based on feature matching pairs, camera parameters for each keyframe are recovered and sparse point clouds are generated through motion reconstruction structure processing. Motion reconstruction structure processing can employ incremental or global reconstruction frameworks. The incremental framework progressively adds images and optimizes camera pose and point clouds, while the global framework solves for all parameters simultaneously. This process minimizes reprojection error through the bundleleadjustment optimization method, thereby accurately calculating the camera's intrinsic and extrinsic parameter matrices. The intrinsic parameter matrix includes focal length and principal point coordinates, while the extrinsic parameter matrix defines the camera's rotation and translation in the world coordinate system. Simultaneously, a sparse point cloud of the target scene is generated from the matched feature points using a triangulation method. This point cloud serves as the geometric basis of the scene, providing accurate initial conditions for subsequent Gaussian splash training.
[0079] The data input was optimized by adaptive keyframe extraction, a reliable correspondence was established by feature matching, and high-precision recovery of camera parameters and sparse point clouds was achieved by motion reconstruction structure processing.
[0080] S2. Train the sparse point cloud using Gaussian splashing technology to generate a real-world 3D Gaussian model, and then perform lightweight processing on the real-world 3D Gaussian model to output a lightweight real-world model.
[0081] Furthermore, lightweighting includes:
[0082] Data compression processing is performed on the Gaussian point cloud in the real-world 3D Gaussian model by quantizing and encoding the spatial location, color attributes, and transparency attributes of the Gaussian points.
[0083] A hierarchical detail structure is constructed for the compressed Gaussian point cloud. By establishing a multi-level Gaussian point cloud representation from low density to high density, a lightweight real-world model suitable for multi-scale rendering on the web is generated.
[0084] Specifically, the training and lightweight processing steps of the Gaussian splashing technique are described in detail. This step aims to convert sparse point clouds into high-quality realistic 3D models and optimize them for real-time rendering on the web.
[0085] In the specific implementation process, the sparse point cloud is first trained using Gaussian splashing technology. This training process uses the sparse point cloud generated by motion reconstruction structure processing as a geometric prior, representing each 3D point as a Gaussian function with spatial position, covariance matrix, color attribute, and transparency attribute. During the training process, these Gaussian function parameters are optimized using a differentiable renderer, and its core rendering formula can be expressed as:
[0086]
[0087] in, Represents the final color value of a pixel. This represents the number of Gaussian functions along the ray. Indicates the first A Gaussian function color value, Indicates the first The opacity value of a Gaussian function. Indicates the first The formula uses the opacity value of a Gaussian function to achieve high-quality new perspective compositing through volume rendering principles.
[0088] After model training is completed, the real-world 3D Gaussian model undergoes lightweight processing. This lightweighting process first compresses the Gaussian point cloud data by quantizing and encoding the spatial coordinates, color attribute values, and transparency attribute values of the Gaussian points. The quantization process can be represented as follows:
[0089]
[0090] in, Represents the original attribute value. This represents the quantified attribute value. and These represent the minimum and maximum values of the attribute, respectively. This indicates the number of quantization bits; this attribute-separated encoding method ensures both compression efficiency and maintains model rendering quality.
[0091] Based on data compression, a hierarchical detail structure is further constructed for the Gaussian point cloud. This hierarchical detail structure is achieved by establishing a multi-level Gaussian point cloud representation from low density to high density. Specifically, an octree-based spatial clustering method is used to divide the space into voxel grids of different levels, with the Gaussian point density of each level satisfying the following:
[0092]
[0093] in, Indicates the first Gaussian point density of the layer This represents the Gaussian point density of the finest layer. Indicates the detail attenuation factor. It indicates the hierarchy level; during rendering, the appropriate hierarchy level is automatically selected based on the view distance to achieve seamless switching of detail levels;
[0094] The Gaussian splashing technique was used to convert sparse point clouds into high-quality 3D models. Data compression reduced the amount of model data, and hierarchical detail structure enabled adaptive rendering. This solved the data transmission and real-time rendering performance problems faced by large-scale real-scene models when rendering on the Web, and provided a lightweight real-scene model that ensures both visual quality and performance requirements for subsequent virtual-real fusion.
[0095] S3. Obtain the intrinsic and extrinsic parameter matrices of the physical camera and its real-time video stream; based on the extrinsic parameter matrix, create a proxy geometry in the virtual 3D scene that corresponds to the field of view of the physical camera;
[0096] Furthermore, creating the proxy geometry includes:
[0097] The view frustum of a physical camera in virtual 3D space is determined based on its extrinsic parameter matrix.
[0098] Calculate the geometric intersection region between the view frustum and the virtual 3D scene;
[0099] Within the intersecting region, corresponding proxy geometry is created based on the geometric features of the scene. The proxy geometry includes one or more planes that correspond to the main plane in the scene.
[0100] Furthermore, when the proxy geometry includes multiple planes, it also includes:
[0101] The video frames of the real-time video stream are divided into multiple image blocks according to the geometric structure of the virtual 3D scene;
[0102] Create a corresponding independent proxy plane for each image patch, and adjust the normal and spatial position of each proxy plane;
[0103] Each image block is treated as a texture and mapped onto the corresponding proxy plane for rendering.
[0104] Specifically, the view frustum in the virtual 3D space is first determined based on the physical camera's extrinsic matrix. The view frustum represents the camera's visible range in 3D space, and its spatial definition can be derived from the camera's extrinsic matrix. Specifically, the view frustum is enclosed by six planes, including the near clipping plane, the far clipping plane, and four side planes. The equation of each plane can be calculated by combining the camera's projection matrix and extrinsic matrix.
[0105] When calculating the geometric intersection region between the view frustum and the virtual 3D scene, a spatial geometric intersection algorithm is used. This process determines the 3D spatial region where the view frustum intersects with the scene model by testing the positional relationship between the geometric boundary of the scene and each plane of the view frustum. The intersection test can be based on the separating axis theorem or use a hierarchical bounding box acceleration structure to improve computational efficiency.
[0106] Within the defined intersecting regions, corresponding proxy geometries are created based on the scene's geometric features. For scene regions with obvious planar structures, proxy planes corresponding to the scene's main planes are created. The position and orientation of the proxy planes are consistent with the actual planar structures in the scene, such as the ground, walls, and other main planar structures. Each proxy plane is defined by its plane equation and boundary extent. The plane equation can be expressed as:
[0107]
[0108] in, Represents the plane normal vector. Represents the coordinates of any point on the plane. This is a plane constant term; the boundary of the proxy plane is determined by the intersection polygon of the view frustum and the scene plane.
[0109] When the scene geometry is complex or contains multiple main planes, a multi-plane projection approximation method is adopted. This method first divides the video frames of the real-time video stream into multiple image blocks according to the geometry of the virtual 3D scene. The image block segmentation is based on the geometric boundary features in the scene, such as obvious boundaries like corner lines and object edges. Each image block corresponds to a continuous planar region in the scene.
[0110] For each image patch, a corresponding independent proxy plane is created. These proxy planes have their own spatial position and normal direction. The normal and spatial position of each proxy plane are optimized and adjusted to best match the corresponding local geometric features of the scene. The adjustment process takes into account the continuity between adjacent proxy planes to ensure the visual coherence of the video projection.
[0111] During the rendering phase, each image block is treated as a texture and mapped onto the corresponding proxy plane for rendering. Texture mapping uses perspective-correct texture coordinates for calculation to ensure that the display of video content on each proxy plane conforms to geometric perspective. Multiple proxy planes together constitute an approximate representation of the geometry of the complex scene, achieving a natural fusion of the video stream and the 3D scene.
[0112] The effective area of video projection was determined by accurate view frustum calculation and geometric intersection analysis. The geometric relationship between video and scene was established by creating proxy geometry. The video fusion problem in complex scenes was solved by multi-plane projection approximation method.
[0113] S4. Calculate the projection matrix of the physical camera based on the intrinsic and extrinsic parameter matrices; during the rendering process, use the video frames of the real-time video stream as textures, and use the projection matrix to map the textures onto the proxy geometry for rendering;
[0114] Furthermore, before using video frames from a real-time video stream as textures, a video frame preprocessing step is also included:
[0115] Obtain the distortion coefficients of the physical camera calibrated using a calibration board;
[0116] Geometric distortion correction is performed on each video frame of the real-time video stream using distortion coefficients.
[0117] The corrected video frames are output as texture data to be mapped.
[0118] Furthermore, mapping textures onto proxy geometry for rendering includes:
[0119] In the shader, the projection matrix and view matrix of the physical camera are used to transform the world coordinates of each vertex on the proxy geometry to the clip space of the physical camera.
[0120] The clipping space coordinates are normalized to calculate the corresponding video texture UV coordinates.
[0121] Use UV coordinates to sample the texture of the video frame;
[0122] Based on the sampling results, the proxy geometry is rendered and output using the virtual camera's view matrix. When the virtual camera's view changes, the projection matrix of the physical camera is used to calculate the UV coordinates.
[0123] Specifically, the first step is to perform video frame preprocessing. Preprocessing includes obtaining the physical camera distortion coefficients calibrated using a calibration board. These distortion coefficients include radial and tangential distortion parameters, which are pre-obtained using camera calibration methods such as the Zhang Zhengyou calibration method. Geometric distortion correction is then performed on each video frame of the real-time video stream using these distortion coefficients. The correction process is based on a distortion model, which can be expressed as:
[0124]
[0125]
[0126] in, Represents the coordinates of the original image. Indicates the corrected image coordinates. Represents the radial distortion coefficient. Indicates the tangential distortion coefficient. This represents the squared distance to the image center; the corrected video frame is output as texture data to be mapped, providing a geometrically correct image source for subsequent rendering;
[0127] After preprocessing, the projection matrix of the physical camera is calculated based on the intrinsic and extrinsic parameter matrices. The projection matrix is obtained by combining the intrinsic and extrinsic parameter matrices, and the specific calculation formula is as follows:
[0128]
[0129] in, Represents the projection matrix. Represents the intrinsic parameter matrix, Represents the rotation matrix. This represents the translation vector; the projection matrix establishes the mapping relationship between three-dimensional spatial points and two-dimensional image coordinates.
[0130] During rendering, video frames from the real-time video stream are used as textures, and projection matrices are used to map these textures onto proxy geometry for rendering. This process is implemented in the shader and includes the following steps: using the projection matrix and view matrix of the physical camera, the world coordinates of each vertex on the proxy geometry are transformed to the clip space of the physical camera; the clip space coordinates are obtained through the projection matrix transformation, and the transformation formula is:
[0131]
[0132] in, Represents the world coordinates of the vertex. The crop space coordinates are represented by the transformed coordinates; the crop space coordinates are normalized to calculate the corresponding video texture UV coordinates; the normalization process is achieved through perspective division.
[0133]
[0134] in, Represents normalized device coordinates;
[0135] UV coordinates are obtained from normalized device coordinates through linear mapping:
[0136]
[0137] in, Represents the UV coordinates of the video texture, with values ranging from [0,1].
[0138] The UV coordinates are used to sample the texture of the video frame to obtain the texture color value at the corresponding position. Based on the sampling results, the proxy geometry is rendered and output in combination with the virtual camera's view matrix. In particular, when the virtual camera's view changes, the projection matrix of the physical camera is used to calculate the UV coordinates. This feature ensures that the video content is always projected based on the original perspective relationship of the physical camera and will not produce incorrect deformation as the virtual camera moves.
[0139] Precise distortion correction ensures the geometric accuracy of the video images, correct perspective relationships are established through projection transformation based on physical camera parameters, and visual consistency is maintained when the viewing angle changes by using a fixed projection matrix in the shader.
[0140] S5. Load the lightweight real-scene model, the artificially refined 3D model, and the proxy geometry with video textures in an integrated manner on the Web platform; establish the spatial mapping relationship between the lightweight real-scene model and the artificially refined 3D model;
[0141] Furthermore, establishing spatial mapping relationships includes:
[0142] In the lightweight real-world model, determine the spatial region corresponding to the finely crafted 3D model;
[0143] Establish a three-dimensional bounding box for the spatial region and obtain the coordinate range of the bounding box in the world coordinate system;
[0144] Establish a corresponding 3D spatial bounding box for a finely crafted 3D model and obtain its coordinate range in the world coordinate system;
[0145] Establish the association between the coordinate ranges of the two bounding boxes and store this association as a spatial mapping relationship.
[0146] Specifically, the lightweight real-world model, the meticulously crafted 3D model, and the proxy geometry rendered with video textures are loaded in an integrated manner on the web platform. The loading process is based on the WebGL graphics API, and the rendering order and spatial position of different types of models are coordinated through a unified scene graph management mechanism. The lightweight real-world model is represented by Gaussian point clouds with data compression and hierarchical detail processing, the meticulously crafted 3D model is represented by a mesh based on triangular facets, and the video proxy geometry is represented by planar geometry with video textures. These heterogeneous data are presented in an integrated manner in a unified web rendering environment.
[0147] When establishing spatial mapping relationships, the spatial region corresponding to the fine 3D artificial model is first determined in the lightweight real-scene model. This determination process is based on scene semantic information and geometric feature matching. By analyzing the correspondence between the structural features in the real-scene model and the objects represented by the artificial model, the common coverage area of the two in 3D space is identified. The determination of the corresponding region takes into account the geographic coordinate alignment and scale consistency in the actual scene.
[0148] A three-dimensional bounding box is constructed for the defined corresponding spatial region. This bounding box is defined as the smallest axially aligned cuboid that can completely contain the spatial region. The bounding box is determined by the coordinates of its smallest and largest vertices, and the coordinate range is represented in the world coordinate system. The smallest vertex coordinates... and the coordinates of the largest vertex satisfy:
[0149]
[0150]
[0151] in, These represent the spatial regions in all vertices. Minimum value of coordinates These represent the corresponding maximum values;
[0152] Simultaneously, a corresponding 3D spatial bounding box is established for the artificially refined 3D model. This bounding box completely contains all the geometric elements of the artificially refined 3D model. The coordinate range of the bounding box of the artificially refined 3D model is also represented in the world coordinate system to ensure that it is in the same coordinate reference system as the bounding box of the real scene model. The establishment of the bounding box adopts an extreme value search algorithm based on the model vertex coordinates to ensure that it accurately reflects the spatial occupancy range of the model.
[0153] A relationship is established between the coordinate ranges of two bounding boxes. This relationship is achieved by pairing the bounding boxes of the spatial regions in the real-world model with the bounding boxes of the artificially refined 3D model. The relationship records the correspondence between the two bounding boxes in the world coordinate system, including information on position offset, scale ratio, and spatial orientation consistency. The established relationship is stored in the form of structured data, including the source bounding box identifier, the target bounding box identifier, and the spatial transformation parameters between the two.
[0154] The stored spatial mapping relationship is queried and used in real time during subsequent interactions. When a user selects a specific area in the real scene model, the system quickly locates the corresponding detailed 3D model by querying the mapping relationship and triggers the corresponding model switching operation. The establishment of the mapping relationship provides an accurate spatial positioning basis for seamless switching from macro real scene to micro details.
[0155] The integrated loading mechanism enables unified management of multi-source heterogeneous data and establishes spatial relationships between real-world models and artificial models through precise spatial mapping. This ensures that users can maintain continuous spatial awareness during macro-level browsing and micro-level exploration, providing accurate spatial context information for subsequent interactive operations.
[0156] S6. Respond to user interaction with specific areas in the lightweight real-world model, hide or make the lightweight real-world model of the specific area transparent according to the spatial mapping relationship, and simultaneously display the artificially refined 3D model mapped to the specific area.
[0157] Furthermore, the method also includes section analysis, which includes:
[0158] Respond to user commands to cut a finely detailed 3D model;
[0159] Real-time calculation of cross-sections of finely crafted 3D models using shader programming;
[0160] Render and display the cross section.
[0161] Specifically, the system first responds to user interactions with specific areas of the lightweight real-world model. These interactions include, but are not limited to, clicking, selecting, or hovering, triggered by mouse or touch events on the web interface. When user interaction is detected, the system obtains the coordinates of the interaction location in three-dimensional space and determines the intersection point with the lightweight real-world model using a ray casting algorithm.
[0162] Based on the pre-established spatial mapping relationship, the system queries the artificially refined 3D model corresponding to the interaction area; the spatial mapping relationship is established based on the association relationship of 3D spatial bounding boxes, and the target artificially refined 3D model is determined by comparing the coordinates of the interaction point with the spatial inclusion relationship of each bounding box; this query process is accelerated by the spatial index structure to ensure real-time response performance.
[0163] After determining the target model, the system hides or makes transparent specific areas in the lightweight real-world model. Hiding is achieved by disabling Gaussian point cloud rendering in the corresponding area; transparency is achieved by adjusting the transparency attribute value of the Gaussian point cloud to make it semi-transparent. The degree of transparency can be dynamically adjusted based on the interaction distance between the user and the scene, providing a smooth visual transition effect.
[0164] Synchronously, the system displays a finely crafted 3D model mapped to a specific area; the display process includes activating the rendering state of the model, loading the necessary texture resources, and setting appropriate material parameters; to ensure visual continuity, the display position of the finely crafted 3D model is spatially aligned with the corresponding area in the real-world model, and this alignment is achieved based on the consistency of a pre-established world coordinate system;
[0165] During model switching, the system maintains the normal rendering of the video proxy geometry to ensure that the fusion effect of the real-time video stream and the 3D scene is not affected by the model switching operation; the entire switching process is completed within a single rendering frame to avoid visual flickering or screen tearing.
[0166] Furthermore, the system provides a sectioning analysis function; this function responds to sectioning commands initiated by the user on a finely detailed 3D model; the sectioning commands are triggered by tool buttons or gestures in the graphical user interface and include the position and orientation parameters of the sectioning plane;
[0167] The cross-sections of a finely detailed 3D model are calculated in real time using shader programming. In the fragment shader, the cross-section effect is calculated based on the geometric relationship between the cutting plane equation and the model's surface position. The cutting plane equation is expressed as:
[0168]
[0169] in, Represents the components of the plane normal vector. Represents the plane constant term. Represents the world coordinates of points on the model surface;
[0170] For each fragment, calculate its directed distance to the cutting plane:
[0171]
[0172] Based on directed distance The positive and negative values determine which side of the cutting plane the fragment is located on; fragments on the negative side are clipped, while fragments on the positive side are retained; at the edge of the section, anti-aliasing is achieved through a distance threshold to ensure smooth display of the section boundary;
[0173] When rendering and displaying cross sections, specific visual styles are applied to the section areas; optional visual styles include monochrome fill, semi-transparent display, or special highlight; the section rendering simultaneously displays the model's internal structure and external shape, providing a comprehensive spatial understanding; the section display supports dynamic adjustment, allowing users to modify the position and angle of the cutting plane in real time to obtain different section views.
[0174] This section achieves a seamless connection between macro and micro perspectives through precise interactive responses and model switching, and provides in-depth exploration capabilities of the model's internal structure through real-time cross-sectional analysis; it enhances the user's interactive experience and information acquisition efficiency in 3D scenes, and improves the functionality of the virtual-real fusion system.
[0175] Example 2:
[0176] In the construction of digital twin cities, urban management departments need to simultaneously observe the macroscopic real-world scene, view the details of local facilities, and monitor real-time dynamics. However, existing systems suffer from problems such as data fragmentation between the real-world 3D model and the meticulously crafted model, separation of video surveillance streams from the 3D scene space, difficulty in smoothly rendering large-scale real-world models on the web, and inconsistent switching between macroscopic and microscopic perspectives. To address these issues, this invention provides a virtual-real fusion system based on Gaussian splashing real-world reconstruction, the structure of which is as follows: Figure 11 As shown. The specific implementation process of this system is as follows:
[0177] The motion reconstruction processing module is used to perform motion reconstruction processing on the acquired target scene image sequence to generate a sparse point cloud of the target scene.
[0178] The Gaussian splashing processing module is used to train sparse point clouds using Gaussian splashing technology, generate a real-world 3D Gaussian model, and perform lightweight processing on the real-world 3D Gaussian model to output a lightweight real-world model.
[0179] The proxy geometry creation module is used to obtain the intrinsic parameter matrix, extrinsic parameter matrix and real-time video stream of the physical camera, and create a proxy geometry in the virtual 3D scene corresponding to the field of view of the physical camera based on the extrinsic parameter matrix;
[0180] The video texture mapping module is used to calculate the projection matrix of the physical camera based on the intrinsic and extrinsic parameter matrices. During the rendering process, the video frames of the real-time video stream are used as textures, and the projection matrix is used to map the textures onto the proxy geometry for rendering.
[0181] The scene loading and mapping module is used to load lightweight real-scene models, artificially refined 3D models, and proxy geometry with video textures in an integrated manner on the Web, and to establish the spatial mapping relationship between lightweight real-scene models and artificially refined 3D models.
[0182] The interactive response module is used to respond to user interaction with specific areas in the lightweight real-world model. It hides or makes the lightweight real-world model of the specific area transparent according to the spatial mapping relationship, and simultaneously displays the artificially refined 3D model mapped to the specific area.
[0183] Specifically, the motion reconstruction processing module first processes the acquired target scene image sequence; this module uses an adaptive keyframe extraction algorithm to filter the keyframe set based on the image similarity and information entropy change between adjacent frames; then, feature matching pairs are generated through feature point extraction and matching algorithms; finally, the camera parameters of each keyframe are recovered through motion reconstruction and sparse point cloud is generated; the sparse point cloud serves as the geometric basis of the scene, providing accurate initial conditions for subsequent processing.
[0184] The Gaussian splashing processing module receives sparse point cloud data and trains the model using Gaussian splashing technology. During training, the sparse point cloud is used as a geometric prior to optimize the spatial location, covariance matrix, color attribute, and transparency attribute parameters of the Gaussian function. After training, the generated real-world 3D Gaussian model undergoes lightweight processing, including compressing and encoding the Gaussian point cloud attribute data and constructing a hierarchical detail structure. This lightweight processing reduces data volume and establishes multi-level detail representations, making the model suitable for real-time rendering on the web.
[0185] The proxy geometry creation module is responsible for processing the real-time video stream fusion of physical cameras. This module obtains the intrinsic and extrinsic parameter matrices of the physical cameras, and determines the view frustum of the cameras in virtual space based on the extrinsic parameter matrix. By calculating the geometric intersection area between the view frustum and the virtual scene, a proxy geometry corresponding to the main plane of the scene is created within the intersection area. For complex scenes, a multi-plane projection approximation method is used to divide the video frames into multiple image blocks and create an independent proxy plane for each image block.
[0186] The video texture mapping module performs preprocessing and texture mapping of video frames. In the preprocessing stage, the distortion coefficients of the physical camera are used to perform geometric distortion correction on the video frames to eliminate the effects of lens distortion. The corrected video frames are used as input textures, and the projection matrix of the physical camera is used in the shader to perform perspective-correct texture mapping. During the mapping process, the projection matrix of the physical camera is used to calculate the UV coordinates to ensure that the correct perspective relationship is maintained when the virtual camera's viewpoint changes.
[0187] The scene loading and mapping module enables integrated management of multi-source data on the web. This module simultaneously loads lightweight real-scene models, finely crafted 3D models, and proxy geometry with video textures, and establishes a spatial mapping relationship between the real-scene model and the crafted model. Spatial mapping is achieved through a 3D bounding box, recording the correspondence between the two models in the world coordinate system. The establishment of the mapping relationship provides a spatial positioning basis for seamless switching between models.
[0188] The interactive response module handles user interactions with the 3D scene. When a user interacts with a specific area in the lightweight real-world model, the module determines the corresponding finely detailed 3D model by querying spatial mapping relationships. Then, the specific area in the real-world model is hidden or made transparent, while the corresponding finely detailed 3D model is activated and displayed. This module also provides a sectioning analysis function, which calculates and renders the cross sections of the model in real time through shader programming, supporting the exploration of the model's internal structure.
[0189] This system achieves integrated fusion and interaction of real-world 3D models, detailed artificial models, and real-time video streams on the web through the collaborative work of its various modules.
[0190] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A virtual-real fusion method based on Gaussian splash real scene reconstruction, characterized in that, Includes the following steps: S1. Perform motion reconstruction processing on the acquired target scene image sequence to generate a sparse point cloud of the target scene; S2. Train the sparse point cloud using Gaussian splashing technology to generate a real-world 3D Gaussian model, and then perform lightweight processing on the real-world 3D Gaussian model to output a lightweight real-world model. S3. Obtain the intrinsic parameter matrix, extrinsic parameter matrix, and real-time video stream of the physical camera; Based on the extrinsic parameter matrix, a proxy geometry corresponding to the field of view of the physical camera is created in the virtual 3D scene; S4. Calculate the projection matrix of the physical camera based on the intrinsic and extrinsic parameter matrices; During the rendering process, video frames from the real-time video stream are used as textures, and a projection matrix is used to map the textures onto the proxy geometry for rendering. S5. Load lightweight real-world models, meticulously crafted 3D models, and proxy geometry with rendered video textures in an integrated manner on the web platform; Establish a spatial mapping relationship between lightweight real-world models and meticulously crafted 3D models; Establishing spatial mapping relationships includes: In the lightweight real-world model, determine the spatial region corresponding to the finely crafted 3D model; Establish a three-dimensional bounding box for the spatial region and obtain the coordinate range of the bounding box in the world coordinate system; Establish a corresponding 3D spatial bounding box for a finely crafted 3D model and obtain its coordinate range in the world coordinate system; Establish the association between the coordinate ranges of the two bounding boxes, and store this association as a spatial mapping relationship; S6. Respond to user interaction with specific areas in the lightweight real-world model, hide or make the lightweight real-world model of the specific area transparent according to the spatial mapping relationship, and simultaneously display the artificially refined 3D model mapped to the specific area. The process of restoring the motion structure includes: Adaptive keyframe extraction is performed on the target scene image sequence, and a set of keyframes is selected from the image sequence based on the image similarity and information entropy change between adjacent frames; Image feature points are extracted and matched from the set of keyframes to generate feature matching pairs; Based on the feature matching pairs, the camera's internal and external parameter matrices for each key frame are recovered through motion recovery structure processing, and a sparse point cloud of the target scene is generated simultaneously. The lightweighting process includes: The Gaussian point cloud in the real-world 3D Gaussian model is subjected to data compression processing, and the spatial position, color attribute and transparency attribute of the Gaussian points are quantized and encoded. A hierarchical detail structure is constructed on the compressed Gaussian point cloud. By establishing a multi-level Gaussian point cloud representation from low density to high density, a lightweight real-world model suitable for multi-scale rendering on the Web is generated. Creating proxy geometry includes: The view frustum of a physical camera in virtual 3D space is determined based on its extrinsic parameter matrix. Calculate the geometric intersection region between the view frustum and the virtual 3D scene; Within the intersecting region, a corresponding proxy geometry is created based on the geometric features of the scene. The proxy geometry includes one or more planes corresponding to the main plane in the scene.
2. The virtual-real fusion method based on Gaussian splash real scene reconstruction according to claim 1, characterized in that, When the proxy geometry includes multiple planes, it also includes: The video frames of the real-time video stream are divided into multiple image blocks according to the geometric structure of the virtual 3D scene; Create a corresponding independent proxy plane for each image patch, and adjust the normal and spatial position of each proxy plane; Each image block is treated as a texture and mapped onto the corresponding proxy plane for rendering.
3. The virtual-real fusion method based on Gaussian splash real scene reconstruction according to claim 1, characterized in that, Before using video frames from a real-time video stream as textures, a video frame preprocessing step is also included: Obtain the distortion coefficients of the physical camera calibrated using a calibration board; Geometric distortion correction is performed on each video frame of the real-time video stream using distortion coefficients. The corrected video frames are output as texture data to be mapped.
4. The virtual-real fusion method based on Gaussian splash real-scene reconstruction according to claim 1, characterized in that, The step of mapping textures onto proxy geometry for rendering includes: In the shader, the projection matrix and view matrix of the physical camera are used to transform the world coordinates of each vertex on the proxy geometry to the clip space of the physical camera. The clipping space coordinates are normalized to calculate the corresponding video texture UV coordinates. Use UV coordinates to sample the texture of the video frame; Based on the sampling results, the proxy geometry is rendered and output using the virtual camera's view matrix. When the virtual camera's view changes, the projection matrix of the physical camera is used to calculate the UV coordinates.
5. The virtual-real fusion method based on Gaussian splash real-scene reconstruction according to claim 1, characterized in that, The method further includes cross-sectional analysis, which includes: Respond to user commands to cut a finely detailed 3D model; Real-time calculation of cross-sections of finely crafted 3D models using shader programming; Render and display the cross section.
6. A virtual-real fusion system based on Gaussian splash real-scene reconstruction, characterized in that, The virtual-real fusion method based on Gaussian splash real-scene reconstruction as described in any one of claims 1-5, the system comprising the following modules: The motion reconstruction processing module is used to perform motion reconstruction processing on the acquired target scene image sequence to generate a sparse point cloud of the target scene. The Gaussian splashing processing module is used to train sparse point clouds using Gaussian splashing technology, generate a real-world 3D Gaussian model, and perform lightweight processing on the real-world 3D Gaussian model to output a lightweight real-world model. The proxy geometry creation module is used to obtain the intrinsic parameter matrix, extrinsic parameter matrix and real-time video stream of the physical camera, and create a proxy geometry in the virtual 3D scene corresponding to the field of view of the physical camera based on the extrinsic parameter matrix; The video texture mapping module is used to calculate the projection matrix of the physical camera based on the intrinsic and extrinsic parameters. During the rendering process, the video frames of the real-time video stream are used as textures, and the projection matrix is used to map the textures onto the proxy geometry for rendering. The scene loading and mapping module is used to load lightweight real-scene models, artificially refined 3D models, and proxy geometry with video textures in an integrated manner on the Web, and to establish the spatial mapping relationship between lightweight real-scene models and artificially refined 3D models. The interactive response module is used to respond to user interaction with specific areas in the lightweight real-world model. It hides or makes the lightweight real-world model of the specific area transparent according to the spatial mapping relationship, and simultaneously displays the artificially refined 3D model mapped to the specific area.
Citation Information
Patent Citations
Determination method and device of sectioning plane of three-dimensional model
CN108171804A
Three-dimensional visualization multi-scene rendering application integration method
CN117689791A
Three-dimensional immersive content interaction method and system based on 3D Gaussian spattering
CN120472126A
Three-dimensional video fusion method based on camera self-calibration and projection texture mapping
CN120852711A
