Unstructured volume data hybrid rendering method and space interaction system

Abstract

The invention relates to the technical field of computer graphics and virtual reality, and discloses an unstructured volume data hybrid rendering method based on WebXR and Gaussian splash technologies and a space interaction system. According to the method, a standardized visibility container protocol is constructed, a hysteresis space state machine is combined, and real-time logic cutting, efficient streaming rendering and smooth space interaction of unstructured volume data in any shape are achieved through general rendering entity abstraction, streaming zero-copy rendering pipelines and core algorithm innovation. According to the method, the problems of difficulty in geometric constraint of 3D Gaussian splashing data in the existing WebXR environment, bottleneck of rendering performance of a mobile Web end, single jitter of spatial state perception and the like are solved, and the method is suitable for various scenes such as AR transmission doors, photoscopes, holographic showcases and the like and has wide universality and practicability.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and virtual reality technology, specifically to a method and system for real-time logical pruning, streaming rendering, and spatial interactive control of unstructured volumetric data such as 3D Gaussian splashes in a WebXR environment. Background Technology

[0002] In the current WebXR field, 3D Gaussian Splatting (3DGS) technology has attracted much attention due to its ability to render photorealistic 3D scenes on mobile devices with low computational cost. However, 3D Gaussian splatting data is essentially unstructured volumetric data composed of discrete light spots, which presents many challenges when applied to complex mixed reality (MR) interactions.

[0003] One of the challenges stems from the geometric constraints of unstructured data. 3DGS data consists of discrete point clouds lacking topological mesh structures. Traditional Boolean operations in graphics (such as punching holes and cutting) are extremely expensive and difficult to implement in real time, preventing developers from using standard geometry to accurately clip the shape of Gaussian scenes. For example, it is difficult to "display Gaussian clouds only inside a virtual cube placed on a desktop" or "view the internal structure of an object through a circular virtual lens," often resulting in rendering overflow or depth occlusion errors.

[0004] The second challenge stems from the performance bottleneck of rendering large amounts of data on mobile web devices. High-fidelity volumetric data typically contains millions of particles. Existing web loading solutions mostly rely on the CPU main thread for decompression or geometric sorting, which can easily cause the main thread to be blocked when performing high-frequency spatial interactions (such as 6DoF roaming) on ​​mobile devices, resulting in loss of SLAM (Simultaneous Localization) tracking and screen stuttering.

[0005] The third challenge stems from the limited and fluctuating spatial state perception. Existing WebAR rendering engines lack a universal mechanism to perceive the topological relationship (such as inclusion, intersection, and separation) between the user device (camera) and the virtual volume region (Container). Furthermore, the lack of a hysteresis processing mechanism at the physical boundaries between virtual and reality (such as door frames) leads to misjudgments of view frustum culling, causing screen flickering or logical reversals, which greatly undermines the sense of immersion.

[0006] Therefore, there is an urgent need for a technical solution that can overcome the above-mentioned technical bottlenecks and achieve real-time logical pruning, efficient streaming rendering, and smooth spatial interaction of unstructured volumetric data. Summary of the Invention

[0007] This invention provides a method for hybrid rendering of unstructured volumetric data and a spatial interaction system based on WebXR and Gaussian splashing technology, which effectively overcomes the above-mentioned defects of the prior art.

[0008] Specifically, this invention provides a method for hybrid rendering of unstructured volumetric data based on WebXR and Gaussian splashing technology. The method includes the following steps: Entity abstraction and scene construction: Objects in the WebXR-perceived scene perceived by the WebXR camera are abstracted into logical containers, volumetric entities, and physical boundaries; wherein the logical container is a closed mesh, configured as invisible and with a preset template reference value; the volumetric entity is a Gaussian splashed point cloud, configured to forcibly disable frustum culling and subject to template testing constraints; the physical boundary is a normal mesh, configured as visible and with depth written; Streaming zero-copy loading: A data path is constructed directly from the network stream to the video memory. By intercepting the Fetch response stream and reading binary data blocks, the data is directly written to the video memory for instantiation and rendering, thereby bypassing the array buffer parsing and script object construction process of the main thread; Hysteresis spatial state determination: The world pose of the WebXR camera and the world pose of the logical container are obtained. The depth component of the WebXR camera in the local coordinate system of the logical container is calculated through coordinate inverse transformation, and a hysteresis threshold is introduced. Based on the hysteresis logic, the WebXR... The spatial state of the camera relative to the logical container; adaptive geometric clipping steps: according to the preset rendering order, firstly, the logical container is used as a clipping mold, and template reference values ​​are written to the template buffer without outputting colors; then, the clipping rules are dynamically switched according to the determined spatial state, and template testing is performed when drawing the volume body, retaining only the discrete point cloud fragments of the current area that conform to the clipping rules, so as to realize the constraint clipping of arbitrary shapes of the volume body.

[0009] Preferably, in the determination step, the local position of the camera in the container's local coordinate system is calculated through an inverse coordinate system transformation. ,in The inverse of the container world matrix. Extract local position for camera world pose The Z-axis component in the depth direction in the coordinate system Introducing a hysteresis threshold The spatial state is determined based on hysteresis logic.

[0010] Preferably, when At that time, the next spatial state Set to immersive mode, ignore cropping, and display the main body in full screen; when At that time, the next spatial state When set to window mode, apply clipping and observe the main volume through the window defined by the logical container; otherwise, when... At that time, the next spatial state Maintain the current spatial state.

[0011] Preferably, the volumetric body is a Gaussian splash point cloud, with the attribute set to be constrained by template testing and the view frustum culling is forcibly disabled, used to carry visual content.

[0012] Preferably, the logical container is an arbitrary closed Mesh, with its attribute set to invisible and only written to template values, used to define the "visible window" or "clipping region".

[0013] Preferably, the physical boundary is a regular Mesh, with the attribute set to visible and depth written, to provide occlusion relationships and physical texture, and to refine the clipping edges.

[0014] The present invention also provides an unstructured volumetric data spatial interaction system based on WebXR and Gaussian splashing technology, for performing the methods described above.

[0015] In summary, this invention provides a hybrid rendering method and spatial interaction system for unstructured volumetric data based on WebXR and Gaussian splashing technology. This method constructs a standardized visibility container protocol, combines it with a hysteretic spatial state machine, and achieves real-time logical pruning, efficient streaming rendering, and smooth spatial interaction for unstructured volumetric data of arbitrary shapes through general rendering entity abstraction, a streaming zero-copy rendering pipeline, and core algorithm innovation. This invention solves problems such as the difficulty of geometric constraints on 3D Gaussian splashed data, the performance bottleneck of mobile web rendering, and the single-dimensional jitter of spatial state perception in existing WebXR environments. It is applicable to various scenarios such as AR portals, perspective mirrors, and holographic display cases, and has broad versatility and practicality. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be discussed below. Obviously, the technical solutions described in conjunction with the accompanying drawings are only some embodiments of the present invention. For those skilled in the art, other embodiments and their accompanying drawings can be obtained based on the embodiments shown in these drawings without creative effort.

[0017] Figure 1 A system block diagram of a spatial interaction system for hybrid rendering of unstructured volumetric data based on WebXR and Gaussian splashing technology according to the present invention is shown. Figure 2The flowchart of the unstructured volumetric data hybrid rendering method based on WebXR and Gaussian splashing technology according to the present invention is shown. Detailed Implementation

[0018] The technical solutions of various embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments described in the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Unlike existing technologies that have shortcomings, this invention provides a hybrid rendering method and spatial interaction system for unstructured volumetric data based on WebXR and Gaussian splashing technology. This method enables real-time logical pruning, efficient streaming rendering, and smooth spatial interaction of unstructured volumetric data of arbitrary shapes, solving geometric constraint problems, performance bottlenecks, and spatial interaction jitter issues.

[0020] The hybrid rendering method of the present invention will be described step by step below with reference to the accompanying drawings. Figure 1 A system block diagram of a spatial interaction system for hybrid rendering of unstructured volumetric data based on WebXR and Gaussian splashing technology according to the present invention is shown. Figure 2 The flowchart of the unstructured volumetric data hybrid rendering method based on WebXR and Gaussian splashing technology according to the present invention is shown.

[0021] The initial steps of this hybrid rendering method are general rendering entity abstraction and scene construction. In this step, objects in the scene are abstracted into three types of entities: logical containers, volumetric entities, and physical boundaries, and a strict rendering order is defined.

[0022] The logical container is any closed Mesh, with the property set to invisible (ColorMask=False) and only written to the template value (Stencil Write Ref=1), used to define the "visible window" or "clipping region".

[0023] The main volume is a Gaussian splash point cloud, with its properties set to be constrained by the Stencil Test and frustum culling forcibly disabled to prevent content from disappearing due to differences in bounding box calculations when viewed from the side, and is used to carry visual content.

[0024] The physical boundary is a regular Mesh, with the property set to visible and depth write to provide occlusion relationships and physical texture (such as door frames and flashlight casings), and to refine the clipping edges. The rendering order of the three types of entities is: logical container [Order 0] → main volume [Order 1] → physical boundary [Order 2].

[0025] The next step is the volumetric data streaming and zero-copy rendering step, which builds a data path from the network stream directly to the video memory (VRAM) to achieve real-time response to gigabyte-level volumetric data.

[0026] The specific workflow for this step is as follows: first, the Fetch response stream is intercepted; then, binary data chunks are read and directly written to the WebGLBuffer / WebGPUBuffer, which is then instantiated and drawn by the GPU. This process bypasses the main thread's array buffer parsing and JS object construction process, avoiding main thread blocking and resolving the computing power bottleneck on mobile web devices.

[0027] The next step is spatial state determination and hysteresis control. This step first obtains the world pose (MatrixWorld) of the WebXR camera and the world pose (MatrixWorld) of the container / anchor point, and calculates the local position of the camera in the local coordinate system of the container through inverse coordinate system transformation (Latex operation). (in The inverse of the container world matrix. (For camera world pose), extract The Z-axis component (depth distance) is Introducing a hysteresis threshold (Hysteresis Threshold) determines the spatial state based on hysteresis logic. The determination process will be explained in detail below.

[0028] when At that time, the next spatial state Set to INSIDE (immersive mode). In this mode, the template test function is set to ALWAYS, ignoring cropping and displaying the main body in full screen.

[0029] when At that time, the next spatial state Set to OUTSIDE (window mode), then set the template test function to EQUAL, apply clipping, and observe the main volume through the window defined by the logical container.

[0030] In other cases, that is Next spatial state Maintain the current spatial state.

[0031] This step implements an anti-jitter spatial triggering mechanism, eliminates boundary flickering issues, and supports multiple interaction modes.

[0032] Next, the geometric clipping steps for unstructured data are introduced. During the rasterization stage, a logical container is used as a "clipping dummy," and a marker (Ref=1) is written to the stencil buffer without outputting color. In the Gaussian splashing fragment shader stage, a stencil test (Func=EQUAL) is performed, retaining only the volumetric content of the region where the stencil value equals the reference value, thus achieving arbitrary shape constraint clipping of the discrete point cloud. The computational cost of this method is independent of the number of point clouds and only depends on the screen resolution, making it highly suitable for mobile devices.

[0033] In this step, the "spatial state (the relative topological relationship between the WebXR camera and the logical container)" is the "switch and mode controller" of the clipping logic. It does not directly participate in low-level clipping operations such as stencil buffer writing and fragment shader testing. Instead, it determines "whether to clip" and "how to clip" by dynamically switching "clipping rules", ultimately achieving "precise clipping that adapts to scene interaction".

[0034] The underlying core of unstructured data geometric clipping is "template logic operation" (based on writing template tags to a logic container to test template values ​​in the fragment shader), but "spatial state" can be used in a specific way to upgrade clipping from "fixed rules" to "adaptive interaction".

[0035] When the WebXR camera moves, causing a change in the "spatial state", the system will first trigger the "cropping rule configuration".

[0036] Specifically, if the state switches to INSIDE, the system sets the stencil test function to ALWAYS. At this time, the stencil test in the fragment shader stage is "disabled" (all Gaussian point cloud fragments can pass the test), and geometric clipping is "disabled". In this case, the user has entered the "interactive core area" defined by the logical container (such as passing through an AR portal or getting close to the inside of a holographic display case) and needs to view the complete unstructured data (Gaussian point cloud) in full screen without clipping restrictions.

[0037] If the state switches to OUTSIDE, the system sets the template test function to EQUAL. In this case, only the area where the logical container writes the template value (Ref=1) can pass the test. Gaussian point cloud fragments outside this area are discarded, and geometric clipping is "enabled". In this situation, when the user is outside the interactive area, they need to observe the content through the "visible window" of the logical container (such as looking inside through a lens or looking at a virtual scene through a door frame). Clipping can prevent rendering overflow and improve the realism.

[0038] Furthermore, it is necessary to consider that the core defect of unstructured data (Gaussian point clouds) is that it "has no topological structure and cannot be pruned by Boolean operations," while "spatial state" can be adapted to this characteristic by switching pruning rules.

[0039] When OUTSIDE enables clipping, template logic operations (which are independent of the number of point clouds and depend only on screen resolution) can efficiently achieve "arbitrary shape clipping," solving the geometric constraint problem of unstructured data.

[0040] When INSIDE disables clipping, the system forcibly disables "frustum culling" for the main volume, preventing the point cloud content from disappearing due to differences in camera side view and bounding box calculations. At this time, the combination of "no clipping + no culling" ensures the complete presentation of unstructured data while avoiding rendering anomalies caused by its topological defects.

[0041] This concludes the basic introduction to the hybrid rendering method provided by this invention. Next, we will introduce the spatial interaction system based on this method.

[0042] The spatial interaction system of this invention is built around a "multi-level rendering compositor" and a "spatial relative pose state machine". The following sections will introduce the interaction system in modules.

[0043] The system's starting module is the general rendering entity abstraction module. This module is used to abstract scene objects into three types of entities: logical containers, volumetric entities, and physical boundaries. It defines the attributes (such as visibility, template writing, depth writing, etc.) and rendering order of each entity, providing a foundation for subsequent rendering and pruning.

[0044] The next module is the streaming zero-copy rendering module, which is used to build a data path from the network to the video memory, execute the streaming loading and zero-copy rendering process of volume data, avoid main thread blocking, and ensure real-time response to large volume data.

[0045] The next module is the hysteresis spatial state machine module, which is used to obtain the world pose of the camera and the container, calculate the relative position through inverse coordinate transformation, determine the spatial state based on hysteresis logic, dynamically switch the rendering mode, support multiple interaction modes such as "passing through a door", "probe", and "close-range triggering", and eliminate boundary flicker.

[0046] The next module is the geometry trimming module. This module uses logical containers as molds to trim unstructured data into arbitrary shapes through template logic operations. It supports trimming molds of various geometric shapes such as spheres, cylinders, and custom models, and is suitable for a variety of application scenarios.

[0047] The system is ultimately configured with a multi-level rendering compositor, which is used to render and composite three types of entities according to a preset rendering order (e.g., from logical containers to volumetric bodies and then to physical boundaries), handle rendering logic such as template testing and depth testing, and output the final mixed reality rendering image.

[0048] To describe the invention more intuitively, examples of its application in specific scenarios will be introduced below.

[0049] This example is based on an AR X-Ray Vision application scenario. In this scenario, the hybrid rendering method and spatial interaction system of this invention are used to realize the AR X-Ray Vision function, allowing users to view a 3D Gaussian splash model inside an object through a virtual lens on their mobile device.

[0050] In this example, the system deployment and entity configuration can be as follows: The general rendering entity abstract module configures three types of entities: logical container (entity A), which uses a circular closed mesh with a radius of 5cm, sets ColorMask=False, Stencil WriteRef=1, and renders in order 0, defining the visible area of ​​the perspective mirror; volume body (entity B): loads 3D Gaussian splatter point cloud data (approximately 2GB) containing the object's internal structure, sets it to be subject to stencil testing constraints, disables FrustumCulling, and renders in order 1; physical boundary (entity C): uses a lens shell mesh, sets it to be visible, sets DepthWrite=True, and renders in order 2, providing physical texture and refining the lens edges.

[0051] The streaming zero-copy rendering module is deployed on the web frontend, and its data loading path is configured as follows: network stream → binary chunk read → WebGLBuffer write → GPU instantiation and drawing. The hysteresis spatial state machine module sets the hysteresis threshold and defines the mapping relationship between spatial states and rendering modes.

[0052] The specific operation process will be described below under the above entity configuration.

[0053] First, the user opens the WebXR application, the mobile device's camera captures the real-world scene, and the streaming zero-copy rendering module begins loading 3D Gaussian splash point cloud data. The data is streamed directly to the video memory via the network, without needing to be parsed by the main thread. The loading process is smooth, and SLAM tracking is normal.

[0054] Subsequently, the general rendering entity abstraction module renders the three types of entities in the order of Order 0→Order 1→Order 2, and the logical container (circular Mesh) writes the flag Ref=1 to the template buffer.

[0055] Then, the hysteresis spatial state machine module acquires the world pose of the camera and the logic container (virtual lens) in real time and calculates the position of the camera in the local coordinate system of the lens.

[0056] When a user brings their mobile device close to a real-world object, the spatial state switches to INSIDE, the rendering mode switches to immersive mode, the template test function is set to ALWAYS, and the user can view the 3D Gaussian model inside the object in full screen.

[0057] When the user moves away from the object, the spatial state switches to OUTSIDE, the rendering mode switches to window mode, the template test function is set to EQUAL, and only the circular lens area displays the internal model, while other areas display the real scene.

[0058] The spatial state remains unchanged when the user moves within the boundary area to avoid screen flickering.

[0059] The geometric clipping module achieves the clipping of circular areas through template testing, retaining only the point cloud content within the lens range. The clipping edges are modified by physical boundaries (lens shell), resulting in a natural visual effect.

[0060] This concludes the basic introduction of the invention. Specifically, the "volume-cutting container" technology proposed in this invention has broad applicability, allowing developers to define arbitrary geometric shapes as "logic molds." It is not only applicable to AR portals but can also directly support various application scenarios such as AR X-Ray Vision, holographic display cases, and virtual excavation, demonstrating strong versatility.

[0061] Furthermore, in this invention, the streaming zero-copy rendering pipeline constructs a data path directly from the network stream to the video memory, bypassing the main thread's data processing, enabling real-time response to GB-level data volumes, effectively solving the performance bottleneck of large-scale rendering on mobile web devices, and ensuring that the SLAM tracing thread is not blocked.

[0062] Moreover, this invention proposes an anti-jitter spatial triggering mechanism for the adaptive hysteresis spatial state machine, which separates the rendering logic from the interaction logic. A single codebase can support multiple interaction modes, eliminating screen flickering at the boundary between virtual and reality, and ensuring the continuity and immersion of the interactive experience.

[0063] Furthermore, by applying the basic architecture of this invention, the geometric clipping method based on template logic operation solves the problem that discrete point clouds cannot be clipped using traditional geometric Boolean operations. Moreover, the computational overhead is only related to the screen resolution and is independent of the number of point clouds, making it suitable for low-computing-power scenarios on mobile devices. It also offers high clipping accuracy and good real-time performance.

[0064] The above description is merely an exemplary embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hybrid rendering method for unstructured volumetric data based on WebXR and Gaussian splashing technology, characterized in that, The method includes the following steps: Entity abstraction and scene construction steps: The objects in the WebXR-aware scene perceived by the WebXR camera are abstracted into logical containers, volumetric entities, and physical boundaries; wherein, the logical container is a closed mesh, configured to be invisible and preset template reference values; the volumetric entity is a Gaussian splash point cloud, configured to forcibly disable view frustum culling and subject to template test constraints; the physical boundary is a normal mesh, configured to be visible and have its depth written. Streaming zero-copy loading steps: Construct a data path from the network stream directly to the video memory, intercept the Fetch response stream and read binary data blocks, and write them directly to the video memory for instantiation and drawing, thereby bypassing the array buffer parsing and script object construction process of the main thread; Hysteresis spatial state determination steps: Obtain the world pose of the WebXR camera and the world pose of the logical container, calculate the depth component of the WebXR camera in the local coordinate system of the logical container through inverse coordinate transformation, introduce a hysteresis threshold, and determine the spatial state of the WebXR camera relative to the logical container based on the hysteresis logic. Adaptive geometric clipping steps: According to the preset rendering order, the logical container is first used as a clipping mold, and template reference values ​​are written to the template buffer without outputting colors; then, the clipping rules are dynamically switched according to the determined spatial state, and template testing is performed when drawing the main volume, retaining only the discrete point cloud fragments of the current area that conform to the clipping rules, so as to realize the constraint clipping of arbitrary shapes of the main volume.

2. The method according to claim 1, characterized in that, In the hysteresis space state determination step, the local position of the camera in the container's local coordinate system is calculated through inverse coordinate transformation. ,in The inverse of the container world matrix. Extract local position for camera world pose The Z-axis component in the depth direction in the coordinate system Introducing a hysteresis threshold The spatial state is determined based on hysteresis logic.

3. The method according to claim 2, characterized in that, when When the next spatial state is set to immersive mode, cropping is ignored and the main body is displayed in full screen; when At this time, the next spatial state is set to window mode, and clipping is applied to observe the main volume through the window defined by the logical container; when When the current space state is maintained, the next space state remains unchanged.

4. The method according to claim 1, characterized in that, The logical container is any closed Mesh, with the property set to invisible and only written to the template value, used to define the "visible window" or "clipping region".

5. The method according to claim 1, characterized in that, The physical boundary is a regular Mesh, with the property set to visible and depth written to provide occlusion relationships and physical texture, and to refine the clipping edges.

6. A spatial interaction system for unstructured volumetric data based on WebXR and Gaussian splashing technology, for performing the method as described in any one of claims 1-5.

Images