Low-latency 3D gaussian spattering rendering method, system and application for multi-body dynamic scene

Abstract

This invention provides a low-latency 3D Gaussian splash rendering method, system, and application for multi-body dynamic scenes, relating to the field of computer graphics technology. This scheme divides the 3D Gaussian splash scene into multiple Gaussian semantic volume subsets according to the Gaussian semantic volume constraints of the simulated object. It pre-computes and caches the Boolean index mask and Gaussian semantic volume ID mapping array for each Gaussian point's Gaussian semantic volume. It receives control data for each frame, including virtual camera intrinsic and extrinsic parameters and pose parameters of each Gaussian semantic volume, with the pose parameters transmitted in the form of a compact binary array of preset dimensions. It copies the static Gaussian template to the output buffer, extracts the dynamic Gaussian points and their corresponding Gaussian semantic volume poses using the Boolean index mask and Gaussian semantic volume ID mapping array, and completes the position transformation and rotation synthesis of the dynamic Gaussian points. Finally, it directly submits the rendered output tensor to the hardware encoder for encoding processing.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, and in particular to a low-latency 3D Gaussian splash rendering method, system and application for multi-body dynamic scenes. Background Technology

[0002] In fields such as AI inference testing (e.g., Vision-Language-Action network, i.e., VLA success rate verification), robot simulation, autonomous driving scene generation, virtual reality and augmented reality (AR / VR), 3D Gaussian Splatting (GS) is widely used to generate dynamic simulation scenes due to its high-fidelity rendering capabilities.

[0003] However, existing technologies have the following key drawbacks: Inefficient dynamic scene updates: When a scene contains multiple movable Gaussian semantic bodies (such as robot joints or vehicle parts), traditional GS implementations need to perform global pose updates on all Gaussian points, resulting in an update delay of >10ms per frame, which severely slows down the AI ​​testing iteration speed.

[0004] Rendering-encoding pipeline redundancy: The rendering result needs to be copied from the GPU memory to the CPU memory, and then processed by a software encoder (such as x264) or a general hardware encoding interface, introducing an additional 5-20ms delay and occupying PCIe bandwidth.

[0005] Inefficient transmission of control commands: Pose data is often transmitted in JSON / Protobuf format (with protocol header), resulting in low uplink bandwidth utilization (>8KB / frame), which significantly increases network load in batch simulation tests.

[0006] Limited applicability: Existing solutions are not optimized for simulation testing scenarios and cannot meet the needs of rapid iterative verification of AI models. Summary of the Invention

[0007] Purpose of the invention: To propose a low-latency 3D Gaussian splash rendering method, system and application for multi-body dynamic scenes, so as to solve the above-mentioned problems existing in the prior art.

[0008] To achieve the above objectives, this invention first proposes a low-latency 3D Gaussian splash rendering method for multi-body dynamic scenes, comprising the following steps: The 3D Gaussian splash scene is divided into multiple Gaussian semantic volume subsets according to the Gaussian semantic volume constraints of the simulation object. The Boolean index mask and Gaussian semantic volume ID mapping array of the Gaussian semantic volume to which each Gaussian point belongs are pre-calculated and cached. Receive control data for each frame, which includes virtual camera intrinsic and extrinsic parameters and pose parameters of each Gaussian semantic volume, wherein the pose parameters are transmitted in the form of a compact binary array of a preset dimension; Copy the static Gaussian template to the output buffer, extract the dynamic Gaussian points and their corresponding Gaussian semantic body poses using the Boolean index mask and Gaussian semantic body ID mapping array, and complete the position transformation and rotation synthesis of the dynamic Gaussian points by executing kernel functions in parallel. The tensor of the rendering output is directly submitted to the hardware encoder, and the pointer in the GPU memory is passed through the memory-level encoding and decoding interface to complete the encoding process.

[0009] As a preferred embodiment, both the Boolean index mask and the Gaussian semantic body ID mapping array are cached in GPU memory.

[0010] As a preferred embodiment, the compact binary array of the preset dimensions has a shape of (N_body, 7), and the 7 dimensions correspond to [x, y, z, qx, qy, qz, qw] respectively. The control data has no protocol header and no text serialization processing.

[0011] As a preferred solution, the kernel function module of the dynamic compilation is called to execute the kernel function to complete the position transformation and rotation synthesis of the dynamic Gaussian points; The position transformation satisfies the formula ; The rotational synthesis satisfies the formula ; Where xyz_new represents the position of the transformed Gaussian point in the world coordinate system; body_quat represents the pose quaternion of the Gaussian semantic body; R(*) represents the transformation function from quaternion to rotation matrix; xyz_local represents the original position of the Gaussian point in the local coordinate system of the Gaussian semantic body; body_pos represents the position of the Gaussian semantic body; rot_new represents the rotation quaternion of the transformed Gaussian point; and rot_local represents the original rotation quaternion of the Gaussian point in the local coordinate system of the Gaussian semantic body. @ indicates quaternion multiplication; @ indicates matrix multiplication.

[0012] As a preferred embodiment, the encoding parameters of the hardware encoder are configured as preset=ll low latency mode and gop=1 full I-frame mode, and the tensor is an RGBA CUDA Tensor.

[0013] As a preferred solution, after encoding, the encoded bitstream is sent via a lightweight communication mechanism for the receiving end to decode and use in the target application. The lightweight communication mechanism is a lightweight message queue, which uses ZeroMQ PAIR mode, and the bitstream is an H.264 bitstream.

[0014] As a preferred embodiment, the source of the control data includes client-side generation, server-side physics simulator generation, or a combination of client-side and server-side generation.

[0015] As a preferred embodiment, the hardware encoder includes Intel Quick Sync, AMD VCE, and other encoders that support GPU surface pass-through, and the lightweight communication mechanism can be replaced with WebRTC Data Channel.

[0016] As a preferred embodiment, the Gaussian representation of the 3D Gaussian splash scene supports the SuperSplat compression format, is decompressed to standard Gaussian at runtime, and supports dynamic quantization processing of SH coefficients.

[0017] In addition, the present invention proposes a low-latency 3D Gaussian splash rendering system, which includes a server and a client, and the server and the client are connected via a network.

[0018] The server-side includes a multi-Gaussian semantic volume model management module, a pose update module, a GPU zero-copy encoding module, and a lightweight communication module. The multi-Gaussian semantic volume model management module constructs a multi-volume Gaussian model, partitions the Gaussian semantic volume into subsets, and pre-computes and caches relevant index data. The pose update module receives control data and performs GPU vectorized pose update processing. The GPU zero-copy encoding module implements zero-copy hardware encoding of the rendered output tensor. The lightweight communication module sends the encoded bitstream via a lightweight communication mechanism.

[0019] The client includes a command sending module and a video decoding and display module. The command sending module is used to send vector structured control commands. The video decoding and display module is used to receive the bitstream and decode and display it for use by applications related to the target scene.

[0020] As a preferred embodiment, the server integrates a physics simulator, which is used to generate Gaussian semantic volume pose parameters in the control data.

[0021] As a preferred embodiment, the client integrates a physics simulator for generating Gaussian semantic volume pose parameters in the control data.

[0022] As a preferred embodiment, the server and client each integrate a portion of the physical simulation logic to collaboratively generate the Gaussian semantic volume pose parameters in the control data.

[0023] The aforementioned low-latency 3D Gaussian splash rendering method or system for multi-body dynamic scenes can be applied to AI inference testing, robot simulation, autonomous driving scene generation, and remote 3D Gaussian splash rendering scenes in virtual reality or augmented reality.

[0024] Compared with the prior art, the present invention has at least the following beneficial effects: (1) A multi-Gaussian semantic body decoupling modeling mechanism is adopted. By using a Boolean index mask and a Gaussian semantic body ID mapping array, the 3DGS scene is divided into static and dynamic Gaussian subsets. Only the pose update operation is performed on the dynamic Gaussian points, and the pose calculation is not required for the static part. At the same time, with the help of the GPU's vectorized parallel processing capability, the dynamically compiled kernel function module is called to synchronously complete the position transformation and rotation synthesis, maximizing the computational efficiency of dynamic updates and getting rid of the performance constraints of full scene recalculation.

[0025] (2) An innovative address space mapping mechanism between the rendering buffer and the hardware encoder input buffer is designed to directly submit the tensor of the rendering output to the hardware encoder. The pointer in the GPU memory is passed through the memory-level encoding and decoding interface to avoid redundant data transmission paths. At the same time, low-latency encoding parameters are configured to further ensure the efficient operation of the encoding process and achieve seamless connection between rendering and encoding.

[0026] (3) This solution is compatible with a variety of physical simulation deployment modes, including client-side integrated physical simulator, server-side integrated physical simulator and hybrid simulation deployment. It can be flexibly adjusted according to terminal computing power, intellectual property protection requirements, scene scale, etc. Pose data can be obtained from a variety of mainstream simulation engines. Gaussian representation supports compressed format loading and dynamic quantization. Encoders, communication protocols, etc. can be flexibly replaced or upgraded according to hardware environment and network conditions. At the same time, it supports dynamic adjustment of rendering resolution and can adapt to various scenes that require 3DGS rendering, such as AI inference testing, robot simulation, autonomous driving scene generation, AR / VR remote rendering, etc. Attached Figure Description

[0027] Figure 1 This is an architecture diagram of a low-latency 3D Gaussian splash rendering system for multi-body dynamic scenes in this embodiment. Detailed Implementation

[0028] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0029] This embodiment discloses a low-latency 3D Gaussian splash rendering system for multi-body dynamic scenes, the architecture of which is shown in the figure. Figure 1 As shown, the system consists of a server (cloud rendering / local cluster) and a client (test terminal), which are connected via a network.

[0030] The server-side consists of a multi-Gaussian semantic volume model management module, a pose update module, a GPU zero-copy encoding module, and a lightweight communication module. The multi-Gaussian semantic volume model management module constructs a multi-volume Gaussian model, partitions the Gaussian semantic volume into subsets, and pre-computes and caches relevant index data. The pose update module receives control data and performs GPU vectorized pose update processing. The GPU zero-copy encoding module implements zero-copy hardware encoding of the rendered output tensor. The lightweight communication module sends the encoded bitstream via a lightweight communication mechanism.

[0031] The client consists of a command sending module and a video decoding and display module. The command sending module is used to send vector-based structured control commands. The video decoding and display module is used to receive the bitstream and decode and display it for use by relevant applications in the target scene.

[0032] This system is used to execute specific processes to achieve low-latency 3D Gaussian splash rendering, with the aim of achieving the following goals: Gaussian semantic volume pose update: only update the dynamic Gaussian subset to avoid recalculation across the entire scene; GPU zero-copy video encoding: The rendering result is directly fed into the hardware encoder, eliminating CPU-GPU data copying; Compact Vector Control Instructions: Transmit pose data as a (N_body,7) binary array, compressing uplink bandwidth to ≤2KB / frame; Cross-scenario versatility: Applicable to all scenarios requiring remote 3DGS rendering, such as AI inference testing, robot simulation, and autonomous driving scene generation; End-to-end latency optimization: Achieve ≤35ms latency in remote cloud environments and ≤10ms latency in local area networks, improving AI testing efficiency by 5-20 times.

[0033] The low-latency 3D Gaussian splash rendering process in this embodiment is as follows: Step 1: Construct a multi-body Gaussian model (1) Divide the 3DGS scene into multiple Gaussian semantic volume subsets (such as robot joints and vehicle parts) according to the Gaussian semantic volume constraints of the simulation objects. (2) Pre-computation: Generate the Boolean index mask (dynamic_mask) of the Gaussian semantic body to which each Gaussian point belongs and the Gaussian semantic body ID mapping array (point_to_body_idx), and cache them in the GPU memory.

[0034] The Gaussian semantic volume partitioning follows the following rules: (1) It is divided according to the geometric structure of the simulation object. Each independent moving part constitutes a Gaussian semantic volume. For example, the robot's base, link, joint, and end effector correspond to different Gaussian semantic volumes. (2) The background static environment is an independent Gaussian semantic volume and is marked as background. It does not participate in dynamic pose updates. (3) The range of Gaussian point indices of each Gaussian semantic volume is determined by the loading order of the model file (PLY format) during initialization. The starting index and ending index are calculated by accumulating according to the loading order.

[0035] The generation process of the Boolean index mask `dynamic_mask` and the Gaussian semantic body ID mapping array `point_to_body_idx` is as follows: First, based on the index range (start index to end index) of each Gaussian semantic body, a Boolean array `dynamic_mask` with a length equal to the total number of Gaussian points is initialized. The index positions belonging to dynamic Gaussian semantic bodies (excluding background) are set to `True`, and static Gaussian points are set to `False`. Second, an integer array `point_to_body_idx` with a length equal to the total number of Gaussian points is initialized. For each dynamic Gaussian point, according to the order of its Gaussian semantic body, the corresponding position is set to the ID of that Gaussian semantic body (incrementing from 0); the mapping value of static Gaussian points is set to an invalid index (e.g., -1). Both arrays are cached in GPU memory once after initialization to avoid repeated calculations in each frame.

[0036] Step 2: Receive vector structured control commands Receive each frame of control data (binary format) sent by the client: (1) Virtual camera intrinsic and extrinsic parameters (such as focal length and view matrix); (2) The pose parameters of each Gaussian semantic body are represented by a compact array of shape (N_body,7) (7-dimensional: [x,y,z,qx,qy,qz,qw]).

[0037] Step 3: GPU Vectorized Pose Update (1) Copy the static Gaussian template to the output buffer; (2) Using dynamic_mask and point_to_body_idx, only dynamic Gaussian points and their corresponding Gaussian semantic body poses are extracted; The extraction logic for dynamic Gaussian points is as follows: First, the pre-cached boolean index mask `dynamic_mask` and the Gaussian semantic body ID mapping array `point_to_body_idx` are read from the GPU memory. Then, utilizing the GPU's parallel computing capabilities, subsequent pose update calculations are performed only on Gaussian points marked as `dynamic_mask` as `True` through boolean index operations. Static Gaussian points directly copy the original template data. For each dynamic Gaussian point, its corresponding Gaussian semantic body ID is queried through the `point_to_body_idx` array, and then the position (x, y, z) and pose quaternion (qx, qy, qz, qw) of the corresponding Gaussian semantic body are extracted from the pose parameter array.

[0038] The index conflict handling mechanism includes: during the initialization phase, by strictly dividing the range of the Gaussian semantic body index, ensuring that each Gaussian point belongs to only one Gaussian semantic body, thus avoiding index overlap at the physical storage level; during the runtime phase, if abnormal input is detected (such as pose data containing NaN or infinity values), the pose update of that frame is skipped, and the valid pose data of the previous frame is used to avoid rendering anomalies; if the Gaussian semantic body ID is detected to be out of the valid range, the Gaussian point is treated as a static point and does not participate in dynamic updates.

[0039] (3) Call dynamically compiled kernel function modules to execute in parallel: ① Position transformation: xyz_new = R(body_quat) @ xyz_local + body_pos; ② Rotation composition: rot_new = body_quat⊗rot_local; Step 4: Zero-copy hardware coding (1) The RGBA CUDA Tensor of the rendering output is directly submitted to the hardware encoder; (2) Pass pointers within GPU memory through the memory-level encoding / decoding interface to avoid Device-to-Host copying; (3) Encoding parameter configuration: preset=ll (low latency), gop=1 (full I-frame).

[0040] Step 5: Low-overhead network transmission (1) Send H.264 bitstreams via lightweight message queues (such as ZeroMQ PAIR mode); (2) Client: After decoding, it is used for VLA network input, scene visualization or other purposes.

[0041] The exception handling mechanism includes the following aspects: (1) Pose data validity verification: After receiving the pose parameters, the position component and quaternion component of each Gaussian semantic body are checked for validity. If NaN, infinity or exceeding the preset reasonable range is detected (e.g., the position exceeds the scene boundary ±1000 meters, the quaternion modulus deviates from 1 by more than 0.01), the frame data is discarded and the valid pose of the previous frame is used; (2) GPU memory management: The memory buffer is pre-allocated during the initialization phase. During runtime, Gaussian point data is managed through the memory pool mechanism. If insufficient memory is detected (the available memory is lower than the threshold, such as 500MB), the degradation strategy is automatically triggered, including reducing the rendering resolution (e.g., from 720P to 480P), reducing the number of Gaussian semantic bodies rendered at the same time, or releasing the level of detail (LOD) data of non-critical Gaussian points; (3) Network transmission exception handling: If the bit stream is detected to fail to send or the network latency exceeds the threshold (e.g., 100ms), the connection is automatically reconnected and the key frame data is cached to ensure rendering continuity.

[0042] Compared with existing technologies, the present invention improves performance in multiple dimensions, including dynamic update granularity, rendering-encoding path, control instruction bandwidth, AI testing efficiency, applicable scenarios, and local network performance, as shown in Table 1.

[0043] Table 1 Comparison of this application with existing technologies in various dimensions In summary, the core of this invention lies in the following five points: 1. Multi-Gaussian semantic volume decoupling modeling mechanism: The semantic body-level partitioning of Gaussian scenes is achieved through dynamic_mask and point_to_body_idx, which only updates the dynamic subset and avoids global recomputation.

[0044] 2. GPU Vectorized Pose Update Pipeline: By using advanced indexing and compilation optimization of kernel functions, the pose update latency was reduced from >10ms to 1.3ms (actual test).

[0045] 3. Direct address space mapping mechanism between the rendering buffer and the hardware encoder input buffer: The rendering tensor memory layout is aligned with the hardware encoder input requirements, enabling direct address space mapping and eliminating CPU-GPU data copying (encoding latency of 0.2-0.3ms), which differs from the indirect transmission method of general GPU encoding.

[0046] 4. Compact Vector Control Protocol: (N_body,7) binary array transmits pose, uplink bandwidth compressed to ≤2KB / frame (compared to ≥8KB for JSON).

[0047] 5. Cross-scenario versatility: It is suitable for all scenarios that require remote rendering of 3DGS, such as AI inference testing (VLA network verification), robot simulation, and autonomous driving scene generation.

[0048] The logical ideas behind the methods disclosed in the above embodiments can be implemented, in whole or in part, through software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs.

[0049] When the computer instructions or computer program are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium may be a solid-state drive (SSD).

[0050] Finally, it should be emphasized that although the foregoing embodiments use "the client generating and sending the pose data of each movable part" as a typical scenario, the core of this invention—a low-latency 3D Gaussian splash rendering method based on component-level decoupling and local updates—does not depend on the specific source of the pose data or the deployment location of the physics simulation engine. The following alternative solutions can all achieve the objectives of this invention and should be considered equivalent implementations of this invention, falling within the scope of protection of this invention.

[0051] 1. The physical simulator is located on the server side (the client only sends high-level control commands). In this variant, a complete physical simulation environment (such as Isaac Gym, PyBullet, CARLA, etc.) is used. The client is not responsible for calculating or transmitting specific multi-object poses, but rather sends high-level or semantic control commands (e.g., "robotic arm grasps target A", "robotic arm end effector moves forward 1 cm", "vehicle moves at 2 m / s"). 2Accelerate, fly the drone to coordinates (x, y, z) or directly control the robot's joint angles or torque; After receiving the instruction, the server performs a complete physical simulation in its local simulator, generating the precise pose (position + rotation) of all relevant movable parts in real time. Subsequently, the server invokes the local pose update module of this invention (using dynamic_mask and point_to_body_idx) for efficient rendering.

[0052] Advantages and applicability: This mode is suitable for scenarios where the client is a lightweight terminal (such as a mobile phone or VR headset) or where the core simulation logic needs to be kept in the cloud to protect intellectual property rights.

[0053] 2. The physics simulator is located on the client side (original embodiment). The client integrates complete physical simulation capabilities, enabling it to autonomously calculate the poses of all dynamic objects in the scene; The set camera intrinsic and extrinsic parameters and the calculated pose are sent to the server as compact vector structured data (such as a (N_body,7) binary array); The server acts solely as a rendering and encoding node and does not participate in any physical simulation calculations.

[0054] Advantages and applicability: This mode maximizes the use of client computing power and significantly reduces server load, making it particularly suitable for large-scale parallel AI inference testing (such as simultaneously verifying thousands of VLA network policies).

[0055] 3. Hybrid simulation deployment Some of the system's simulation logic (such as the main robot directly controlled by the user) runs on the client side, while other parts (such as complex environmental interactions and non-player character NPCs) are managed by the simulator on the server side. Pose data is generated separately by the client and the server, and synchronized and merged through a unified protocol before being input into the rendering module.

[0056] Advantages and applicability: This model offers maximum deployment flexibility, achieving the best balance between computing resources, network bandwidth, and interaction latency, and is suitable for complex scenarios involving ultra-large-scale, multi-agent collaborative simulations.

[0057] 4. Other general-purpose technology variations (1) Data source generalization: ① Pose data can come from any simulation engine (such as Isaac Gym, Omniverse, PyBullet), and is not limited to robots; ②Applicable to autonomous driving simulation (vehicle component pose) or VR content generation (virtual character joints).

[0058] (2) Processing flow variants: ① The encoder can be replaced with Intel Quick Sync (VAAPI) or AMD VCE, as long as it supports GPU surface pass-through; ② The communication protocol can be upgraded to WebRTC DataChannel (with FEC), which is suitable for high packet loss networks; ③ Dynamically adjust rendering resolution (e.g., 720P↔480P) to balance image quality and latency according to testing requirements.

[0059] (3) Gaussian representation extension: ① Supports loading compression formats such as SuperSplat, and decompresses to standard Gaussian at runtime; ②The SH coefficient is dynamically quantized to further compress transmitted data.

[0060] All variants rely on the core idea of ​​multi-Gaussian semantic body local update + zero-copy encoding + compact control protocol, and fall within the protection scope of this invention.

[0061] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to any specific implementation. Clearly, many other modifications and variations can be made based on the content of this specification.

Claims

1. A low-latency 3D Gaussian splash rendering method for multi-body dynamic scenes, characterized in that, Includes the following steps: The 3D Gaussian splash scene is divided into multiple Gaussian semantic volume subsets according to the Gaussian semantic volume constraints of the simulation object. The Boolean index mask and Gaussian semantic volume ID mapping array of the Gaussian semantic volume to which each Gaussian point belongs are pre-calculated and cached. Receive control data for each frame, which includes virtual camera intrinsic and extrinsic parameters and pose parameters of each Gaussian semantic volume, wherein the pose parameters are transmitted in the form of a compact binary array of a preset dimension; Copy the static Gaussian template to the output buffer, extract the dynamic Gaussian points and their corresponding Gaussian semantic body poses using the Boolean index mask and Gaussian semantic body ID mapping array, and complete the position transformation and rotation synthesis of the dynamic Gaussian points by executing kernel functions in parallel. The tensor of the rendering output is directly submitted to the hardware encoder, and the pointer in the GPU memory is passed through the memory-level encoding and decoding interface to complete the encoding process.

2. The low-latency 3D Gaussian splash rendering method for multi-body dynamic scenes according to claim 1, characterized in that, Both the Boolean index mask and the Gaussian semantic body ID mapping array are cached in GPU memory.

3. The low-latency 3D Gaussian splash rendering method for multi-body dynamic scenes according to claim 1, characterized in that, The compact binary array of the preset dimension has a shape of (N_body, 7), and the 7-dimensional data correspond to [x, y, z, qx, qy, qz, qw] respectively. The control data has no protocol header and no text serialization processing.

4. The low-latency 3D Gaussian splash rendering method for multi-body dynamic scenes according to claim 1, characterized in that, The dynamically compiled kernel function module is called to execute the kernel function to complete the dynamic transformation and rotation synthesis of Gaussian points; The position transformation satisfies the formula ; The rotational synthesis satisfies the formula ; Where xyz_new represents the position of the transformed Gaussian point in the world coordinate system; body_quat represents the pose quaternion of the Gaussian semantic body; R(*) represents the transformation function from quaternion to rotation matrix; xyz_local represents the original position of the Gaussian point in the local coordinate system of the Gaussian semantic body; body_pos represents the position of the Gaussian semantic body; rot_new represents the rotation quaternion of the transformed Gaussian point; and rot_local represents the original rotation quaternion of the Gaussian point in the local coordinate system of the Gaussian semantic body. @ indicates quaternion multiplication; @ indicates matrix multiplication.

5. The low-latency 3D Gaussian splash rendering method for multi-body dynamic scenes according to claim 1, characterized in that, After the encoding process is completed, it also includes: The encoded bitstream is sent through a lightweight communication mechanism, which is then decoded by the receiving end for use in the target scenario-related applications. The lightweight communication mechanism is a lightweight message queue, which adopts the ZeroMQ PAIR mode, and the bitstream is an H.264 bitstream.

6. The low-latency 3D Gaussian splash rendering method for multi-body dynamic scenes according to claim 1, characterized in that, The sources of the control data include client-side generation, server-side physics simulator generation, or a combination of client-side and server-side generation.

7. The low-latency 3D Gaussian splash rendering method for multi-body dynamic scenes according to claim 1, characterized in that, The Gaussian representation of the 3D Gaussian splash scene supports the SuperSplat compression format, decompresses to standard Gaussian at runtime, and supports dynamic quantization processing of SH coefficients.

8. A low-latency 3D Gaussian splash rendering system, characterized in that, It includes a server and a client, with the server and client connected via a network; The server-side includes: The multi-Gaussian semantic body model management module is used to build multi-body Gaussian models, divide Gaussian semantic body subsets, and pre-compute cached related index data; The pose update module is used to receive control data and perform GPU vectorized pose update processing. The GPU zero-copy encoding module is used to implement zero-copy hardware encoding of rendering output tensors. Lightweight communication module, used to send encoded bitstreams via lightweight communication mechanism; The client includes: The instruction sending module is used to send vector-based structured control instructions; The video decoding and display module is used to receive the bitstream and decode and display it for use in relevant applications in the target scene.

9. The low-latency 3D Gaussian splash rendering system according to claim 8, characterized in that, The server integrates a physics simulator, which is used to generate Gaussian semantic volume pose parameters in the control data. Alternatively, the client may integrate a physics simulator to generate Gaussian semantic volume pose parameters in the control data; Alternatively, the server and client may each integrate a portion of the physical simulation logic to collaboratively generate the Gaussian semantic volume pose parameters in the control data.

10. The application of the low-latency 3D Gaussian splash rendering method for multi-body dynamic scenes as described in any one of claims 1 to 7, characterized in that, include: Applications in AI inference testing; Applications in robot simulation; Applications in autonomous driving scenario generation; Applications in remote 3D Gaussian splash rendering scenarios in virtual reality or augmented reality.

Images