Hybrid rendering method, device and equipment for 3DGS scene and medium

Abstract

The invention provides a hybrid rendering method, device and equipment for 3DGS scenes and a medium, and the method comprises the steps: carrying out the cutting of view cones on a plurality of 3DGS scenes participating in the hybrid rendering, and storing all screened nodes to be rendered in a to-be-rendered node list; splitting each to-be-rendered node in the to-be-rendered node list into data segments with the size of one or more tile blocks according to the data size of the to-be-rendered node list, and sequentially writing the data segments into the idle tile blocks of the virtual texture; on the basis of the page table structural body, traversing all the 3DGS primitives loaded to the virtual texture, and performing global sorting according to the depth value of each primitive to determine a sorting index sequence; and sequentially accessing corresponding primitive parameters in the virtual texture according to the sorting index sequence, and performing primitive-by-primitive rendering in a GPU (Graphics Processing Unit) by adopting a semitransparent mixing mode without writing in a depth buffer area, thereby solving the rendering problem caused by abnormal Gaussian primitive sorting of each scene during multi-scene rendering.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional scene rendering, in particular to a 3DGS scene hybrid rendering method and device, equipment and medium. BACKGROUND

[0002] In recent years, three-dimensional scene reconstruction technology has evolved from traditional mesh modeling based on multi-view geometry and voxel rasterization to implicit representation paradigm represented by neural radiance field (NeRF). Among them, 3D Gaussian splatting (3DGS) as an explicit acceleration alternative to NeRF, by expressing the scene as a large number of three-dimensional Gaussian primitives (i.e. ellipsoids) with covariance matrix, while maintaining extremely low memory overhead, it realizes high-fidelity rendering with millisecond-level frame rate, and has become the mainstream technology path of three-dimensional reconstruction and real-time rendering in the industry. However, when multiple independently constructed 3DGS scenes need to be hybrid rendered under the same view, if the single-scene independent sorting strategy is still used, the internal primitive sorting results of each scene cannot be coordinated, and depth order disorder is easily generated in the scene overlap area (such as door frame penetrating wall, furniture embedded in floor), which shows serious rendering abnormalities such as visual "flicker", "penetration" and "level reversal". The industry has not yet formed a mature multi-3DGS scene hybrid solution. The closest practice is the coarse-grained overlay strategy based on scene bounding box: each 3DGS scene is regarded as a whole, and the scene-level rendering order is determined according to the distance from the center of the bounding box to the camera (from near to far). Although this method can alleviate the level error under some viewing angles, it has the following essential defects: ①viewing angle sensitivity: only relying on the depth of the bounding box center, ignoring the internal geometric distribution of the scene, in the oblique viewing angle or non-convex scene, the bounding box depth cannot represent the true occlusion relationship, leading to the failure of the ordering of the local penetration area; ②non-divisible: the scene is forced to be regarded as an atomic unit, and it is completely impossible to handle the geometric penetration between scenes (such as a pipe penetrating through two adjacent rooms of 3DGS model), at this time the bounding box strategy is completely disabled; ③unordered primitive overlay: even if the scene-level order is correct, the Gaussian primitives in each scene are still independently sorted, and the depth comparison between the primitives across scenes is missing, and channel overlay conflicts will still occur when hybrid rendering is performed. Therefore, it is urgent to develop a hybrid rendering architecture that breaks through the scene boundary, uniformly manages all Gaussian primitives, supports fine-grained depth sorting and dynamic cache scheduling, and fundamentally solves the correctness and robustness problems of multi-3DGS scene fusion. SUMMARY

[0003] Therefore, the present application aims to provide a 3DGS scene hybrid rendering method, device, equipment and medium, which uses a virtual texture to cache all to-be-rendered nodes, thereby converting the 3DGS nodes of multiple scenes into scene-independent data blocks, and then performing global unified sorting to solve the rendering problem caused by abnormal sorting of Gaussian primitives of each scene during multi-scene rendering.

[0004] The present application embodiment provides a 3DGS scene hybrid rendering method, which comprises: Performing a view frustum culling on multiple 3DGS scenes participating in hybrid rendering, and storing all filtered to-be-rendered nodes to a to-be-rendered node list; wherein each to-be-rendered node is only a scene-attribute-free data block carrying a 3DGS primitive; Splitting each to-be-rendered node in the to-be-rendered node list into one or more data segments of a tile block size according to its data amount size, and sequentially writing each data segment into an idle tile block of a virtual texture; Based on a pre-constructed page table structure, traversing all 3DGS primitives loaded into the virtual texture, and performing global sorting according to the depth value of each 3DGS primitive to determine a sorting index sequence; According to the sorting index sequence, sequentially accessing the corresponding 3DGS primitive parameters in the virtual texture, and performing a per-primitive rendering in a GPU using a semi-transparent hybrid mode without writing into a depth buffer.

[0005] In a possible implementation, the global sorting according to the depth value of each 3DGS primitive to determine a sorting index sequence comprises: Obtaining all 3DGS primitives loaded into the virtual texture; Converting the three-dimensional center coordinates of each 3DGS primitive through a model-view projection matrix to determine the four-dimensional homogeneous coordinates of each 3DGS primitive in a homogeneous clipping space; Performing a perspective division on the four-dimensional homogeneous coordinates to determine a normalized coordinate, and determining the depth value of each 3DGS primitive based on the Z component value in the normalized coordinate; Performing a parallel sorting operation with the depth values of all 3DGS primitives as keys to generate a one-dimensional integer index array, and taking the one-dimensional integer index array as a sorting index sequence; wherein the numerical value of the first element in the one-dimensional integer index array represents the original linear index number of the 3DGS primitive located at the first position in the sorted sequence in the virtual texture. i i

[0006] ​​In one possible implementation, the sequentially accessing the corresponding 3DGS primitive parameters in the virtual texture according to the sequence of the sorting index sequence, and performing per-primitive rendering in the GPU using a semi-transparent blending mode without writing into the depth buffer, comprises: reading the 3DGS primitive parameters indicated by each element in the sequence of the sorting index sequence in turn, wherein each 3DGS primitive parameter at least includes a three-dimensional center coordinate, an ellipsoid scaling factor, an ellipsoid rotation factor, an opacity value, and a color RGB value; For each 3DGS primitive, determining a projection ellipse region and a pixel coverage weight of the 3DGS primitive in the current frame based on the three-dimensional center coordinate and the covariance matrix of the 3DGS primitive; At each target pixel covered by the projection ellipse region, accumulating the color RGB value and the opacity value of the 3DGS primitive, and updating the frame buffer color value of the target pixel.

[0007] In one possible implementation, after splitting each to-be-rendered node in the to-be-rendered node list into one or more tile block size data segments according to the data size of the to-be-rendered node, for the newly added to-be-rendered node data of the current frame, the hybrid rendering method further comprises: determining whether there is a free tile in the virtual texture; if yes, writing the data of the newly added to-be-rendered node of the current frame into the free tile block; if no, identifying the node data that does not participate in rendering of the current frame, releasing the virtual texture address interval corresponding to the node data to determine a free tile block, and writing the data of the newly added to-be-rendered node into the free tile block.

[0008] In one possible implementation, after identifying the node data that does not participate in rendering of the current frame, releasing the virtual texture address interval corresponding to the node data to determine a free tile block, and writing the data of the newly added to-be-rendered node into the free tile block, the hybrid rendering method further comprises: deleting the node data that does not participate in rendering of the current frame in the page table structure, and adding the unique identifier of the newly added to-be-rendered node, the starting offset of the to-be-rendered node in the virtual texture, and the data byte length of the to-be-rendered node into the index item.

[0009] In one possible implementation, the page table structure is a structure array, and the page table structure comprises: Each structure element contains two-dimensional coordinates of the tile block in the virtual texture, the number of valid data bytes in the tile block, and a Boolean flag indicating whether the tile block is visible in the current frame; wherein, the page table structure is synchronously maintained in a read-only copy in the GPU memory, for real-time indexing by the GPU shader program.

[0010] In one possible implementation, before performing per-primitive rendering in the GPU using a semi-transparent blending mode without writing to the depth buffer, the blending rendering method further comprises: invoking a GPU driver interface to query the current available memory capacity; if the current available memory capacity is lower than a preset threshold, triggering the CPU end to pre-release the tile block with the lowest historical access frequency of the page table structure and invisible in the current frame, to generate a pre-release candidate set; wherein, the pre-release candidate set is preferentially selected in the next frame.

[0011] The embodiments of the present application also provide a blending rendering device for a 3DGS scene, the blending rendering device comprising: a screening module configured to perform a view frustum culling on a plurality of 3DGS scenes participating in blending rendering, and store all to-be-rendered nodes screened out to a to-be-rendered node list; wherein, each to-be-rendered node is only a scene attribute data block carrying a 3DGS primitive; a division module configured to split each to-be-rendered node in the to-be-rendered node list into one or more data segments of tile block size according to the data amount of the to-be-rendered node, and write each data segment into an idle tile block of the virtual texture in sequence; a global sorting module configured to traverse all 3DGS primitives loaded into the virtual texture based on a pre-constructed page table structure, and determine a sorting index sequence according to the depth values of each 3DGS primitive; a rendering module configured to sequentially access corresponding 3DGS primitive parameters in the virtual texture according to the sorting index sequence, and perform per-primitive rendering in the GPU using a semi-transparent blending mode without writing to the depth buffer.

[0012] The embodiments of the present application also provide an electronic device comprising a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory communicating through the bus when the electronic device is running, and the machine-readable instructions being executed by the processor to perform the steps of the blending rendering method for a 3DGS scene as described above.

[0013] The embodiments of the present application also provide a computer-readable storage medium storing a computer program, the computer program being executed by a processor to perform the steps of the blending rendering method for a 3DGS scene as described above.

[0014] The embodiment of the present application provides a 3DGS scene hybrid rendering method, device and equipment and medium, the hybrid rendering method comprises the following steps: performing a view frustum culling on a plurality of 3DGS scenes participating in hybrid rendering, and storing all screened to-be-rendered nodes to a to-be-rendered node list; wherein each to-be-rendered node is only a scene attribute data block carrying a 3DGS primitive; splitting each to-be-rendered node in the to-be-rendered node list into one or more tile block size data segments according to the data size of the to-be-rendered node, and sequentially writing each data segment into an idle tile block of a virtual texture; traversing all 3DGS primitives loaded into the virtual texture based on a pre-constructed page table structure, determining a sorting index sequence according to the depth value of each 3DGS primitive for global sorting; sequentially accessing corresponding 3DGS primitive parameters in the virtual texture according to the sorting index sequence, and performing per-primitive rendering in a GPU by using a semi-transparent hybrid mode without writing into a depth buffer. All to-be-rendered nodes are cached by using the virtual texture, so that the 3DGS nodes of multiple scenes are converted into scene-independent data blocks, and then global uniform sorting is performed, so that the rendering problem caused by abnormal sorting of Gaussian primitives of each scene during multi-scene rendering is solved.

[0015] In order to make the above objectives, characteristics and advantages of the present application more apparent, the following preferred embodiments are specifically described below, and the accompanying drawings are referred to for detailed description. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation to the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0017] Figure 1 A flowchart of a 3DGS scene hybrid rendering method provided by the embodiment of the present application; Figure 2 One of the structural schematic diagrams of a 3DGS scene hybrid rendering device provided by the embodiment of the present application; Figure 3 The second structural schematic diagram of a 3DGS scene hybrid rendering device provided by the embodiment of the present application; Figure 4 The structural schematic diagram of an electronic device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0018] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings of the embodiments of the present application to make a clear and complete description of the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, every other embodiment obtained by a person skilled in the art without creative work belongs to the scope of protection of the present application.

[0019] Firstly, the application scenarios applicable to the present application are introduced. The present application can be applied to the technical field of three-dimensional scene rendering.

[0020] Based on this, the embodiments of the present application provide a hybrid rendering method of 3DGS scene, which utilizes virtual texture to cache all to-be-rendered nodes, so as to convert the 3DGS nodes of multiple scenes into scene-unrelated data blocks, and then perform global unified sorting, thereby solving the rendering problem caused by abnormal sorting of Gaussian primitives of each scene during multiple scene rendering.

[0021] Please refer to Figure 1 , Figure 1 The flowchart of the hybrid rendering method of 3DGS scene provided by the embodiments of the present application is shown in FIG. 1. Figure 1 As shown in FIG. 1, the hybrid rendering method provided by the embodiments of the present application includes the following steps. S101: performing view frustum culling on multiple 3DGS scenes participating in hybrid rendering, and storing all to-be-rendered nodes screened out into a to-be-rendered node list; wherein each to-be-rendered node is only a scene-attribute-free data block carrying a 3DGS primitive.

[0022] Here, in the rendering initialization stage, multiple 3D Gaussian splatting (3DGS) scenes that have been constructed are received, each scene being composed of a plurality of nodes in three-dimensional space; the node is a spatial division unit of the scene, and at least one 3DGS primitive is encapsulated in each node, the 3DGS primitive being structured attribute data having a three-dimensional center coordinate, an ellipsoid scaling factor, an ellipsoid rotation factor, an opacity value, a color RGB value, etc.

[0023] Wherein, uniform frustum culling is performed on all 3DGS scenes: based on the current camera parameters (including field of view angle, near / far clipping plane distance, viewing direction and up vector), a standard right-handed system frustum six-plane equation is constructed; the center coordinates and radius of the bounding sphere of each node are substituted into the six-plane equation for point-to-plane distance discrimination; if the bounding sphere has no positive intersection with any clipping plane (i.e. the bounding sphere is completely outside the plane), it is determined that the node is invisible and is discarded.

[0024] It should be noted that each item in the list Q only contains the geometry and attribute data pointer of the node itself, does not store, carry or encode any meta information identifying the original 3DGS scene to which the node belongs; in other words, in the data structure definition of the list Q, there is no field to record the scene ID, scene name, scene creation timestamp or any context identifier traceable to the source scene. All nodes are treated equally in the list Q, and their data essence is abstracted as a pure data block (DataBlock) carrying only 3DGS primitives, the content structure of the data block conforms to the preset binary memory layout specification, including but not limited to: node starting address offset (relative to its memory buffer base address); total number of 3DGS primitives contained in the node; three-dimensional center coordinates, ellipsoid scaling factor, ellipsoid rotation factor, opacity value, color RGB value and other attribute data of each primitive, node-level LOD level index (for subsequent tile adaptation).

[0025] In this application, the list of nodes to be rendered realizes semantic decoupling of multi-source 3DGS scenes: the topological attribution relationship output by the upstream scene reconstruction module is actively stripped at this link, and the downstream rendering pipeline only needs to perform depth calculation and ordering based on the spatial attributes of the internal primitives of the node, and no longer depends on or is aware of the scene source. This design fundamentally eliminates the ordering ambiguity caused by the lack of cross-scene occlusion relationship modeling, providing a data isomorphic basis for subsequent global unified depth ordering.

[0026] S102: split each node to be rendered in the list of nodes to be rendered into one or more data segments of tile block size according to its data size, and write each data segment into the idle tile block of the virtual texture in turn.

[0027] In this step, each node to be rendered in the list of nodes to be rendered is split into one or more data segments of tile block size according to its data size, and each data segment is written into the idle tile block of the virtual texture in turn.

[0028] It should be noted that a virtual texture is maintained in the GPU memory, with a total capacity of a preset size. This virtual texture is logically divided into N consecutive, equal-length, fixed-size tiles, each with a unique linear index number. i It is marked in real time with its occupied / idle status by a bitmap structure.

[0029] Here, the node data volume is calculated as follows: The node's metadata field is read to obtain the actual number of bytes of video memory required. This number of bytes is the total serialized length of all 3DGS primitives and their attribute data (3D center coordinates, ellipsoid scaling factor, ellipsoid rotation factor, opacity value, and RGB color value) contained in the node, after padding to an integer multiple of the tile size. The required number of tiles is determined, and consecutive free tile fragments are searched: The virtual texture bitmap is traversed to locate the first consecutive free tile interval whose length is greater than or equal to the required number of bytes of video memory. If such an interval exists, its starting tile index is recorded. If multiple consecutive free tiles do not exist, fragmentation is performed, and the search is retried after fragmentation. The node's serialized data is written to the tile block, and the resource status is updated.

[0030] S103: Based on the pre-built page table structure, traverse all 3DGS primitives that have been loaded into the virtual texture, and determine the sorting index sequence by globally sorting according to the depth value of each 3DGS primitive.

[0031] In this step, a pre-built page table structure is used to traverse all 3DGS primitives that have been loaded into the virtual texture, and a sorting index sequence is determined by globally sorting based on the depth value of each 3DGS primitive.

[0032] It should be noted that a page table structure is created and maintained to record the offset and length in bytes of each node data to be rendered in the virtual texture.

[0033] In one possible implementation, the step of determining the sorting index sequence by globally sorting based on the depth value of each 3DGS primitive includes: A: Obtain all 3DGS primitives that have been loaded into the virtual texture; B: The three-dimensional center coordinates of each 3DGS primitive are transformed by the model-view projection matrix to determine the four-dimensional homogeneous coordinates of each 3DGS primitive in the homogeneous clipping space.

[0034] Here, the three-dimensional center coordinates of each 3DGS primitive are transformed by the model-view projection matrix to determine the four-dimensional homogeneous coordinates of each 3DGS primitive in the homogeneous clipping space.

[0035] C: Perform perspective division on the four-dimensional homogeneous coordinates to determine the normalized coordinates, and determine the depth value of each 3DGS primitive based on the Z component value in the normalized coordinates.

[0036] Here, the depth value of each 3DGS primitive is determined based on the Z component value in the normalized coordinates.

[0037] D: Perform a parallel sorting operation using the depth values ​​of all 3DGS primitives as keys to generate a one-dimensional integer index array, and use the one-dimensional integer index array as the sorting index sequence; wherein, the first value in the one-dimensional integer index array... i The value of the nth element represents its position in the sorted sequence. i The original linear index number of the 3DGS primitive in the virtual texture.

[0038] Here, a one-dimensional integer array SortedIndex is maintained, the length of which is equal to the total number K of all valid 3D Gaussian primitives in the virtual texture; using all K 3D Gaussian primitives as the sort key, the parallel radix sort algorithm provided by the GPU is called to sort the index sequence [0, 1, ..., K]. 1] Perform an ascending reordering to sort the primitives of the resulting one-dimensional integer array. Here, the array stores a sequence of logical indices of primitives arranged from closest to furthest (smallest to largest) in depth.

[0039] S104: Based on the sorted index sequence, sequentially access the corresponding 3DGS primitive parameters in the virtual texture, and perform primitive-by-primitive rendering in the GPU using a semi-transparent blending method that does not write to the depth buffer.

[0040] In this step, the corresponding 3DGS primitive parameters in the virtual texture are accessed sequentially according to the sorted index sequence, and primitive-by-primitive rendering is performed in the GPU using a semi-transparent blending method that does not write to the depth buffer.

[0041] In one possible implementation, the step of sequentially accessing the corresponding 3DGS primitive parameters in the virtual texture according to the sorted index sequence, and performing per-primitive rendering in the GPU using a semi-transparent blending method that does not write to the depth buffer, includes: (1): Read the 3DGS primitive parameters indicated by each element in the sorted index sequence according to the order of the elements; wherein each 3DGS primitive parameter includes at least: three-dimensional center coordinates, ellipsoid scaling factor, ellipsoid rotation factor, opacity value, and color RGB.

[0042] (2): For each of the 3DGS primitives, based on the three-dimensional center coordinates and covariance matrix of the 3DGS primitive, the projection ellipse region and pixel coverage weight of the 3DGS primitive in the current frame are determined.

[0043] Here, for each 3DGS primitive, the projection ellipse region and pixel coverage weight of the 3DGS primitive in the current frame are determined based on the three-dimensional center coordinates and covariance matrix of the 3DGS primitive.

[0044] It should be noted that the covariance matrix is ​​determined by the ellipsoid scaling factor, ellipsoid rotation factor, and rendering camera parameters.

[0045] (3): At each target pixel covered by the projected ellipse region, the RGB color value and opacity value of the 3DGS primitive are accumulated, and the frame buffer color value of the target pixel is updated.

[0046] Here, an atomic accumulation operation is performed on each covered pixel: read the current pixel value in the frame buffer, calculate the color increment contributed by that primitive, and update the frame buffer.

[0047] In one possible implementation, the hybrid rendering method further includes splitting each node in the list of nodes to be rendered into one or more data segments the size of a tile, based on its data size. For newly added node data in the current frame, the hybrid rendering method also includes: a: Determine whether there are any empty tiles in the virtual texture.

[0048] b: If so, then write the data of the newly added node to be rendered in the current frame into the free tile block.

[0049] Here, if there is enough space in the virtual texture to place the node data, then the node data is placed directly.

[0050] c: If not, identify the node data that is not involved in rendering in the current frame, release the virtual texture address range corresponding to the node data to determine the free tile block, and write the data of the newly added node to be rendered into the free tile block.

[0051] Here, if not, the page table structure is traversed to identify node data that is not involved in rendering in the current frame, the virtual texture address range corresponding to the node data is marked as space to be released, and the virtual texture is released. The newly added node to be rendered is written into the continuous free tile block obtained after the release.

[0052] In one possible implementation, a virtual texture fragmentation operation can also be performed during the data processing and partitioning stage, including: The spaces to be released are merged into continuous free intervals, and the starting offsets of the remaining valid node entries in the page table structure are updated to maintain the logical continuity of the node data in the virtual texture.

[0053] Here, the virtual texture fragmentation operation process is as follows: (a) Identify nodes to be released and free tiles: Traverse all elements in the page table structure Index, filter out the index entries of all node entries that are not involved in rendering in the current frame, mark each such index entry as a node to be released, and record the tile interval it occupies. Merge the tile intervals of all nodes to be released to generate a set of free tiles, the elements of which are non-overlapping continuous tile intervals; (b) Construct continuous free intervals: Sort all intervals in the free tile set in ascending order by left endpoint, traverse the sorted interval sequence, and perform interval merging: If the current interval [Lk, Rk] and the next interval [Lk+1, Rk+1] satisfy Rk+1≥Lk+1, then merge them into a new interval [Lk, max(Rk, Rk+1)]; repeat until there are no more to merge; finally, a set of non-overlapping and separate maximum continuous free tile intervals is obtained. (c) Relocate valid node data: Initialize the target and write the starting tile index; traverse all valid index entries with `is_valid==1` in the Index, according to the original physical location of the node in the virtual texture; calculate the original occupied tile interval for each valid node. If the first free interval is sufficient to accommodate the node, copy all data segments of the node from the original address to the new address; otherwise, try subsequent free intervals in turn until the first interval that meets the capacity requirement is found, and perform the same copy and update operation; if there is still not enough space after traversing all free intervals, report an error and terminate the reorganization process. (d) Update bitmap and index status: clear the corresponding bits of all tiles to be released in the bitmap; set the corresponding position of the newly allocated tile interval in the bitmap to "occupied"; ensure that the offset values ​​of all `is_valid==1` Index[i] have been updated to the new physical address, and that the data segments of any two valid nodes have no address overlap or gap insertion in the virtual texture, and that the data segments of any adjacent valid nodes are logically continuous in the buffer.

[0054] In this application, through the aforementioned virtual texture fragmentation operation, the physical memory layout of the virtual texture is reconstructed into a compact form of a single large block of continuous valid data area + a single continuous free area, completely eliminating the small tile holes caused by random addition or deletion of nodes; at the same time, the starting offset field of all valid nodes in the page table structure is synchronously corrected to ensure that it always accurately points to the unique, continuous, and non-overlapping physical address of the node data in the current buffer, providing a stable and predictable memory access basis for depth sorting and rendering of subsequent frames.

[0055] In one possible implementation, after identifying node data that is not involved in rendering in the current frame, releasing the virtual texture address range corresponding to the node data to determine free tile blocks, and writing the data of the newly added node to be rendered into the free tile blocks, the hybrid rendering method further includes: Delete the node data that is not involved in rendering in the current frame from the page table structure, and add the unique identifier of the newly added node to be rendered, its starting offset in the virtual texture, and its data byte length to the index entry.

[0056] In one possible implementation, the page table structure is an array of structures, each of which includes: the two-dimensional coordinates of the tile in the virtual texture, the number of valid data bytes within the tile, and a boolean flag indicating whether the tile is visible in the current frame; wherein, a read-only copy of the page table structure is maintained synchronously in the GPU memory for real-time indexing by the GPU shader program.

[0057] In one possible implementation, prior to per-primal rendering using a semi-transparent blending method that does not write to the depth buffer in the GPU, the blending rendering method further includes: i: Call the GPU driver interface to query the current available video memory.

[0058] Here, the current available memory is calculated by calling the standard interface provided by the underlying graphics driver (for example, calling the `cudaMemGetInfo()` function in the CUDA ecosystem to obtain the current total GPU memory capacity (`total_memory`) and the allocated memory size (`used_memory`).

[0059] ii: If the current available video memory is lower than a preset threshold, the CPU is triggered to pre-release the tile blocks with the lowest historical access frequency in the page table structure and which are not visible in the current frame, generating a pre-release candidate set; wherein, the pre-release candidate set is preferentially selected in the next frame.

[0060] Here, the preset threshold is an empirical safety level, preferably set to `512MB` (which can be dynamically adjusted according to the hardware platform configuration, with a suggested range of `128MB~1024MB`). This value ensures that there is still enough space in subsequent frames to accommodate newly uploaded visible tiles and temporary rendering buffers.

[0061] If the current available video memory is less than a preset threshold, the GPU video memory is determined to be under pressure, and the CPU-side tile pre-release process is initiated. This process does not directly release tiles that are participating in the current frame sorting or rendering, but instead filters inactive tile blocks in the historical cache that are not visible in the current frame.

[0062] Here, a pre-release candidate set is constructed: The CPU maintains a page table structure in which each entry records the tile's logical address in the virtual texture (`virtual_address`), physical memory address (`gpu_address`), and data size (`tile_size`). It also continuously counts and updates the historical access frequency counter (`access_count`) and the timestamp of the last access since the tile was loaded (`last_access_timestamp`). When pre-release is triggered, all entries in the page table are traversed, and tiles that meet all of the following conditions are selected as the pre-release candidate set (PCS): (a) The node to which the corresponding tile belongs has not been selected into the rendering list Q in the view frustum clipping result of the current frame (i.e., `is_visible_in_current_frame==false`); (b) The historical access frequency `access_count` of the tile is the minimum value among all non-visible tiles in the page table (if there are multiple tiles with the same frequency, the earliest `last_access_timestamp` is selected first); (c) The tile is not currently locked by any asynchronous DMA transfer task or rendering pipeline (verified by the atomic flag `is_locked`). All tiles that meet the above conditions constitute the pre-release candidate set, which has a minimum of 1 and an upper limit of no more than 30% of the total number of non-visible tiles in the page table, to prevent over-release from causing frequent reloads in the next frame.

[0063] It should be noted that the pre-release candidate set does not immediately perform memory reclamation operations, but rather marks them as "priority reuse objects for the next frame." That is, in the tile release phase when the virtual texture space is insufficient in the next frame, the system prioritizes selecting tiles from the pre-release candidate set to perform memory release and virtual texture space return, rather than randomly or by scanning the entire page table according to an LRU strategy. This design significantly reduces the tile replacement latency caused by insufficient memory in the next frame, improving frame rate stability and cross-frame consistency.

[0064] This application achieves coordinated control of memory resources and the lifecycle of virtual texture tiles by converting GPU memory pressure signals into CPU-executable cache eviction policies. Compared to traditional schemes that rely solely on immediate release within a frame, the pre-release candidate set mechanism identifies low-value caches one frame cycle in advance, ensuring the integrity of the current frame's rendering while reserving deterministic memory bandwidth for the next frame.

[0065] This application provides a hybrid rendering method for 3DGS scenes. The hybrid rendering method includes: performing frustum clipping on multiple 3DGS scenes participating in hybrid rendering, and storing all selected nodes to be rendered in a list of nodes to be rendered; wherein each node to be rendered is only a data block without scene attributes carrying 3DGS primitives; splitting each node to be rendered in the list of nodes to be rendered into one or more tile-sized data segments according to its data size, and writing each data segment sequentially into the free tile of the virtual texture; based on a pre-built page table structure, traversing all 3DGS primitives loaded into the virtual texture, and globally sorting according to the depth value of each 3DGS primitive to determine a sorting index sequence; according to the sorting index sequence, sequentially accessing the corresponding 3DGS primitive parameters in the virtual texture, and performing primitive-by-primitive rendering in the GPU using a semi-transparent blending method without writing to the depth buffer. By using the virtual texture to cache all nodes to be rendered, the 3DGS nodes of multiple scenes are transformed into scene-independent data blocks, and then globally unified sorting is performed, solving the rendering problem caused by abnormal sorting of Gaussian primitives in multiple scenes during rendering.

[0066] Please see Figure 2 , Figure 3 , Figure 2 One of the structural schematic diagrams of a hybrid rendering device for a 3DGS scene provided in an embodiment of this application; Figure 3 This is a second schematic diagram of the structure of a hybrid rendering device for a 3DGS scene provided in an embodiment of this application. Figure 2 As shown, the hybrid rendering apparatus 200 for the 3DGS scene includes: The filtering module 210 is used to perform view frustum clipping on multiple 3DGS scenes participating in hybrid rendering and store all the nodes to be rendered in the list of nodes to be rendered; wherein, each node to be rendered is only used as a data block without scene attributes to carry 3DGS primitives; The partitioning module 220 is used to split each node in the list of nodes to be rendered into one or more data segments of the size of a tile according to its data size, and write each data segment into the free tile of the virtual texture in sequence; The global sorting module 230 is used to traverse all 3DGS primitives loaded into the virtual texture based on a pre-built page table structure, and to determine the sorting index sequence by performing global sorting based on the depth value of each 3DGS primitive. The rendering module 240 is used to sequentially access the corresponding 3DGS primitive parameters in the virtual texture according to the sorted index sequence, and perform primitive-by-primitive rendering in the GPU using a semi-transparent blending method that does not write to the depth buffer.

[0067] Furthermore, the global sorting module 230 is used to determine the sorting index sequence by performing global sorting based on the depth value of each 3DGS primitive: Retrieve all 3DGS primitives that have been loaded into the virtual texture; The three-dimensional center coordinates of each 3DGS primitive are transformed by the model-view projection matrix to determine the four-dimensional homogeneous coordinates of each 3DGS primitive in the homogeneous clipping space. A perspective division is performed on the four-dimensional homogeneous coordinates to determine the normalized coordinates. Based on the Z component values ​​in the normalized coordinates, the depth value of each 3DGS primitive is determined. Parallel sorting is performed using the depth values ​​of all 3DGS primitives as keys to generate a one-dimensional integer index array, which is then used as the sorting index sequence; wherein, the first integer in the one-dimensional integer index array... i The value of the nth element represents its position in the sorted sequence. i The original linear index number of the 3DGS primitive in the virtual texture.

[0068] Furthermore, the rendering module 240 is used to sequentially access the corresponding 3DGS primitive parameters in the virtual texture according to the sorted index sequence, and perform primitive-by-primitive rendering in the GPU using a semi-transparent blending method that does not write to the depth buffer. Read the 3DGS primitive parameters indicated by each element in the sorted index sequence according to the order of the elements; each 3DGS primitive parameter includes at least: 3D center coordinates, ellipsoid scaling factor, ellipsoid rotation factor, opacity value, and color RGB value. For each 3DGS primitive, based on the three-dimensional center coordinates and covariance matrix of the 3DGS primitive, the projection ellipse region and pixel coverage weight of the 3DGS primitive in the current frame are determined. At each target pixel covered by the projected ellipse region, the RGB color value and opacity value of the 3DGS primitive are accumulated, and the frame buffer color value of the target pixel is updated.

[0069] Furthermore, the partitioning module 220 is also used for: Determine whether there are any empty tiles in the virtual texture; If so, the data of the newly added node to be rendered in the current frame is written into the free tile block; If not, identify the node data that is not involved in rendering in the current frame, release the virtual texture address range corresponding to the node data to determine the free tile block, and write the data of the newly added node to be rendered into the free tile block.

[0070] Furthermore, such as Figure 3As shown, the 3DGS scene hybrid rendering apparatus 200 also includes an update module 250, which is used for: Delete the node data that is not involved in rendering in the current frame from the page table structure, and add the unique identifier of the newly added node to be rendered, its starting offset in the virtual texture, and its data byte length to the index entry.

[0071] Furthermore, the update module 250 is also used for: Call the GPU driver interface to query the current available video memory. If the available video memory is lower than a preset threshold, the CPU is triggered to pre-release the tile blocks that have the lowest historical access frequency in the page table structure and are not visible in the current frame, generating a pre-release candidate set; wherein, the pre-release candidate set is preferentially selected in the next frame.

[0072] This application provides a hybrid rendering apparatus for 3DGS scenes, comprising: a filtering module for performing frustum clipping on multiple 3DGS scenes participating in hybrid rendering, and storing all filtered nodes to be rendered in a node list; wherein each node to be rendered is only a data block without scene attributes carrying 3DGS primitives; a partitioning module for splitting each node to be rendered in the node list into one or more tile-sized data segments according to its data size, and writing each data segment sequentially into the free tile of the virtual texture; a global sorting module for traversing all 3DGS primitives loaded into the virtual texture based on a pre-built page table structure, and determining a sorting index sequence by globally sorting according to the depth value of each 3DGS primitive; and a rendering module for sequentially accessing the corresponding 3DGS primitive parameters in the virtual texture according to the sorting index sequence, and performing primitive-by-primitive rendering in the GPU using a semi-transparent blending method without writing to the depth buffer. By using virtual textures to cache all nodes to be rendered, the 3DGS nodes of multiple scenes are transformed into scene-independent data blocks, and then globally sorted to solve the rendering problem caused by abnormal sorting of Gaussian nodes in each scene during multi-scene rendering.

[0073] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0074] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 1 The steps of the hybrid rendering method for the 3DGS scene in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0075] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 1 The steps of the hybrid rendering method for the 3DGS scene in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0076] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0078] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0080] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0081] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A hybrid rendering method for 3DGS scenes, characterized in that, The hybrid rendering method includes: View frustum clipping is performed on multiple 3DGS scenes participating in hybrid rendering, and all nodes to be rendered are filtered out and stored in the node to be rendered list; each node to be rendered is only a data block without scene attributes that carries 3DGS primitives; Each node in the list of nodes to be rendered is split into one or more data segments the size of a tile according to its data size, and each data segment is written into the free tile of the virtual texture in sequence; Based on the pre-built page table structure, all 3DGS primitives loaded into the virtual texture are traversed, and the sorting index sequence is determined by global sorting according to the depth value of each 3DGS primitive. Based on the sorted index sequence, the corresponding 3DGS primitive parameters in the virtual texture are accessed sequentially, and per-primitive rendering is performed in the GPU using a semi-transparent blending method that does not write to the depth buffer.

2. The hybrid rendering method according to claim 1, characterized in that, The step of determining the sorted index sequence by globally sorting based on the depth value of each 3DGS primitive includes: Retrieve all 3DGS primitives that have been loaded into the virtual texture; The three-dimensional center coordinates of each 3DGS primitive are transformed by the model-view projection matrix to determine the four-dimensional homogeneous coordinates of each 3DGS primitive in the homogeneous clipping space. A perspective division is performed on the four-dimensional homogeneous coordinates to determine the normalized coordinates. Based on the Z component values ​​in the normalized coordinates, the depth value of each 3DGS primitive is determined. Parallel sorting is performed using the depth values ​​of all 3DGS primitives as keys to generate a one-dimensional integer index array, which is then used as the sorting index sequence; wherein, the first integer in the one-dimensional integer index array... i The value of the nth element represents its position in the sorted sequence. i The original linear index number of the 3DGS primitive in the virtual texture.

3. The hybrid rendering method according to claim 1, characterized in that, The step of sequentially accessing the corresponding 3DGS primitive parameters in the virtual texture according to the sorted index sequence, and performing per-primitive rendering in the GPU using a semi-transparent blending method that does not write to the depth buffer, includes: Read the 3DGS primitive parameters indicated by each element in the sorted index sequence according to the order of the elements; each 3DGS primitive parameter includes at least: 3D center coordinates, ellipsoid scaling factor, ellipsoid rotation factor, opacity value, and color RGB value. For each 3DGS primitive, based on the three-dimensional center coordinates and covariance matrix of the 3DGS primitive, the projection ellipse region and pixel coverage weight of the 3DGS primitive in the current frame are determined. At each target pixel covered by the projected ellipse region, the RGB color value and opacity value of the 3DGS primitive are accumulated, and the frame buffer color value of the target pixel is updated.

4. The hybrid rendering method according to claim 1, characterized in that, The hybrid rendering method further includes splitting each node in the list of nodes to be rendered into one or more data segments the size of a tile, based on its data volume. For newly added node data in the current frame, the hybrid rendering method also includes: Determine whether there are any empty tiles in the virtual texture; If so, the data of the newly added node to be rendered in the current frame is written into the free tile block; If not, identify the node data that is not involved in rendering in the current frame, release the virtual texture address range corresponding to the node data to determine the free tile block, and write the data of the newly added node to be rendered into the free tile block.

5. The hybrid rendering method according to claim 4, characterized in that, After identifying node data that is not involved in rendering in the current frame, releasing the virtual texture address range corresponding to the node data to determine free tile blocks, and writing the data of the newly added node to be rendered into the free tile blocks, the hybrid rendering method further includes: Delete the node data that is not involved in rendering in the current frame from the page table structure, and add the unique identifier of the newly added node to be rendered, its starting offset in the virtual texture, and its data byte length to the index entry.

6. The hybrid rendering method according to claim 1, characterized in that, The page table structure is an array of structures, and the page table structure includes: Each structure element contains the two-dimensional coordinates of the tile in the virtual texture, the number of valid data bytes within the tile, and a boolean flag indicating whether the tile is visible in the current frame; wherein, the page table structure is maintained synchronously in GPU memory as a read-only copy for the GPU shader program to index in real time.

7. The hybrid rendering method according to claim 1, characterized in that, Prior to per-primal rendering using a semi-transparent blending method that does not write to the depth buffer in the GPU, the blending rendering method also includes: Call the GPU driver interface to query the current available video memory. If the available video memory is lower than a preset threshold, the CPU is triggered to pre-release the tile blocks that have the lowest historical access frequency in the page table structure and are not visible in the current frame, generating a pre-release candidate set; wherein, the pre-release candidate set is preferentially selected in the next frame.

8. A hybrid rendering device for 3DGS scenes, characterized in that, The hybrid rendering apparatus includes: The filtering module is used to perform view frustum clipping on multiple 3DGS scenes participating in hybrid rendering, and store all the nodes to be rendered in the list of nodes to be rendered; where each node to be rendered is only a data block without scene attributes that carries 3DGS primitives; The partitioning module is used to split each node in the list of nodes to be rendered into one or more data segments the size of a tile according to its data size, and write each data segment into the free tile of the virtual texture in sequence; The global sorting module is used to traverse all 3DGS primitives loaded into the virtual texture based on a pre-built page table structure, and determine the sorting index sequence by performing a global sort based on the depth value of each 3DGS primitive. The rendering module is used to sequentially access the corresponding 3DGS primitive parameters in the virtual texture according to the sorted index sequence, and perform primitive-by-primitive rendering in the GPU using a semi-transparent blending method that does not write to the depth buffer.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the hybrid rendering method for a 3DGS scene as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the hybrid rendering method for a 3DGS scene as described in any one of claims 1 to 7.

Images