Three-dimensional Gaussian data time domain filtering method and device, equipment and storage medium
By performing temporal filtering on 3D Gaussian data, sorting and dividing it into multiple blocks, calculating the block difference degree and correcting parameters, the screen flickering problem in the presentation of 3D Gaussian data was solved, improving screen continuity and user experience.
Patent Information
- Application Number
- CN202511282002.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-09
- Publication Date
- 2025-12-12
AI Technical Summary
During the presentation of 3D Gaussian data, noise can cause issues such as screen flickering, affecting the user's viewing experience.
The three-dimensional Gaussian data is subjected to temporal filtering, including sorting and dividing into multiple blocks. The block difference is calculated, and when the difference is within a threshold range, the block parameters of the current frame are corrected using the blocks of the previous frame, until all blocks are traversed.
It improves the smoothness and continuity of 3D Gaussian data, solves the problems of screen flickering and discontinuity, and enhances the user experience.
Smart Images
Figure CN121120401A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a method, apparatus, device, and storage medium for temporal filtering of three-dimensional Gaussian data. Background Technology
[0002] 3D Gaussian Splatting, as an emerging 3D reconstruction method, significantly outperforms traditional techniques in terms of realistic representation and real-time rendering capabilities, gradually becoming a core tool for next-generation 3D scene modeling. This method is based on explicit Gaussian primitives and achieves high-quality modeling of complex scenes through a progressive densification algorithm. As a high-fidelity 3D scene representation method, 3D Gaussian Splatting has a wide variety of applications, such as online shopping, immersive communication / conference, and immersive education.
[0003] 3D Gaussian data is a special data structure used to represent 3D scenes. A frame of a 3D scene consists of multiple 3D Gaussian primitives, each of which can be viewed as a 3D Gaussian ellipsoid, and each primitive is represented by a series of parameters. The quality of 3D Gaussian data is affected by the quality of the original captured photos / videos and the performance of the training algorithm. Moreover, since the training process is also based on the original captured photos / videos, noise generated during the shooting process will continue to propagate during training, resulting in a detrimental viewing experience for the user during the final presentation, such as screen flickering. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide a three-dimensional Gaussian data temporal filtering method, apparatus, device, and storage medium, which can improve the smoothness and continuity of the image and alleviate the image flickering problem caused by anomalies. The specific solution is as follows:
[0005] In a first aspect, this application discloses a three-dimensional Gaussian data temporal filtering method, comprising:
[0006] Acquire initial 3D Gaussian data, sort and divide the 3D Gaussian primitives corresponding to the 3D scene of the kth frame according to the target sorting and partitioning rules, and obtain multiple blocks corresponding to the 3D scene of the kth frame after sorting.
[0007] Traverse the blocks of the sorted k-th frame 3D scene and calculate the block difference degree of the blocks; the block difference degree is the difference degree between two blocks at corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene;
[0008] If the difference between the blocks is within the threshold range, the three-dimensional Gaussian primitives of the corresponding blocks in the sorted (k-1)th frame are used to correct the three-dimensional Gaussian primitives of the current block in the sorted kth frame of the three-dimensional scene, until all blocks in each frame of the three-dimensional scene have been traversed, and the temporally filtered three-dimensional Gaussian data is obtained.
[0009] Optionally, the step of sorting and dividing the 3D Gaussian elements corresponding to the 3D scene in the k-th frame according to the target sorting and partitioning rules to obtain multiple blocks corresponding to the 3D scene in the k-th frame after sorting includes:
[0010] Map all three-dimensional Gaussian elements corresponding to the three-dimensional scene in the k-th frame to a target number of two-dimensional planes; the target number is the total number of dimensions of the target three-dimensional Gaussian element parameters determined in advance;
[0011] The parallel linear allocation sorting method is used to sort the three-dimensional Gaussian elements mapped to each layer of two-dimensional plane to obtain the sorted three-dimensional scene of the k-th frame.
[0012] Each two-dimensional plane corresponding to the sorted k-th frame 3D scene is divided into multiple two-dimensional sub-blocks to obtain multiple blocks corresponding to the sorted k-th frame 3D scene; wherein, a block contains the two-dimensional sub-blocks corresponding to that block on all two-dimensional planes.
[0013] Optionally, traversing the blocks of the sorted k-th frame of the 3D scene and calculating the block difference degree includes:
[0014] Calculate the difference between two pixels at corresponding positions on the same type of two-dimensional plane corresponding to the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene to obtain the pixel difference degree;
[0015] Based on the pixel differences corresponding to the two-dimensional sub-blocks, the differences between two two-dimensional sub-blocks at corresponding positions on the same type of two-dimensional plane are determined, and the two-dimensional sub-block differences are obtained.
[0016] Based on the difference degree of all two-dimensional sub-blocks corresponding to the block, the difference between the two blocks in the corresponding positions in the sorted k-th frame 3D scene and the sorted k-1-th frame 3D scene is determined by weighted summation, and the block difference degree is obtained.
[0017] Optionally, determining the difference between two two-dimensional sub-blocks at corresponding positions on the same type of two-dimensional plane based on the pixel differences corresponding to the two-dimensional sub-blocks includes:
[0018] Based on the pixel differences corresponding to the two-dimensional sub-blocks, the difference between two two-dimensional sub-blocks at corresponding positions on the same type of two-dimensional plane is calculated using the error sum of squares algorithm.
[0019] Optionally, the step of sorting and dividing the 3D Gaussian elements corresponding to the 3D scene in the k-th frame according to the target sorting and partitioning rules to obtain multiple blocks corresponding to the 3D scene in the k-th frame after sorting includes:
[0020] Based on the center point coordinates of the three-dimensional Gaussian elements, the space of all three-dimensional Gaussian elements corresponding to the k-th frame of the three-dimensional scene is divided into multiple three-dimensional grids to obtain multiple blocks corresponding to the sorted k-th frame of the three-dimensional scene.
[0021] Optionally, traversing the blocks of the sorted k-th frame of the 3D scene and calculating the block difference degree includes:
[0022] Calculate the difference between two Gaussian primitives at corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene in the same parameter dimension, and obtain the primitive difference in the single parameter dimension;
[0023] Based on the differences of all primitives corresponding to the same parameter dimension in the 3D mesh, the difference between two 3D meshes at corresponding positions in two frames in the same parameter dimension is determined by summation, and the mesh difference degree in a single parameter dimension is obtained.
[0024] Based on the differences between all grids in each parameter dimension of the 3D grid, the differences between two blocks in corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene are determined by weighted summation, thus obtaining the block difference degree.
[0025] Optionally, the step of correcting the 3D Gaussian elements of the current block in the 3D scene of the k-th frame using the 3D Gaussian elements of the corresponding block in the sorted (k-1)-th frame includes:
[0026] Obtain the first weight for the corresponding block in the sorted (k-1)th frame, and the second weight for the current block in the 3D scene of the sorted (k)th frame;
[0027] Based on the first weight, the corresponding block in the sorted (k-1)th frame, the second weight, and the current block in the sorted kth frame 3D scene, the filtered 3D Gaussian elements of the current block in the sorted kth frame 3D scene are obtained by weighted averaging.
[0028] Secondly, this application discloses a three-dimensional Gaussian data time-domain filtering device, comprising:
[0029] The data sorting and partitioning module is used to acquire the initial 3D Gaussian data, sort and partition the 3D Gaussian primitives corresponding to the 3D scene of the kth frame according to the target sorting and partitioning rules, and obtain multiple blocks corresponding to the 3D scene of the kth frame after sorting.
[0030] The block difference calculation module is used to traverse the blocks of the sorted k-th frame 3D scene and calculate the block difference of the blocks; the block difference is the difference between two blocks at corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene.
[0031] The three-dimensional Gaussian element correction module is used to correct the three-dimensional Gaussian elements of the current block in the three-dimensional scene of the k-th frame after sorting by using the three-dimensional Gaussian elements of the corresponding block in the sorted (k-1)-th frame, until all blocks in the three-dimensional scene of each frame have been traversed, and then obtains the three-dimensional Gaussian data after temporal filtering.
[0032] Thirdly, this application discloses an electronic device, including:
[0033] Memory, used to store computer programs;
[0034] A processor is used to execute the computer program to implement the aforementioned three-dimensional Gaussian data time-domain filtering method.
[0035] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned three-dimensional Gaussian data time-domain filtering method.
[0036] In this application, initial 3D Gaussian data is obtained, and the 3D Gaussian primitives corresponding to the k-th frame 3D scene are sorted and divided according to the target sorting and partitioning rules to obtain multiple blocks corresponding to the sorted k-th frame 3D scene; the blocks of the sorted k-th frame 3D scene are traversed, and the block difference degree of the blocks is calculated; the block difference degree is the difference between two blocks in corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene; if the block difference degree is within a threshold range, the 3D Gaussian primitives of the current block in the sorted k-th frame 3D scene are corrected using the 3D Gaussian primitives of the corresponding block in the sorted (k-1)-th frame 3D scene, until all blocks in each frame 3D scene are traversed, and the temporally filtered 3D Gaussian data is obtained.
[0037] The beneficial effects of this application are as follows: by sorting and dividing the three-dimensional Gaussian primitives of each frame to obtain multiple corresponding blocks, by comparing the blocks at corresponding positions in two adjacent frames, the difference between the two blocks is calculated. If the difference is within the threshold range, it means that the two blocks in the two frames correspond to the same object. At this time, the blocks of the current frame are corrected by the blocks of the previous frame. Thus, the image generated based on the three-dimensional Gaussian data after temporal filtering can improve the smoothness and continuity of the image and solve some problems of discontinuity and flicker caused by anomalies. Attached Figure Description
[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0039] Figure 1 A flowchart of a three-dimensional Gaussian data time-domain filtering method provided in this application;
[0040] Figure 2 A flowchart of a specific three-dimensional Gaussian data temporal filtering method provided in this application;
[0041] Figure 3 A schematic diagram of a three-dimensional Gaussian data time-domain filtering device is provided in this application;
[0042] Figure 4 This application provides a structural diagram of an electronic device. Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0044] The generation and presentation process of 3D Gaussian data is briefly described as follows: A camera captures photos / videos from multiple angles; an algorithm reconstructs the original point cloud data corresponding to the current scene from the captured photos / videos; the original point cloud data is represented as M 3D Gaussian primitives; through machine learning training, the initial M 3D Gaussian primitives are continuously optimized, ultimately obtaining N 3D Gaussian primitives, which is the 3D Gaussian data representation of the current scene; based on the user's viewing / operation behavior, K (K≤N) 3D Gaussian primitives corresponding to the current viewpoint are obtained; a projection algorithm is used to obtain a 2D image corresponding to the K 3D Gaussian primitives, and this 2D image is presented to the user. It is known that the quality of 3D Gaussian data is affected by the quality of the original captured photos / videos and the performance of the training algorithm. Moreover, since the training process is also based on the original captured photos / videos, noise generated during the shooting process will continue to propagate during training, resulting in a detrimental effect on the user's viewing experience during the final presentation, such as screen flickering in the 3D Gaussian data presentation. To overcome the above technical problems, this application proposes a temporal filtering method for 3D Gaussian data, which can improve the smoothness and continuity of the image and mitigate screen flickering caused by anomalies.
[0045] This application discloses a three-dimensional Gaussian data temporal filtering method. See [link to relevant documentation]. Figure 1 As shown, the method may include the following steps:
[0046] Step S11: Obtain the initial 3D Gaussian data, sort and divide the 3D Gaussian primitives corresponding to the 3D scene of the kth frame according to the target sorting and partitioning rules, and obtain multiple blocks corresponding to the 3D scene of the kth frame after sorting.
[0047] In this embodiment, each frame of the 3D scene is traversed sequentially. According to the target sorting and partitioning rules, all 3D Gaussian primitives corresponding to the currently traversed k-th frame of the 3D scene are sorted and partitioned, resulting in multiple blocks corresponding to the sorted k-th frame of the 3D scene; that is, all 3D Gaussian primitives corresponding to a frame of the 3D scene are divided into multiple sub-parts. Subsequent analysis and correction are performed on a block-by-block basis, which is more accurate and has a better correction effect compared to analysis and correction of the entire frame. The aforementioned target sorting and partitioning rules can include at least two types: one is to project the 3D Gaussian primitives onto a two-dimensional plane and further partition them on the two-dimensional plane; the other is to directly partition the 3D Gaussian primitives in three-dimensional space.
[0048] Step S12: Traverse the blocks of the sorted k-th frame 3D scene and calculate the block difference degree of the blocks; the block difference degree is the difference degree between two blocks in corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene.
[0049] For the current k-th frame of the 3D scene, traverse all blocks in the frame and calculate the block difference degree of each block. The block difference degree is the difference between two blocks in corresponding positions in the sorted k-th frame of the 3D scene and the sorted (k-1)-th frame of the 3D scene. Corresponding positions are the same positions. That is, calculate the difference between a block in the sorted k-th frame of the 3D scene and the block corresponding to that block in the sorted (k-1)-th frame of the 3D scene.
[0050] It should be noted that each 3D Gaussian primitive is represented by a series of parameters, including: center point coordinates (a 3D vector, x, y, z), size (a 3D vector representing the extent of the Gaussian ellipsoid's extension along each axis), rotation information (a 4D vector defining the ellipsoid's orientation in space), and color spherical harmonic coefficients (the dimension of these coefficients depends on the order of the spherical harmonic function (SHdegree), specifically...). The three-dimensional Gaussian primitives contain dozens of parameter dimensions, such as the x-coordinate of the center point (which encodes the viewpoint-dependent appearance color) and opacity (a one-dimensional scalar). For example, the x-coordinate of the center point is one dimension, and the y-coordinate is another. When comparing differences, parameters of the same dimension need to be compared, but it is not necessary to compare all parameter dimensions. A selection of parameter dimensions can be chosen for comparison, and the results can be analyzed. The specific parameter dimensions selected can be customized by the user.
[0051] Step S13: If the block difference is within the threshold range, the 3D Gaussian primitives of the corresponding block in the sorted (k-1)th frame are used to correct the 3D Gaussian primitives of the current block in the sorted kth frame 3D scene until all blocks in each frame 3D scene have been traversed, and the temporally filtered 3D Gaussian data is obtained.
[0052] In this embodiment, after obtaining the block difference degree of the current block, it is determined whether the block difference degree is within a preset threshold range. If it is within the threshold range, it indicates that the real objects corresponding to the two blocks involved in the block difference degree are the same. At this time, the block in the current frame is corrected by using the block in the previous frame corresponding to the current block in the current frame. If the block difference degree is not within the preset threshold range, it indicates that the real objects corresponding to the two blocks involved in the block difference degree are different. In this case, a large block difference degree is normal, and no processing is performed on the block. After processing the current block of the k-th frame, the next block of the k-th frame is processed. The principle is the same, that is, comparing the difference degree between the next block of the k-th frame and the block corresponding to the next block in the (k-1)-th frame. It should be noted that the object of correction in this step is the parameter of the three-dimensional Gaussian unit. Specifically, it can be the correction of the parameter of the target three-dimensional Gaussian unit involved in the analysis in step S12. That is, the parameter value of the current Gaussian unit is filtered according to the parameter values of the Gaussian units at the same or adjacent positions in adjacent frames.
[0053] In a specific embodiment, the step of using the 3D Gaussian elements of the corresponding block in the sorted (k-1)th frame to correct the 3D Gaussian elements of the current block in the sorted k-th frame 3D scene may include: obtaining a first weight for the corresponding block in the sorted (k-1)th frame and a second weight for the current block in the sorted k-th frame 3D scene; and obtaining the filtered 3D Gaussian elements of the current block in the sorted k-th frame 3D scene by weighted averaging based on the first weight, the corresponding block in the sorted (k-1)th frame, the second weight, and the current block in the sorted k-th frame 3D scene.
[0054] Specifically, calculate the first weight and the block in the (k-1)th frame after sorting. The The first product is obtained by multiplying the first parameters of each 3D Gaussian element; the second weight is calculated and combined with the sorted blocks in the k-th frame of the 3D scene. The The product of the first parameters of each 3D Gaussian element is used to obtain the second product. The sum of the first product and the second product is used as the block in the filtered 3D scene of the k-th frame. The The parameter values of the first parameter of a three-dimensional Gaussian element.
[0055] As can be seen from the above, in this embodiment, initial 3D Gaussian data is obtained, and the 3D Gaussian primitives corresponding to the k-th frame 3D scene are sorted and divided according to the target sorting and partitioning rules to obtain multiple blocks corresponding to the sorted k-th frame 3D scene; the blocks of the sorted k-th frame 3D scene are traversed, and the block difference degree of the blocks is calculated; the block difference degree is the difference degree between two blocks in corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene; if the block difference degree is within the threshold range, the 3D Gaussian primitives of the current block in the sorted k-th frame 3D scene are corrected using the 3D Gaussian primitives of the corresponding block in the sorted (k-1)-th frame 3D scene, until all blocks in each frame 3D scene are traversed, and the temporally filtered 3D Gaussian data is obtained. As can be seen, by sorting and dividing the three-dimensional Gaussian primitives of each frame to obtain multiple corresponding blocks, and by comparing the blocks at corresponding positions in two adjacent frames, the difference between the two blocks is calculated. If the difference is within the threshold range, it means that the two blocks in the two frames correspond to the same object. At this time, the blocks of the current frame are corrected by the blocks of the previous frame. Thus, the image generated based on the three-dimensional Gaussian data after temporal filtering can improve the smoothness and continuity of the image and solve some problems of image discontinuity and flicker caused by anomalies.
[0056] Based on the above embodiments, the specific three-dimensional Gaussian data temporal filtering method includes the following steps:
[0057] Step S21: Map all three-dimensional Gaussian elements corresponding to the three-dimensional scene in the k-th frame to a target number of two-dimensional planes; the target number is the total number of dimensions of the target three-dimensional Gaussian element parameters determined in advance.
[0058] Get all ( ) corresponding to the 3D scene in frame k. After (one) three-dimensional Gaussian elements, A 3D Gaussian primitive is mapped to P (the total dimension of all parameters) 2D planes, where P is the number of targets. For example, if x-coordinate, y-coordinate, z-coordinate, opacity, and 0th-order color coefficient (0th-order color coefficient corresponds to 3 dimensions) are selected as the target 3D Gaussian primitive parameters, then P is 7.
[0059] Step S22: Use the parallel linear allocation sorting method to sort the three-dimensional Gaussian primitives mapped to each layer of two-dimensional plane to obtain the sorted three-dimensional scene of the k-th frame.
[0060] The k-th frame of the 3D scene is obtained by sorting using the Parallel Linear Assignment Sorting (PLAS) algorithm. The PLAS algorithm is suitable for parallel processing of high-dimensional, large-scale data. Its core principle is to iteratively optimize the Gaussian parameters in the 2D mesh to gradually approach the smooth target mesh. The specific steps are as follows: 1. Initialization and Iterative Framework: Randomly map all Gaussian parameters to the initial position of the 2D mesh to avoid getting trapped in local optima; gradually optimize the mesh's orderliness through a loop of "Gaussian blur to generate the target mesh, reallocate elements, and reduce filter size". 2. Block Parallel Processing: To process millions of Gaussian parameters, the 2D mesh is divided into multiple blocks, with block size... Set as ( The block size is no less than 16 (to ensure parallel efficiency), and the block size is the Gaussian blur radius. To avoid discontinuous sorting caused by block boundaries, all blocks are randomly offset along the x and y directions before each reallocation. This ensures that Gaussian parameters at the boundaries are also effectively sorted. Each block is processed independently, with Gaussian parameters within the block grouped into groups of four. All 24 possible permutations are tried, and the permutation with the smallest difference from the smooth target mesh is selected to achieve local optimum allocation. 3. Integration with the training process. Sorting is not a post-processing step independent of 3DGS training, but is deeply coupled with training. An initial sorting is performed at the beginning of training, and then re-sorting is performed after each "densification" step. Multi-attribute collaborative sorting is adopted, where all Gaussian parameters (position, color, scale, rotation, etc.) are mapped to the same location on different 2D planes, forming multi-layer data. Sorting must satisfy the collaborative optimization of all attributes, rather than sorting a single attribute, to ensure the consistency of the mesh layout.
[0061] Step S23: Divide each two-dimensional plane corresponding to the sorted k-th frame 3D scene into multiple two-dimensional sub-blocks to obtain multiple blocks corresponding to the sorted k-th frame 3D scene; wherein, a block contains the two-dimensional sub-blocks corresponding to the block on all two-dimensional planes;
[0062] The k-th frame corresponds to P two-dimensional planes, and each two-dimensional plane is divided into Mk two-dimensional sub-blocks; that is, one block corresponds to P two-dimensional sub-blocks.
[0063] Step S24: Calculate the difference between two pixels at corresponding positions on the same type of two-dimensional plane corresponding to the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene, and obtain the pixel difference degree;
[0064] Step S25: Based on the pixel difference degree corresponding to the two-dimensional sub-block, determine the difference between two two-dimensional sub-blocks at corresponding positions on the same type of two-dimensional plane, and obtain the two-dimensional sub-block difference degree;
[0065] Specifically, the difference between two two-dimensional sub-blocks at corresponding positions on the same type of two-dimensional plane can be calculated using the sum of squared errors algorithm based on the difference of all pixels corresponding to the two-dimensional sub-blocks.
[0066] Using x-coordinate, y-coordinate, z-coordinate, opacity, and 0th-order color coefficient as the target 3D Gaussian primitive parameters, for example: based on P 2D planes, we calculate the difference between pixels in each sub-block of frame k (each pixel is a parameter value of a 3D Gaussian primitive) and pixels in the corresponding sub-block of frame (k-1). This can be calculated using the SSD (Sum of Squared Difference) algorithm, as shown in the formula below:
[0067] ;
[0068] Where cur(i,j) is the value of pixel (i,j) on the p-th plane of the k-th frame, and ref(i,j) is the value of pixel (i,j) on the p-th plane of the (k-1)-th frame. The difference between cur(i,j) and ref(i,j) is the pixel difference. Pixel (i,j) belongs to a two-dimensional sub-block. The difference between two two-dimensional sub-blocks at corresponding positions on the p-th plane is calculated by the sum of squared errors, resulting in the two-dimensional sub-block difference SSDp.
[0069] Step S26: Based on the difference degree of all two-dimensional sub-blocks corresponding to the block, determine the difference between the two blocks in the corresponding positions in the sorted k-th frame 3D scene and the sorted k-1-th frame 3D scene by weighted summation, and obtain the block difference degree.
[0070] Based on the difference between the blocks in the 2D sub-blocks on the 7 2D planes, and the weights corresponding to the different 2D planes, a weighted sum is used to obtain the difference between two blocks at corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene, thus obtaining the block difference degree. The specific calculation formula is as follows:
[0071] ;
[0072] in, This represents the weight of the two-dimensional plane p. The specific weight of the two-dimensional plane can be customized according to requirements, or a default value of 1 / p can be used.
[0073] Step S26: If the block difference is within the threshold range, the 3D Gaussian primitives of the corresponding block in the sorted (k-1)th frame are used to correct the 3D Gaussian primitives of the current block in the sorted kth frame 3D scene until all blocks in each frame 3D scene are traversed, and the temporally filtered 3D Gaussian data is obtained.
[0074] For block difference D within a certain threshold [Th] lower , Th upper For each 3D Gaussian element in the block, the target parameters (such as transparency, color, or other parameters that can be customized as needed) are weighted and averaged using the corresponding 3D Gaussian elements in the (k-1)th frame as a benchmark.
[0075] ;
[0076] Where wi is the weight. This represents the parameter value of parameter i in the k-th frame. This represents the parameter value of parameter i in the (k-1)th frame.
[0077] As can be seen, by mapping three-dimensional Gaussian primitives onto a number of two-dimensional planes, and dividing each two-dimensional plane into multiple two-dimensional sub-blocks, and further, by calculating the differences between parameters at corresponding positions, the differences between two-dimensional sub-blocks are obtained, thereby determining the differences between blocks; this improves the accuracy of calculating the differences between two blocks at corresponding positions in two adjacent frames; finally, based on a threshold, it is determined whether to correct, and the blocks in the current frame are corrected in combination with the blocks in the previous frame, which can improve the smoothness and continuity of the image and solve some problems of image discontinuity and flicker caused by anomalies.
[0078] This application also discloses another specific method for temporal filtering of three-dimensional Gaussian data, including the following steps:
[0079] Step S31: Based on the center point coordinates of the three-dimensional Gaussian elements, divide the space of all three-dimensional Gaussian elements corresponding to the three-dimensional scene of the k-th frame into multiple three-dimensional grids to obtain multiple blocks corresponding to the sorted three-dimensional scene of the k-th frame.
[0080] For the three-dimensional space partitioning, the Nk three-dimensional Gaussian primitives are divided into Mk three-dimensional meshes directly according to the center point coordinates (x,y,z) of each primitive. The k-th frame is divided into Mk three-dimensional meshes, that is, the three-dimensional space is sorted according to the center point coordinates, and the three-dimensional space sorting is obtained after partitioning.
[0081] Step S32: Calculate the difference between two Gaussian primitives at corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene in the same parameter dimension, and obtain the primitive difference in the single parameter dimension.
[0082] The two Gaussian elements at corresponding positions in the two frames can be determined by their center coordinates, meaning the centerline coordinates of the two Gaussian elements are the same, or the difference between them is within a very small preset range, meaning the two Gaussian elements are close to each other. For example, calculating the Gaussian element i in the k-th frame... k The transparency, and the Gaussian element i in the (k-1)th frame k-1 The differences in transparency yield the primitive differences for various parameter dimensions. Which parameter dimensions' primitive differences to calculate depends on which dimensions' differences are used to determine whether a block needs correction; this can be customized based on actual usage requirements.
[0083] Step S33: Based on the differences of all primitives corresponding to the same parameter dimension in the three-dimensional mesh, the difference between the two three-dimensional meshes at corresponding positions in the two frames in the same parameter dimension is determined by summation, and the mesh difference degree in the single parameter dimension is obtained.
[0084] Using the Sum of Squared Difference (SSD) method, the difference between two 3D meshes at corresponding positions in two frames along the same parameter dimension is calculated. Specifically, it's the sum of squared errors of the differences between all Gaussian primitives within the same 3D meshes in the two frames along the same parameter dimension. The calculation formula is as follows:
[0085] ;
[0086] cur(i) represents the value of parameter p corresponding to the three-dimensional Gaussian i in the three-dimensional mesh of the k-th frame, and ref(i) represents the value of parameter p corresponding to the three-dimensional Gaussian i in the three-dimensional mesh of the (k-1)-th frame.
[0087] Step S34: Based on the difference degree of all grids corresponding to each parameter dimension of the 3D grid, determine the difference between the two blocks in the corresponding positions in the sorted k-th frame 3D scene and the sorted k-1-th frame 3D scene by weighted summation, and obtain the block difference degree.
[0088] Finally, based on the mesh differences corresponding to each parameter dimension (i.e., the parameter dimensions required for analysis, or all parameter dimensions involved in Gaussian elements), the difference between two corresponding blocks (i.e., the 3D meshes) in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene is determined by weighted summation, thus obtaining the block difference degree. The calculation formula is as follows:
[0089] ;
[0090] The parameters are the center point (x, y, z) coordinates of the 3D Gaussian elements in the 3D mesh, the opacity, and the 0th-order color coefficient, with p = 7. Among these, The weights of parameter p.
[0091] Step S34: If the block difference is within the threshold range, the 3D Gaussian primitives of the corresponding block in the sorted (k-1)th frame are used to correct the 3D Gaussian primitives of the current block in the sorted kth frame 3D scene until all blocks in each frame 3D scene have been traversed, and the temporally filtered 3D Gaussian data is obtained.
[0092] The specific process of step S34 can be found in the relevant content disclosed in the foregoing embodiments, and will not be repeated here.
[0093] The Gaussian primitive space is divided into multiple 3D grids. The difference between two Gaussian primitives at corresponding positions in the same parameter dimension is calculated to obtain the primitive difference in a single parameter dimension, and then the grid difference degree in a single parameter dimension. Finally, the difference between two 3D grids is determined. Finally, a threshold is used to determine whether to correct, and the blocks in the current frame are corrected in combination with the blocks in the previous frame. This can improve the smoothness and continuity of the image and solve some image discontinuity and flickering problems caused by anomalies.
[0094] For example Figure 2 As shown, this application can choose either of the two improved block difference calculation methods mentioned above. If the difference is within the threshold range, that is, it indicates that the two blocks of the two frames correspond to the same object, then the blocks of the current frame are corrected by the blocks of the previous frame. The image generated based on the three-dimensional Gaussian data after temporal filtering can improve the smoothness and continuity of the image and solve some problems of discontinuity and flickering caused by abnormalities.
[0095] Furthermore, this application also discloses a method for generating 3D effects, specifically including: acquiring initial 3D Gaussian data; sorting and dividing the 3D Gaussian primitives corresponding to the k-th frame 3D scene according to the target sorting and partitioning rules to obtain multiple blocks corresponding to the sorted k-th frame 3D scene; traversing the blocks of the sorted k-th frame 3D scene and calculating the block difference degree of the blocks; the block difference degree is the difference degree between two blocks in corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene; if the block difference degree is within a threshold range, then using the 3D Gaussian primitives of the corresponding blocks in the sorted (k-1)-th frame to correct the 3D Gaussian primitives of the current block in the sorted k-th frame 3D scene, until all blocks in each frame 3D scene have been traversed to obtain temporally filtered 3D Gaussian data; and generating 3D effects based on the temporally filtered 3D Gaussian data. The 3D video generated through the above steps solves the screen flickering problem existing in the 3D video generated by the prior art and improves the quality of the 3D video.
[0096] Accordingly, this application also discloses a three-dimensional Gaussian data time-domain filtering device, see [link to relevant documentation]. Figure 3 As shown, the device includes:
[0097] The data sorting and partitioning module 11 is used to acquire the initial three-dimensional Gaussian data, sort and partition the three-dimensional Gaussian primitives corresponding to the three-dimensional scene of the kth frame according to the target sorting and partitioning rules, and obtain multiple blocks corresponding to the three-dimensional scene of the kth frame after sorting.
[0098] Block difference calculation module 12 is used to traverse the blocks of the sorted k-th frame 3D scene and calculate the block difference of the blocks; the block difference is the difference between two blocks in corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene.
[0099] The 3D Gaussian primitive correction module 13 is used to correct the 3D Gaussian primitive of the current block in the 3D scene of the sorted k-1th frame by using the 3D Gaussian primitive of the corresponding block in the sorted k-1th frame, until all blocks in the 3D scene of each frame have been traversed, and then obtain the temporally filtered 3D Gaussian data.
[0100] As can be seen from the above, in this embodiment, initial 3D Gaussian data is obtained, and the 3D Gaussian primitives corresponding to the k-th frame 3D scene are sorted and divided according to the target sorting and partitioning rules to obtain multiple blocks corresponding to the sorted k-th frame 3D scene; the blocks of the sorted k-th frame 3D scene are traversed, and the block difference degree of the blocks is calculated; the block difference degree is the difference degree between two blocks in corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene; if the block difference degree is within the threshold range, the 3D Gaussian primitives of the current block in the sorted k-th frame 3D scene are corrected using the 3D Gaussian primitives of the corresponding block in the sorted (k-1)-th frame 3D scene, until all blocks in each frame 3D scene are traversed, and the temporally filtered 3D Gaussian data is obtained.
[0101] As can be seen, by sorting and dividing the three-dimensional Gaussian primitives of each frame to obtain multiple corresponding blocks, and by comparing the blocks at corresponding positions in two adjacent frames, the difference between the two blocks is calculated. If the difference is within the threshold range, it means that the two blocks in the two frames correspond to the same object. At this time, the blocks of the current frame are corrected by the blocks of the previous frame. Thus, the image generated based on the three-dimensional Gaussian data after temporal filtering can improve the smoothness and continuity of the image and solve some problems of image discontinuity and flicker caused by anomalies.
[0102] In some specific embodiments, the data sorting and partitioning module 11 may specifically include:
[0103] A mapping unit is used to map all three-dimensional Gaussian elements corresponding to the three-dimensional scene of the k-th frame to a target number of two-dimensional planes; the target number is the total number of dimensions of the target three-dimensional Gaussian element parameters that are predetermined.
[0104] The sorting unit is used to sort the three-dimensional Gaussian elements mapped to each layer of two-dimensional plane using a parallel linear allocation sorting method, so as to obtain the sorted three-dimensional scene of the k-th frame.
[0105] A partitioning unit is used to divide each two-dimensional plane corresponding to the sorted k-th frame 3D scene into multiple two-dimensional sub-blocks to obtain multiple blocks corresponding to the sorted k-th frame 3D scene; wherein, a block contains the two-dimensional sub-blocks corresponding to the block on all two-dimensional planes.
[0106] In some specific embodiments, the block difference calculation module 12 may specifically include:
[0107] The pixel difference calculation unit is used to calculate the difference between two pixels at corresponding positions on the same type of two-dimensional plane corresponding to the sorted k-th frame 3D scene and the sorted k-1-th frame 3D scene, and obtain the pixel difference degree.
[0108] The two-dimensional sub-block difference determination unit is used to determine the difference between two two-dimensional sub-blocks at corresponding positions on the same type of two-dimensional plane based on the difference degree of all pixels corresponding to the two-dimensional sub-block, and obtain the two-dimensional sub-block difference degree.
[0109] The block difference determination unit is used to determine the difference between two blocks at corresponding positions in the sorted k-th frame 3D scene and the sorted k-1-th frame 3D scene by weighted summation based on the difference degree of all the two-dimensional sub-blocks corresponding to the block, and obtain the block difference degree.
[0110] In some specific embodiments, the two-dimensional sub-block difference determination unit may specifically include:
[0111] The unit is used to calculate the difference between two two-dimensional sub-blocks at corresponding positions on the same type of two-dimensional plane using the sum of squared errors algorithm, based on the difference of all pixels corresponding to the two-dimensional sub-blocks.
[0112] In some specific embodiments, the two-dimensional sub-block difference determination unit can be used to divide the space of all three-dimensional Gaussian elements corresponding to the k-th frame three-dimensional scene into multiple three-dimensional grids based on the center point coordinates of the three-dimensional Gaussian elements, so as to obtain multiple blocks corresponding to the sorted k-th frame three-dimensional scene.
[0113] In some specific embodiments, the block difference calculation module 12 may specifically include:
[0114] The primitive difference calculation unit is used to calculate the difference between two Gaussian primitives at corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene in the same parameter dimension, and obtain the primitive difference in a single parameter dimension.
[0115] The mesh difference calculation unit is used to determine the difference between two 3D meshes at corresponding positions in two frames in the same parameter dimension by summing all the primitive differences of the 3D mesh in the same parameter dimension, and obtain the mesh difference degree in a single parameter dimension.
[0116] The block difference determination unit is used to determine the difference between two blocks in corresponding positions in the sorted k-th frame 3D scene and the sorted k-1-th frame 3D scene by weighted summation based on the difference degree of all grids corresponding to each parameter dimension of the 3D grid, and obtain the block difference degree.
[0117] In some specific embodiments, the three-dimensional Gaussian element correction module 13 may specifically include:
[0118] The weight acquisition unit is used to acquire the first weight for the corresponding block in the sorted (k-1)th frame, and the second weight for the current block in the 3D scene of the sorted kth frame.
[0119] The correction unit is used to obtain the filtered 3D Gaussian elements of the current block in the 3D scene of ...
[0120] Furthermore, this application also discloses an electronic device, see [link to relevant documentation]. Figure 4 As shown, the content in the figure should not be considered as any limitation on the scope of use of this application.
[0121] Figure 4 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the three-dimensional Gaussian data time-domain filtering method disclosed in any of the foregoing embodiments.
[0122] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0123] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include operating system 221, computer program 222 and data 223 including three-dimensional Gaussian data, etc. The storage method can be temporary storage or permanent storage.
[0124] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform calculations and processing on the massive data 223 in the memory 22. The operating system 221 can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the three-dimensional Gaussian data time-domain filtering method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.
[0125] Furthermore, this application also discloses a computer storage medium storing computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, they implement the steps of the three-dimensional Gaussian data time-domain filtering method disclosed in any of the foregoing embodiments.
[0126] Furthermore, this application also discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the three-dimensional Gaussian data time-domain filtering method disclosed in any of the foregoing embodiments.
[0127] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0128] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0129] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0130] The present invention provides a detailed description of a three-dimensional Gaussian data time-domain filtering method, apparatus, device, and storage medium. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A three-dimensional Gaussian data time-domain filtering method, characterized in that, include: Acquire initial 3D Gaussian data, sort and divide the 3D Gaussian primitives corresponding to the 3D scene of the kth frame according to the target sorting and partitioning rules, and obtain multiple blocks corresponding to the 3D scene of the kth frame after sorting. Traverse the blocks of the sorted k-th frame 3D scene and calculate the block difference degree of the blocks; the block difference degree is the difference degree between two blocks at corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene; If the difference between the blocks is within the threshold range, the three-dimensional Gaussian primitives of the corresponding blocks in the sorted (k-1)th frame are used to correct the three-dimensional Gaussian primitives of the current block in the sorted kth frame of the three-dimensional scene, until all blocks in each frame of the three-dimensional scene have been traversed, and the temporally filtered three-dimensional Gaussian data is obtained.
2. The three-dimensional Gaussian data time-domain filtering method according to claim 1, characterized in that, The 3D Gaussian elements corresponding to the 3D scene in the k-th frame are sorted and divided according to the target sorting and partitioning rules to obtain multiple blocks corresponding to the 3D scene in the k-th frame after sorting, including: Map all three-dimensional Gaussian elements corresponding to the three-dimensional scene in the k-th frame to a target number of two-dimensional planes; the target number is the total number of dimensions of the target three-dimensional Gaussian element parameters determined in advance; The parallel linear allocation sorting method is used to sort the three-dimensional Gaussian elements mapped to each layer of two-dimensional plane to obtain the sorted three-dimensional scene of the k-th frame. Each two-dimensional plane corresponding to the sorted k-th frame 3D scene is divided into multiple two-dimensional sub-blocks to obtain multiple blocks corresponding to the sorted k-th frame 3D scene; wherein, a block contains the two-dimensional sub-blocks corresponding to that block on all two-dimensional planes.
3. The three-dimensional Gaussian data time-domain filtering method according to claim 2, characterized in that, The step of traversing the blocks of the sorted k-th frame of the 3D scene and calculating the block difference degree of the blocks includes: Calculate the difference between two pixels at corresponding positions on the same type of two-dimensional plane corresponding to the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene to obtain the pixel difference degree; Based on the pixel differences corresponding to the two-dimensional sub-blocks, the differences between two two-dimensional sub-blocks at corresponding positions on the same type of two-dimensional plane are determined, and the two-dimensional sub-block differences are obtained. Based on the difference degree of all two-dimensional sub-blocks corresponding to the block, the difference between the two blocks in the corresponding positions in the sorted k-th frame 3D scene and the sorted k-1-th frame 3D scene is determined by weighted summation, and the block difference degree is obtained.
4. The three-dimensional Gaussian data time-domain filtering method according to claim 3, characterized in that, The step of determining the difference between two two-dimensional sub-blocks at corresponding positions on the same type of two-dimensional plane based on the pixel differences corresponding to the two-dimensional sub-blocks includes: Based on the pixel differences corresponding to the two-dimensional sub-blocks, the difference between two two-dimensional sub-blocks at corresponding positions on the same type of two-dimensional plane is calculated using the error sum of squares algorithm.
5. The three-dimensional Gaussian data time-domain filtering method according to claim 1, characterized in that, The 3D Gaussian elements corresponding to the 3D scene in the k-th frame are sorted and divided according to the target sorting and partitioning rules to obtain multiple blocks corresponding to the 3D scene in the k-th frame after sorting, including: Based on the center point coordinates of the three-dimensional Gaussian elements, the space of all three-dimensional Gaussian elements corresponding to the k-th frame of the three-dimensional scene is divided into multiple three-dimensional grids to obtain multiple blocks corresponding to the sorted k-th frame of the three-dimensional scene.
6. The three-dimensional Gaussian data time-domain filtering method according to claim 4, characterized in that, The step of traversing the blocks of the sorted k-th frame of the 3D scene and calculating the block difference degree of the blocks includes: Calculate the difference between two Gaussian primitives at corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene in the same parameter dimension, and obtain the primitive difference in the single parameter dimension; Based on the differences of all primitives corresponding to the same parameter dimension in the 3D mesh, the difference between two 3D meshes at corresponding positions in two frames in the same parameter dimension is determined by summation, and the mesh difference degree in a single parameter dimension is obtained. Based on the differences between all grids in each parameter dimension of the 3D grid, the differences between two blocks in corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene are determined by weighted summation, thus obtaining the block difference degree.
7. The three-dimensional Gaussian data time-domain filtering method according to any one of claims 1 to 6, characterized in that, The step of correcting the 3D Gaussian elements of the current block in the 3D scene of the k-th frame using the 3D Gaussian elements of the corresponding block in the sorted (k-1)-th frame includes: Obtain the first weight for the corresponding block in the sorted (k-1)th frame, and the second weight for the current block in the 3D scene of the sorted (k)th frame; Based on the first weight, the corresponding block in the sorted (k-1)th frame, the second weight, and the current block in the sorted kth frame 3D scene, the filtered 3D Gaussian elements of the current block in the sorted kth frame 3D scene are obtained by weighted averaging.
8. A three-dimensional Gaussian data time-domain filtering device, characterized in that, include: The data sorting and partitioning module is used to acquire the initial 3D Gaussian data, sort and partition the 3D Gaussian primitives corresponding to the 3D scene of the kth frame according to the target sorting and partitioning rules, and obtain multiple blocks corresponding to the 3D scene of the kth frame after sorting. The block difference calculation module is used to traverse the blocks of the sorted k-th frame 3D scene and calculate the block difference of the blocks; the block difference is the difference between two blocks at corresponding positions in the sorted k-th frame 3D scene and the sorted (k-1)-th frame 3D scene. The three-dimensional Gaussian element correction module is used to correct the three-dimensional Gaussian elements of the current block in the three-dimensional scene of the k-th frame after sorting by using the three-dimensional Gaussian elements of the corresponding block in the sorted (k-1)-th frame, until all blocks in the three-dimensional scene of each frame have been traversed, and then obtains the three-dimensional Gaussian data after temporal filtering.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the three-dimensional Gaussian data time-domain filtering method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein the computer programs, when executed by a processor, implement the three-dimensional Gaussian data time-domain filtering method as described in any one of claims 1 to 7.