A video image compression method, apparatus, device, and storage medium based on three-dimensional Gaussian pruning.
By constructing a dense 3D Gaussian scene representation, determining the structural importance score and error quality weight, generating an ordered residual coverage candidate set, and determining the skeleton budget ratio based on layout features, the number and position of Gaussian primitives to be retained are optimized, thus solving the problem of poor rendering quality in existing technologies and achieving stable rendering effects under high compression rates.
Patent Information
- Application Number
- CN202611489342.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-23
- Publication Date
- 2026-11-17
AI Technical Summary
Existing 3D Gaussian pruning methods struggle to simultaneously address structural and residual risks at high compression rates, resulting in poor rendering quality.
By constructing a dense 3D Gaussian scene representation, determining the structural importance score and error quality weight, generating an ordered residual cover candidate set, and determining the skeleton budget ratio based on layout features, constructing a structural skeleton and residual fill set for pruning, and optimizing the number and position of retained Gaussian primitives.
It improves the rendering quality after compression, ensuring the stability and quality of scene structure and local details under high compression rates.
Smart Images

Figure CN123123591A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of three-dimensional scene compression technology, and in particular to a video image compression method, apparatus, device and storage medium based on three-dimensional Gaussian pruning. Background Technology
[0002] Currently, 3D Gaussian Splatting (3DGS) utilizes a large number of explicit 3D Gaussian primitives to represent scenes, enabling high-quality real-time rendering. However, the number of Gaussians after dense training typically reaches hundreds of thousands to millions, resulting in high storage, transmission, and rendering overhead. To reduce the number of points, one important approach is to prune the Gaussians based on their importance scores and then perform short-term fine-tuning on the retained subset to restore image quality.
[0003] Current mainstream solutions typically employ a pure Topk selection based on structural importance scores (such as Fisher information and gradients), retaining the N points with the highest scores. While this approach effectively maintains the main structure and appearance of the scene, it fails to adequately cover "residual hotspots" with large local reconstruction errors but low scores. This deficiency is amplified, especially under conditions of high compression ratios and tight budgets, impacting the quality of new perspective synthesis.
[0004] Existing technologies mainly fall into three categories.
[0005] The first type is Topk pruning based on a single importance score, such as based on opacity, gradient, Fisher information, or a combination thereof, directly retaining the N Gaussians with the highest scores. This type of method is simple to implement, but the scores often favor structural contributions from the training perspective, making it difficult to simultaneously cover the residual quality space; image quality loss is significant under high compression rates. Fisher / uncertainty score pruning, represented by PUP (Principled Uncertainty Pruning), belongs to this category.
[0006] The second category involves residual or error-driven resampling / reweighting, such as directly rearranging or replacing importance scores using signals like rendering error or opacity multiplied by the error, and then taking the Topk. This type of method may improve local error regions, but it easily squeezes out high Fisher structural points, leading to global structural collapse.
[0007] The third category involves heuristic hybrid retention or post-hoc point filling. For example, a fixed proportion of highly important points are extracted, and the remaining points are filled using random, farthest point sampling (FPS), or voxel sampling. This type of method introduces the rudiments of skeleton and filling ideas, but the skeleton proportion is mostly a scene-independent fixed hyperparameter, which cannot be automatically adjusted according to whether the residual surplus is reliable or whether the spatial coverage is better in the current scene; too low a proportion will damage the structure, while too high a proportion will degenerate into an approximate pure Topk.
[0008] In summary, existing methods cannot simultaneously achieve the following: taking into account structural risk and residual risk under a fixed point budget, adaptively allocating skeleton and filling quotas using observable statistics, and deterministically filling candidates by covering them with ordered residuals, thereby achieving stable benefits under high compression rates.
[0009] As can be seen from the above, how to improve the rendering quality of compressed video images in the process of 3D Gaussian pruning is an urgent problem to be solved. Summary of the Invention
[0010] In view of this, the purpose of this invention is to provide a video image compression method, apparatus, device, and storage medium based on three-dimensional Gaussian pruning, which can improve the rendered image quality after compression during the video image compression process based on three-dimensional Gaussian pruning. The specific solution is as follows: Firstly, this application provides a video image compression method based on three-dimensional Gaussian pruning, including: A dense 3D Gaussian scene representation is constructed based on several multi-view images corresponding to video images in a 3D Gaussian sputtering scene and training camera parameters; the dense 3D Gaussian scene representation includes several Gaussian primitives. Determine the budget for the number of retention points allowed after pruning, and determine the structural importance score of each Gaussian primitive based on the dense 3D Gaussian scene representation and the training camera parameters, and construct a structural reference set based on each structural importance score and the budget for the number of retention points; Based on the rendering error between the rendered images at each training viewpoint and the corresponding multi-view images of the dense 3D Gaussian scene representation, the error quality weight is determined; Based on the error quality weights and spatial locations of each Gaussian primitive, several candidate Gaussian primitives are determined, and the candidate Gaussian primitives are sorted to obtain an ordered residual cover candidate set. The layout features are determined based on the error quality distribution and spatial distribution corresponding to the structural reference set and the ordered residual coverage candidate set, respectively, and the skeleton budget ratio is determined based on the layout features. The number of elements to be retained is determined based on the skeleton budget ratio and the number of retained points budget. The dense 3D Gaussian scene representation is pruned using the skeleton set and residual fill set selected from the structural reference set based on the structural importance score and the number of elements to be retained, resulting in a compressed 3D Gaussian scene representation. The compressed 3D Gaussian scene representation is then rendered and video encoded to obtain the compressed video image.
[0011] Optionally, the construction of a dense 3D Gaussian scene representation based on several multi-view images corresponding to video images in the 3D Gaussian sputtering scene and training camera parameters includes: Acquire several multi-view images corresponding to video images in a 3D Gaussian sputtering scene and training camera parameters corresponding to each multi-view image, and construct a dense 3D Gaussian scene representation based on the multi-view images and the training camera parameters; The dense 3D Gaussian scene representation includes several Gaussian primitives; each Gaussian primitive includes a center position, a geometric attribute used to characterize the spatial shape, an opacity attribute, and an appearance attribute.
[0012] Optionally, the step of determining the budgeted number of retainable points after pruning, determining the structural importance score of each Gaussian primitive based on the dense 3D Gaussian scene representation and the training camera parameters, and constructing a structural reference set based on each structural importance score and the budgeted number of retainable points includes: Based on the dense 3D Gaussian scene representation, a training camera set and a dense Gaussian set including each Gaussian primitive and attribute information are constructed. The total number of Gaussian primitives in the dense Gaussian set is processed based on a preset pruning ratio and a preset target retention number to obtain the number of Gaussian primitives that can be retained after pruning, and the number of Gaussian primitives is set as the retention point budget. The structural contribution of each Gaussian primitive in the 3D Gaussian sputtering scene is evaluated based on the dense Gaussian set and the training camera set, and a structural importance score corresponding to each Gaussian primitive is obtained. The Gaussian primitives are sorted based on their structural importance scores to obtain a sorting result. Then, based on the retention point budget, a number of Gaussian primitives with high structural importance are selected from the sorting result to construct a structural reference set based on each of the Gaussian primitives to be processed.
[0013] Optionally, the step of determining the error quality weight based on the rendering error between the rendered images at each training viewpoint and the corresponding multi-view images based on the dense 3D Gaussian scene representation includes: Several training cameras are selected from the training camera set, and a training viewpoint set for error evaluation is constructed based on each training camera. Then, the training viewpoints in the training viewpoint set and the dense Gaussian set are used for image rendering to obtain the rendered images corresponding to each training viewpoint. The rendered image is compared pixel-by-pixel with the multi-view image corresponding to the training viewpoint to obtain a pixel error map; the pixel error map is used to determine the local reconstruction error of each Gaussian primitive under each training viewpoint. The center positions of each Gaussian primitive in the dense Gaussian set are projected onto the image planes corresponding to each training viewpoint in the training viewpoint set, thereby obtaining the effective projection positions of each Gaussian primitive in each training viewpoint. The pixel error corresponding to the effective projection position is read from the pixel error map, and the pixel error of the Gaussian primitive under each training viewpoint is accumulated and normalized to obtain the local error value; the local error value is used to characterize the degree of reconstruction error of the Gaussian primitive in the image plane region; The local error value and the corresponding opacity attribute corresponding to each Gaussian primitive are jointly calculated to obtain the corresponding error quality weight; the data size of the error quality weight is positively correlated with the data size of the local error; the data size of the error quality weight is positively correlated with the degree of opacity.
[0014] Optionally, based on the error quality weights and spatial locations of each Gaussian primitive, several candidate Gaussian primitives are determined, and these candidate Gaussian primitives are sorted to obtain an ordered residual cover candidate set, including: Based on the center position of each Gaussian primitive in the dense Gaussian set, the three-dimensional space corresponding to the dense three-dimensional Gaussian scene representation is spatially partitioned to obtain a three-dimensional spatial partition. The error quality weights of the Gaussian primitives in each spatial partition are summarized to determine the residual quality corresponding to each spatial partition, and the corresponding candidate quota is determined based on the residual quality of each spatial partition. In each of the spatial partitions, candidate Gaussian primitives for the candidate slots are determined based on the error quality weights and the spatial positional relationships between the Gaussian primitives. The residual quality is then sorted and concatenated based on the selection order of the candidate Gaussian primitives to obtain an ordered residual coverage candidate set. The ordered residual coverage candidate set is used to evaluate the residual coverage capability.
[0015] Optionally, the step of determining layout features based on the error quality distribution and spatial distribution corresponding to the structural reference set and the ordered residual coverage candidate set, respectively, and determining the skeleton budget ratio based on the layout features, includes: Based on the error quality weight of each Gaussian primitive, the error quality of the Gaussian primitives in the structural reference set is summarized to obtain a first error quality characterization, and the error quality of the estimated number of retained points of the Gaussian primitives in the ordered residual cover candidate set is summarized to obtain a second error quality characterization. Error quality difference features are determined based on the difference between the first error quality characterization and the second error quality characterization; the error quality difference features are used to characterize the gain of the ordered residual coverage candidate set relative to the structural reference set in terms of high error region coverage that satisfies the preset high error condition. Based on the error quality weight of each Gaussian primitive, Gaussian primitives with error quality weights higher than preset conditions are determined from the dense Gaussian set, so as to construct a set of residual hot zones based on each Gaussian primitive to be processed. The coverage of the structural reference set and the ordered residual coverage candidate set over the residual hot zone set is determined respectively, resulting in a first coverage situation and a second coverage situation respectively. The residual hot zone coverage difference feature is determined based on the difference between the first coverage situation and the second coverage situation. The residual hot zone coverage difference feature is used to reflect the degree of omission of the high error region by the structural reference set and the compensation ability of the ordered residual coverage candidate set for the omission region. Based on the error quality weights of each Gaussian primitive, spatial query positions are determined from the dense Gaussian set, and a first distance is determined from each spatial query position to the nearest Gaussian primitive in the structural reference set. Then, a second distance is determined from each spatial query position to the nearest Gaussian primitive in the ordered residual cover candidate set. Spatial coverage difference features are determined based on the difference between the first distance and the second distance corresponding to each of the spatial query locations; the spatial coverage difference features are used to reflect the difference in coverage density of the high-error spatial region by the structural reference set and the ordered residual coverage candidate set, respectively. The error quality difference feature is jointly calculated with the residual hot zone coverage difference feature and / or the spatial coverage difference feature to obtain the skeleton budget ratio; the skeleton budget ratio is used to determine the budget share allocated to the structural skeleton in the budget for the number of reserved points.
[0016] Optionally, the step of determining the number of retained points based on the skeleton budget ratio and the number of retained points budget, and pruning the dense 3D Gaussian scene representation using a set of structural skeletons and a set of residual fills selected from the structural reference set based on the structural importance score and the number of retained points, to obtain a compressed 3D Gaussian scene representation, includes: A first retention quantity is determined based on the skeleton budget ratio and the retention point budget, and a second retention quantity is determined based on the difference between the retention point budget and the first retention quantity; the first retention quantity is used to construct the structural skeleton set; the second retention quantity is used to construct the residual filling set. Based on the structural importance score, the first number of Gaussian primitives are selected from the structural reference set in descending order of structural importance score to obtain the structural skeleton set; From the ordered residual cover candidate set, Gaussian primitives that do not exist in the structural skeleton set are selected sequentially to obtain the second number of retained Gaussian primitives, and thus the residual filling set is obtained. Wherein, when the number of Gaussian primitives available for filling in the ordered residual cover candidate set is less than the second retention number, Gaussian primitives that do not exist in the structural skeleton set and the residual fill set are selected from the structural reference set based on the structural importance score to obtain the residual fill set; The structural skeleton set and the residual filling set are merged to obtain the target retention set, wherein the number of Gaussian primitives in the target retention set is equal to the retention point budget; The retention center position of each Gaussian primitive in the target retention set is determined, and the spatially adjacent Gaussian primitives in the structural reference set are determined using the retention center positions. Based on the geometric, opacity and appearance properties of each adjacent Gaussian primitive, the properties of the corresponding Gaussian primitives are aggregated and updated to obtain the target retention set after attribute migration. Using the multi-view images and the corresponding training camera set, while keeping the number of Gaussian primitives in the target retention set after attribute transfer equal to the budgeted number of retention points, the parameters of the Gaussian primitives in the target retention set after attribute transfer are optimized to obtain an optimized retention set. Based on the optimized retention set, the dense 3D Gaussian scene representation is pruned to obtain a compressed 3D Gaussian scene representation.
[0017] Secondly, this application provides a video image compression device based on three-dimensional Gaussian pruning, comprising: A Gaussian scene representation generation module is used to construct a dense 3D Gaussian scene representation based on several multi-view images corresponding to video images in a 3D Gaussian sputtering scene and training camera parameters; the dense 3D Gaussian scene representation includes several Gaussian primitives. The importance score determination module is used to determine the budget of the number of retention points allowed to be retained after pruning, and to determine the structural importance score of each Gaussian primitive based on the dense 3D Gaussian scene representation and the training camera parameters, and to construct a structural reference set based on each structural importance score and the budget of the number of retention points; The error quality weight generation module is used to determine the error quality weight based on the rendering error between the rendered images of the dense 3D Gaussian scene representation under each training view and the corresponding multi-view images. The Gaussian primitive sorting module is used to determine several candidate Gaussian primitives based on the error quality weights and spatial positions of each Gaussian primitive, and sort each candidate Gaussian primitive to obtain an ordered residual cover candidate set. The layout feature generation module is used to determine the layout features based on the error quality distribution and spatial distribution corresponding to the structure reference set and the ordered residual coverage candidate set, respectively, and to determine the skeleton budget ratio based on the layout features. The video image compression module is used to determine the number of elements to be retained based on the skeleton budget ratio and the number of retained points budget, and to prune the dense 3D Gaussian scene representation using a set of structural skeletons and a set of residual fills selected from the structural reference set based on the structural importance score and the number of elements to be retained, thereby obtaining a compressed 3D Gaussian scene representation. The compressed 3D Gaussian scene representation is then rendered and video encoded to obtain the compressed video image.
[0018] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned video image compression method based on three-dimensional Gaussian pruning.
[0019] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned video image compression method based on three-dimensional Gaussian pruning.
[0020] As can be seen from the above, this application first needs to obtain multi-view images and training camera parameters corresponding to video images in a 3D Gaussian sputtering scene to construct a dense 3D Gaussian scene representation; secondly, it determines the budget of retained points after pruning and calculates the structural importance score of each Gaussian primitive to construct a structural reference set; thirdly, it determines the error quality weight based on the rendering error of the training viewpoint; then, it generates an ordered residual cover candidate set; based on the error quality distribution and spatial distribution of the structural reference set and the residual cover candidate set, it determines the layout features and calculates the skeleton budget ratio using a closed formula; finally, based on this ratio, it constructs a skeleton set and a fill set, prunes the dense scene, and after attribute baking and lock point fine-tuning, generates a compressed 3D Gaussian scene for rendering and video encoding to improve the compressed rendering quality. Attached Figure Description
[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0022] Figure 1 This is a flowchart of a video image compression method based on three-dimensional Gaussian pruning disclosed in this application; Figure 2 This is a schematic diagram of a video image compression device based on three-dimensional Gaussian pruning disclosed in this application. Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation
[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0024] 3D Gaussian sputtering utilizes a large number of explicit 3D Gaussian primitives to represent the scene, but the number of Gaussians after dense training typically reaches hundreds of thousands to millions. To reduce the number of points, one important approach is to prune the Gaussians based on importance scores, and then perform short-term fine-tuning on the retained subset to restore image quality. Existing technologies mainly include three categories. The first category is Topk pruning based on a single importance score; the second category is residual or error-driven resampling / reweighting; and the third category is heuristic hybrid retention or post-processing. To this end, this application provides a video image compression method based on 3D Gaussian pruning, which can improve the compressed rendering quality during the 3D Gaussian pruning video image compression process.
[0025] See Figure 1 As shown, this embodiment of the invention discloses a video image compression method based on three-dimensional Gaussian pruning, including: Step S11: Construct a dense 3D Gaussian scene representation based on several multi-view images corresponding to the video images in the 3D Gaussian sputtering scene and the training camera parameters; the dense 3D Gaussian scene representation includes several Gaussian primitives.
[0026] In this embodiment, the overall process corresponding to the embodiment of this application is shown in Table 1: Table 1. Overall Process Information for Video Image Compression Based on 3D Gaussian Pruning
[0027] In one specific implementation, the pruning ratio of this application embodiment is set as follows: Alternatively, the actual number of points retained using the baseline method can be directly taken to ensure consistency. Comparison; Maximum number of training views in the error field Voxel mesh generation ; Number of neighboring bakeries Fine-tuning iteration Upper and lower bounds of skeleton proportions To prevent degeneration into pure residual candidates or pure Closed-loop scale parameters .
[0028] In this embodiment, the present application embodiment first needs to obtain a dense scene representation obtained by training multi-view images with 3DGaussian Splatting (3DGS).
[0029] In one specific implementation, suppose the scenario includes... The Gaussian; the first A Gaussian must include at least: center location, covariance (or scale and rotation), and opacity. Appearance attributes such as spherical harmonics and color. Simultaneously obtain the training camera set and the test camera set. Given a pruning ratio. Calculate the budget for the number of retained points: ; in, Budget the number of points that can be retained after pruning (retained points). This represents the total number of original dense 3D Gaussian primitives in the scene. `r` is the pruning ratio (e.g., 0.70, 0.80, 0.99, etc.). `round()` is a rounding function used to calculate integers. Alternatively, the actual number of points retained by the baseline pruning method (such as PUP) can be directly read as... It is worth mentioning that the Gaussian side is not deleted in this embodiment of the application.
[0030] Furthermore, in this embodiment, the structural importance score of each Gaussian needs to be calculated from the training perspective according to the existing Fisher information class process. Alternatively, pre-calculated scores can be loaded directly. This invention is not limited to this. The specific estimator implementation only requires using the same set of scores for the same scene. Let the pure Topk baseline hold-out be: ; in, This is a pure Top-k baseline hold-out set (comparison reference set). To take the front The indexing operation with the highest score. s is the structural importance score of each Gaussian primitive. N is the budget for the number of points to retain.
[0031] It is worth mentioning that the above set is used for reference when calculating observables in subsequent calculations.
[0032] Specifically, the construction of a dense 3D Gaussian scene representation based on several multi-view images corresponding to video images in a 3D Gaussian sputtering scene and training camera parameters may include: acquiring several multi-view images corresponding to video images in a 3D Gaussian sputtering scene and training camera parameters corresponding to each of the multi-view images, and constructing a dense 3D Gaussian scene representation based on the multi-view images and the training camera parameters; wherein, the dense 3D Gaussian scene representation includes several Gaussian primitives; each Gaussian primitive includes a center position, a geometric attribute for characterizing spatial morphology, an opacity attribute, and an appearance attribute.
[0033] Step S12: Determine the budget of the number of points to be retained after pruning, and determine the structural importance score of each Gaussian primitive based on the dense 3D Gaussian scene representation and the training camera parameters, and construct a structural reference set based on each structural importance score and the budget of the number of points to be retained.
[0034] In this embodiment, the present application requires selecting at most [number] cameras from the training cameras. Each perspective is denoted as a set. Then, farthest point sampling (FPS) is performed according to the camera pose to disperse the selected viewpoints as much as possible in space.
[0035] Furthermore, for each selected perspective Render the image using a dense model and calculate the pixel-by-pixel L1 error map with the ground truth image from that viewpoint. The first Each Gaussian center is projected onto a pixel plane at each selected viewpoint. If the projection falls within the image, the L1 error at that pixel is accumulated into the Gaussian center, and a hit is counted. The accumulated error... The number of hits is obtained by summing the L1 errors of the Gaussian center projection pixels at each selected viewpoint. Obtained by counting the number of effective projections. Local errors are averaged. ; in, For the first The local error value (average pixel error) of a Gaussian primitive. This represents the sum of pixel L1 errors accumulated at the selected viewpoint. The number of valid projected pixel hits (at least 1 to prevent division by zero). To obtain the maximum of the two values, the denominator must be valid.
[0036] Error quality weights are defined in conjunction with opacity: ; in, For the first Error quality weights for Gaussian primitives, For the first The opacity property of a Gaussian primitive.
[0037] It is worth mentioning that the meaning of error quality weight is that Gaussians with visible contributions and large reconstruction errors are given higher layout weights.
[0038] Specifically, determining the budgeted number of retainable points after pruning, and determining the structural importance score of each Gaussian primitive based on the dense 3D Gaussian scene representation and the training camera parameters, and constructing a structural reference set based on each structural importance score and the budgeted number of retainable points, may include: constructing a training camera set and a dense Gaussian set including each Gaussian primitive and attribute information based on the dense 3D Gaussian scene representation; processing the total number of Gaussian primitives in the dense Gaussian set based on a preset pruning ratio and a preset target retention number to obtain the budgeted number of retainable points after pruning. The number of Gaussian primitives retained is set as the retention point budget. The structural contribution of each Gaussian primitive in the 3D Gaussian sputtering scene is evaluated based on the dense Gaussian set and the training camera set, resulting in a structural importance score corresponding to each Gaussian primitive. The Gaussian primitives are then sorted based on their structural importance scores to obtain a sorting result. Finally, based on the retention point budget, several Gaussian primitives with high structural importance are selected sequentially from the sorting result to construct a structural reference set based on each Gaussian primitive to be processed.
[0039] Step S13: Determine the error quality weight based on the rendering error between the rendered image at each training viewpoint and the corresponding multi-view image, according to the dense 3D Gaussian scene representation.
[0040] In this embodiment, the present application requires, based on the error quality weights and spatial locations of each Gaussian primitive, to adjust the scene bounding box according to... Perform regular voxel partitioning, assigning Gaussian values to the corresponding voxels. Allocate integer quotas based on the sum of the error quality within each voxel, ensuring the total quota equals... The maximum remainder method can be used. Within each voxel, the farthest point of the candidates falling into that voxel is sampled according to weights to select the corresponding quota, resulting in a locally ordered subset. Subsequently, the local subsets are concatenated according to voxel quality or a preset traversal order to form a globally ordered candidate: ; Here, BGT is a residual coverage-oriented proposal list, which is not the same as the final retained set. To guide the selection and sorting of Gaussian primitive indexes based on spatial partitioning and error quality weights, a total of indivual.
[0041] Specifically, determining the error quality weight based on the rendering error between the rendered image and the corresponding multi-view image under each training viewpoint of the dense 3D Gaussian scene representation may include: selecting several training cameras from the training camera set, constructing a training viewpoint set for error evaluation based on each training camera, and then performing image rendering using the training viewpoints in the training viewpoint set and the dense Gaussian set to obtain rendered images corresponding to each training viewpoint; comparing the rendered images with the multi-view images corresponding to the training viewpoints pixel by pixel to obtain a pixel error map; the pixel error map is used to determine the local reconstruction error of each Gaussian primitive under each training viewpoint; and projecting the center positions of each Gaussian primitive in the dense Gaussian set onto the image. The image plane corresponding to each training viewpoint in the training viewpoint set is projected to obtain the effective projection position of each Gaussian primitive in each training viewpoint; the pixel error corresponding to the effective projection position is read from the pixel error map, and the pixel error of the Gaussian primitive under each training viewpoint is accumulated and normalized to obtain a local error value; the local error value is used to characterize the degree of reconstruction error of the Gaussian primitive in the image plane region; the local error value corresponding to each Gaussian primitive and the corresponding opacity attribute are jointly calculated to obtain the corresponding error quality weight; the data size of the error quality weight is positively correlated with the data size of the local error; the data size of the error quality weight is positively correlated with the degree of opacity.
[0042] Step S14: Based on the error quality weights and spatial locations of each Gaussian primitive, determine several candidate Gaussian primitives, and sort each candidate Gaussian primitive to obtain an ordered residual cover candidate set.
[0043] In this embodiment, the application requires determining layout features based on the error quality distribution and spatial distribution corresponding to the structural reference set and the ordered residual coverage candidate set, respectively, and determining the skeleton budget ratio based on the layout features. Define the set error quality: ; Where W(S) is the sum of error quality of Gaussian primitives in set S, and S is the set of Gaussian primitives (such as the structural reference set or residual candidate set).
[0044] And record , .
[0045] Then, the relative surplus and saturation surplus are calculated: ; Where rel is the relative residual quality surplus (representing the gain of BGT relative to PUP in residual quality). It is the sum of error quality of the BGT set (ordered residual cover candidate set). This represents the sum of error quality for the structural reference set. For numerical stability, a small amount (can be taken) ), The residual quality surplus after saturation is used. Approximately 10% of the Gaussians with the highest error quality are used to form the hot zone (Hot90). The leakage ratio and quality coverage ratio of the hot zone are calculated: ; in, It is the set of approximately 10% of Gaussian primitives with the highest error quality weight (residual hot zone). denoted as the hot zone leakage ratio, denoted as the ratio of the number of hot zone leaks to the total number, and c is the hot zone quality coverage ratio, which is the proportion of Gaussian primitive quality belonging to the reference set (PUP) in the hot zone to the total quality of the hot zone. ( ) is the sum of error quality functions. This is a pure Top-k baseline hold-out set.
[0046] Then, according to weight Sampling query points, calculated respectively to Average nearest neighbor distance to BGT , Define spatial coverage difference: ; in, This indicates that BGT is spatially closer to the error quality, and g is the spatial coverage difference (reflecting whether BGT is spatially closer to the region with a high error quality weight). The mean nearest neighbor distance from the query point to the nearest Gaussian primitive in the structured reference set (PUP). The mean nearest neighbor distance from the query point to the nearest Gaussian primitive in the ordered residual cover candidate set (BGT).
[0047] make If it is the Sigmoid function, then: ; ; ; ; ; ; ; Where, σ( ) represents the Sigmoid function. c represents the thermal zone mass coverage ratio. Set the alignment confidence threshold (default 0.12). Temperature parameter for aligning confidence level (default 0.05). Alignment confidence (between 0 and 1, representing the degree of alignment between the structural reference set and the residual candidate set in terms of hot zone coverage). This is the threshold for external leakage of hot zones (default 0.75). This is the temperature parameter leaking from the hot zone (default 0.20). The external leakage strength of the hot zone (trimmed to [0,1]). This is for residual quality surplus. To align the quality factor (primarily composed of confidence level) And the surplus b is determined and subject to leakage. adjust). This is the cold start threshold (default 1.5). This is a cold start factor (allowing for a small amount of residual fill when the alignment confidence is low, to prevent complete non-filling). The quality allocation factor is the sum of the alignment factor and the cold start factor. Poor spatial coverage. This represents the positive portion of the spatial coverage difference. This is the parameter for adjusting spatial coverage (default 0.20). This is a spatial coverage saturation term. For the final spatial coverage adjustment term (subject to) and (Common influence). This is the budget percentage for the skeleton (the final calculated value, the share of the budget allocated to the structural skeleton). This is the lower bound of the skeleton proportion (default 0.25). This is the upper bound of the skeleton ratio (default 0.85), used to prevent degeneration into pure residual candidates or pure Top-k.
[0048] Specifically, the step of determining several candidate Gaussian primitives based on their error quality weights and spatial positions, and sorting these candidate Gaussian primitives to obtain an ordered residual coverage candidate set, may include: spatially partitioning the 3D space corresponding to the dense 3D Gaussian scene representation based on the center positions of the Gaussian primitives in the dense Gaussian set, to obtain 3D spatial partitions; summarizing the error quality weights of the Gaussian primitives within each spatial partition to determine the residual quality corresponding to each spatial partition, and determining corresponding candidate slots based on the residual quality of each spatial partition; determining candidate Gaussian primitives for each candidate slot within each spatial partition based on the error quality weights and the spatial positional relationships between the Gaussian primitives, and sorting and concatenating the residual qualities based on the selection order of the candidate Gaussian primitives to obtain an ordered residual coverage candidate set; the ordered residual coverage candidate set is used to evaluate residual coverage capability.
[0049] Furthermore, the default scale is shared across scenarios in the embodiments of this application: Semantically speaking: To align confidence, For residual quality surplus, For the external leakage intensity of the hot zone, To prevent no padding at all during weak alignment, This is a spatial coverage adjustment term. Upper and lower bounds. Used to prevent degradation into extreme scenarios.
[0050] Step S15: Determine the layout features based on the error quality distribution and spatial distribution corresponding to the structural reference set and the ordered residual coverage candidate set, respectively, and determine the skeleton budget ratio based on the layout features.
[0051] In this embodiment, the number of points to be retained is determined based on the skeleton budget ratio and the retention point budget. Let: ; in, This is the first reserved quantity (used to construct the structural skeleton set). q represents the second retention quantity (used to construct the residual filling set). q is the skeleton budget ratio calculated above. To retain points budget.
[0052] Subsequently, the skeleton set and the padding set are defined as follows: ; ; ; in, For the set of structural skeletons (based on structural importance score) Selected (Gauss). Fill the residual set (from the ordered residual cover candidate set) In the middle, the deduplicated skeleton set Then, select in sequence. (Gauss). The final target set is reserved (consisting of the union of the skeleton set and the fill set, with a size equal to...). ). Indicates by list The indices that have not yet been selected are taken in sequence until the set is full. If BGT has fewer duplicates after deduplication... Then according to Fill in the remaining Gaussian parts to ensure... .
[0053] Subsequently, this embodiment of the application needs to extract the set of retained point locations, and for each retained point, search in the dense model. Find the spatial nearest neighbors (which can be achieved using brute-force distance, Inverted File (IVF) / Faiss, etc., for approximate nearest neighbors). Aggregate the scale, rotation, opacity, spherical harmonic coefficients, and other attributes of the neighborhood by distance or kernel weight, write them into Gaussian preserved models, and obtain the baked model. This step does not change the number of preserved points.
[0054] Specifically, determining the layout features based on the error quality distribution and spatial distribution corresponding to the structural reference set and the ordered residual coverage candidate set, respectively, and determining the skeleton budget ratio based on the layout features, may include: summarizing the error quality of Gaussian primitives in the structural reference set based on the error quality weights of each Gaussian primitive to obtain a first error quality characterization; summarizing the error quality of the Gaussian primitives in the ordered residual coverage candidate set based on the number of retained points to obtain a second error quality characterization; determining an error quality difference feature based on the difference between the first error quality characterization and the second error quality characterization; the error quality difference feature is used to characterize the ordered residual coverage candidate set relative to the structural reference set in satisfying the pre-defined error quality distribution and spatial distribution. Assume a gain in covering high-error regions under high-error conditions; based on the error quality weights of each Gaussian primitive, determine Gaussian primitives to be processed from the dense Gaussian set whose error quality weights are higher than the preset high-weight condition, and construct a residual hotspot set based on each Gaussian primitive to be processed; determine the coverage of the residual hotspot set by the structural reference set and the ordered residual coverage candidate set, respectively, to obtain the corresponding first coverage and second coverage, and determine the residual hotspot coverage difference feature based on the difference between the first coverage and the second coverage; the residual hotspot coverage difference feature is used to reflect the degree of omission of the high-error regions by the structural reference set and the compensation ability of the ordered residual coverage candidate set for the omission regions.
[0055] Subsequently, this embodiment of the application needs to determine each spatial query position from the dense Gaussian set based on the error quality weight of each Gaussian primitive, and determine the first distance from each spatial query position to the nearest Gaussian primitive in the structural reference set, and then determine the second distance from each spatial query position to the nearest Gaussian primitive in the ordered residual coverage candidate set; determine the spatial coverage difference feature based on the difference between the first distance and the second distance corresponding to each spatial query position; the spatial coverage difference feature is used to reflect the difference in coverage density of the high error spatial region by the structural reference set and the ordered residual coverage candidate set respectively; jointly calculate the error quality difference feature with the residual hot zone coverage difference feature and / or the spatial coverage difference feature to obtain the skeleton budget ratio; the skeleton budget ratio is used to determine the budget share allocated to the structural skeleton in the reserved point budget.
[0056] Step S16: Determine the number of elements to be retained based on the skeleton budget ratio and the number of retained points budget. Then, prune the dense 3D Gaussian scene representation using the skeleton set and residual fill set selected from the structure reference set based on the structure importance score and the number of elements to be retained, to obtain a compressed 3D Gaussian scene representation. Finally, render and encode the compressed 3D Gaussian scene representation to obtain the compressed video image.
[0057] In this embodiment, the image reconstruction loss needs to be trained: ; Where L1 is the L1 loss function and D-SSIM is the structural similarity index difference loss.
[0058] Next, optimize the Gaussian retention parameters and then lock the number of points: disable operations that change the number of Gaussians, such as density / clone / split / prune, and always keep the number of points constant. Iteration The final model is then obtained; optionally, metrics such as PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity), and LPIPS (Learned Perceptual Image Patch Similarity) can be calculated on the held-out test set. Point cloud / checkpoint files are exported for storage, transmission, or real-time rendering, thereby completing the compression encoding of the video image.
[0059] In this embodiment, the present application selected 13 commonly used 3D Gaussian scenes from the MipNerf360, Tanks & Temples, and DeepBlending datasets for experiments, comparing them with pure Fisher's Gaussian scenes. The baseline (PUP) uses the same number of reserved points. The same nearest neighbor attribute baking and lock point number fine-tuning protocol (fine-tuning 1000 times) only replaces the retainer set construction method.
[0060] The primary metric is the held-out test perspective ID-PSNR. Different pruning ratios... The average ΔPSNR (ASF-PUP) of ASF relative to PUP is shown in Table 2: Table 2. Information table of experimental data corresponding to each pruning ratio.
[0061] As shown in Table 2, at low compression ratios, the embodiments of this application and the pure Compared to a pure TopK model, the average gain is approximately zero, for example, -0.03 dB at 30% pruning and -0.01 dB at 50% pruning. As the pruning ratio increases and the number of retained samples becomes tighter, the average gain relative to pure TopK in this application's embodiments gradually increases: approximately +0.07 dB at 70% pruning, approximately +0.17 dB at 80% pruning, approximately +0.37 dB at 90% pruning, and approximately +0.54 dB at both 95% and 99%. The gain is mainly concentrated in the high compression ratio stage, for example, when the pruning ratio is not less than 80%, especially between 90% and 99%. This trend is consistent with the method's motivation, that is, the tighter the number of samples, the more critical residual coverage and filling become.
[0062] Specifically, the step of determining the number of retained elements based on the skeleton budget ratio and the retention point budget, and pruning the dense 3D Gaussian scene representation using the structural skeleton set and residual fill set selected from the structural reference set based on the structural importance score and the number of retained elements to obtain a compressed 3D Gaussian scene representation, may include: determining a first number of retained elements based on the skeleton budget ratio and the retention point budget, and determining a second number of retained elements based on the difference between the retention point budget and the first number of retained elements; the first number of retained elements is used to construct the structural skeleton set; the second number of retained elements is used to construct the residual fill set; selecting the first number of Gaussian primitives from the structural reference set in descending order of structural importance score based on the structural importance score to obtain the structural skeleton set; sequentially selecting Gaussian primitives that do not exist in the structural skeleton set from the ordered residual cover candidate set to obtain the second number of Gaussian primitives to obtain the residual fill set; wherein, when the number of Gaussian primitives available for filling in the ordered residual cover candidate set is less than the second number of retained elements, based on the structural reference set... Importance scores are used to select Gaussian primitives from the structural reference set that do not exist in the structural skeleton set and the residual fill set, resulting in a residual fill set. The structural skeleton set and the residual fill set are merged to obtain a target retention set, wherein the number of Gaussian primitives in the target retention set is equal to the retention point budget. The retention center position of each retained Gaussian primitive in the target retention set is determined, and the spatially adjacent neighboring Gaussian primitives in the structural reference set are determined using each retention center position. Based on the geometric, opacity, and appearance attributes of each neighboring Gaussian primitive, the attributes of the corresponding retained Gaussian primitives are aggregated and updated to obtain the attribute-transferred target retention set. Using the multi-view image and the corresponding training camera set, while maintaining the number of Gaussian primitives in the attribute-transferred target retention set equal to the retention point budget, the parameters of the Gaussian primitives in the attribute-transferred target retention set are optimized to obtain an optimized retention set. Based on the optimized retention set, the dense 3D Gaussian scene representation is pruned to obtain a compressed 3D Gaussian scene representation.
[0063] As can be seen from the above, the embodiments of this application first need to construct a dense 3D Gaussian scene representation based on several multi-view images corresponding to video images in a 3D Gaussian sputtering scene and training camera parameters; the dense 3D Gaussian scene representation includes several Gaussian primitives; the budget for the number of retention points allowed after pruning is determined, and the structural importance score of each Gaussian primitive is determined based on the dense 3D Gaussian scene representation and the training camera parameters, and a structural reference set is constructed based on each structural importance score and the budget for the number of retention points; the error quality weight is determined based on the rendering error between the rendered images of the dense 3D Gaussian scene representation under each training viewpoint and the corresponding multi-view images; based on the error quality weight and spatial position of each Gaussian primitive, several candidate Gaussians are determined. The primitives are sorted to obtain an ordered residual coverage candidate set. Layout features are determined based on the error quality distribution and spatial distribution corresponding to the structural reference set and the ordered residual coverage candidate set, respectively. The skeleton budget ratio is determined based on the layout features. The number of retained points is determined based on the skeleton budget ratio and the number of retained points. The dense 3D Gaussian scene representation is pruned using the structural skeleton set and residual filling set selected from the structural reference set based on the structural importance score and the number of retained points, resulting in a compressed 3D Gaussian scene representation. The compressed 3D Gaussian scene representation is then rendered and video encoded to obtain the compressed video image, thereby improving the efficiency of video image compression.
[0064] Accordingly, see Figure 2 As shown, this application also provides a video image compression device based on three-dimensional Gaussian pruning, comprising: Gaussian scene representation generation module 11 is used to construct a dense three-dimensional Gaussian scene representation based on several multi-view images corresponding to video images in a three-dimensional Gaussian sputtering scene and training camera parameters; the dense three-dimensional Gaussian scene representation includes several Gaussian primitives. Importance score determination module 12 is used to determine the budget of the number of retention points allowed to be retained after pruning, and to determine the structural importance score of each Gaussian primitive based on the dense 3D Gaussian scene representation and the training camera parameters, and to construct a structural reference set based on each structural importance score and the budget of the number of retention points; Error quality weight generation module 13 is used to determine error quality weights based on the rendering error between the rendered images of the dense 3D Gaussian scene representation under each training view and the corresponding multi-view images. The Gaussian primitive sorting module 14 is used to determine a number of candidate Gaussian primitives based on the error quality weight and spatial position of each Gaussian primitive, and sort each candidate Gaussian primitive to obtain an ordered residual cover candidate set. The layout feature generation module 15 is used to determine the layout features based on the error quality distribution and spatial distribution corresponding to the structure reference set and the ordered residual coverage candidate set, respectively, and to determine the skeleton budget ratio based on the layout features. The video image compression module 16 is used to determine the number of elements to be retained based on the skeleton budget ratio and the number of retained points budget, and to prune the dense 3D Gaussian scene representation using a set of structural skeletons and a set of residual fills selected from the structural reference set based on the structural importance score and the number of elements to be retained, thereby obtaining a compressed 3D Gaussian scene representation. The compressed 3D Gaussian scene representation is then rendered and video encoded to obtain the compressed video image.
[0065] In some specific embodiments, the Gaussian scene representation generation module 11 may specifically include: The image acquisition unit is used to acquire several multi-view images corresponding to video images in a three-dimensional Gaussian sputtering scene and training camera parameters corresponding to each multi-view image. A scene construction unit is used to construct a dense 3D Gaussian scene representation based on the multi-view images and the training camera parameters; wherein, the dense 3D Gaussian scene representation includes several Gaussian primitives; each Gaussian primitive includes a center position, a geometric attribute for characterizing the spatial shape, an opacity attribute, and an appearance attribute.
[0066] In some specific embodiments, the importance score determination module 12 may specifically include: The budget determination unit is used to construct a training camera set and a dense Gaussian set including each Gaussian primitive and attribute information based on the dense 3D Gaussian scene representation; and to process the total number of Gaussian primitives in the dense Gaussian set based on a preset pruning ratio and a preset target retention number to obtain the number of Gaussian primitives allowed to be retained after pruning, so as to set the number of Gaussian primitives as the retention point budget. The score calculation unit is used to evaluate the structural contribution of each Gaussian primitive in the three-dimensional Gaussian sputtering scene based on the dense Gaussian set and the training camera set, and obtain the structural importance score corresponding to each Gaussian primitive. A reference set construction unit is used to sort each Gaussian primitive based on its structural importance score to obtain a sorting result. Then, based on the retention point budget, a number of Gaussian primitives with high structural importance are selected from the sorting result to construct a structural reference set based on each Gaussian primitive to be processed.
[0067] In some specific embodiments, the error quality weight generation module 13 may specifically include: The viewpoint selection unit is used to select several training cameras from the training camera set, and construct a training viewpoint set for error evaluation based on each training camera. Then, it uses the training viewpoints in the training viewpoint set and the dense Gaussian set to perform image rendering to obtain a rendered image corresponding to each training viewpoint. An error calculation unit is used to compare the rendered image with the multi-view image corresponding to the training viewpoint pixel by pixel to obtain a pixel error map; the pixel error map is used to determine the local reconstruction error of each Gaussian primitive under each training viewpoint. The projection positioning unit is used to project the center position of each Gaussian primitive in the dense Gaussian set onto the image plane corresponding to each training view in the training view set, so as to obtain the effective projection position of each Gaussian primitive in each training view. The local error determination unit is used to read the pixel error corresponding to the effective projection position from the pixel error map, and to accumulate and normalize the pixel error of the Gaussian primitive under each training viewpoint to obtain a local error value; the local error value is used to characterize the degree of reconstruction error of the Gaussian primitive in the image plane region; The weight determination unit is used to jointly calculate the local error value and the corresponding opacity attribute corresponding to each Gaussian primitive to obtain the corresponding error quality weight; the data size of the error quality weight is positively correlated with the data size of the local error; the data size of the error quality weight is positively correlated with the degree of opacity.
[0068] In some specific embodiments, the Gaussian primitive sorting module 14 may specifically include: A spatial partitioning unit is used to partition the three-dimensional space corresponding to the dense three-dimensional Gaussian scene representation based on the center position of each Gaussian primitive in the dense Gaussian set, so as to obtain a three-dimensional spatial partition. The quota allocation unit is used to summarize the error quality weights of Gaussian primitives in each spatial partition, determine the residual quality corresponding to each spatial partition, and determine the corresponding candidate quota based on the residual quality of each spatial partition. The candidate selection unit is used to determine the candidate Gaussian primitives for the candidate quota in each of the spatial partitions based on the error quality weights and the spatial positional relationship between the Gaussian primitives, and to sort and concatenate each of the residual qualities based on the selection order of the candidate Gaussian primitives to obtain an ordered residual coverage candidate set; the ordered residual coverage candidate set is used to evaluate the residual coverage capability.
[0069] In some specific embodiments, the layout feature generation module 15 may specifically include: A quality difference determination unit is used to summarize the error quality of Gaussian primitives in the structural reference set based on the error quality weights of each Gaussian primitive to obtain a first error quality characterization, and to summarize the error quality of the estimated number of Gaussian primitives with the number of retained points in the ordered residual coverage candidate set to obtain a second error quality characterization; and to determine an error quality difference feature based on the difference between the first error quality characterization and the second error quality characterization; the error quality difference feature is used to characterize the gain of the ordered residual coverage candidate set relative to the structural reference set in terms of high-error region coverage that meets a preset high-error condition; A hotspot coverage determination unit is used to determine, based on the error quality weights of each Gaussian primitive, Gaussian primitives from the dense Gaussian set that have error quality weights higher than a preset condition, so as to construct a residual hotspot set based on each Gaussian primitive to be processed; to determine the coverage of the residual hotspot set by the structural reference set and the ordered residual coverage candidate set, respectively, to obtain corresponding first coverage and second coverage, and to determine residual hotspot coverage difference features based on the difference between the first coverage and the second coverage; the residual hotspot coverage difference features are used to reflect the degree of omission of the high-error regions by the structural reference set and the compensation capability of the ordered residual coverage candidate set for the omission regions; A spatial coverage determination unit is configured to determine each spatial query position from the dense Gaussian set based on the error quality weights of each Gaussian primitive, and to determine a first distance from each spatial query position to the nearest Gaussian primitive in the structural reference set, and then to determine a second distance from each spatial query position to the nearest Gaussian primitive in the ordered residual coverage candidate set; and to determine a spatial coverage difference feature based on the difference between the first distance and the second distance corresponding to each spatial query position; the spatial coverage difference feature is used to reflect the difference in coverage density of the high-error spatial region by the structural reference set and the ordered residual coverage candidate set, respectively. The proportion determination unit is used to jointly calculate the error quality difference characteristics with the residual hot zone coverage difference characteristics and / or the spatial coverage difference characteristics to obtain the skeleton budget proportion; the skeleton budget proportion is used to determine the budget share allocated to the structural skeleton in the budget for the number of reserved points.
[0070] In some specific embodiments, the video image compression module 16 may specifically include: The retention quantity determination unit is used to determine a first retention quantity based on the skeleton budget ratio and the retention point budget, and to determine a second retention quantity based on the difference between the retention point budget and the first retention quantity; the first retention quantity is used to construct a structural skeleton set; the second retention quantity is used to construct a residual filling set; The skeleton set determination unit is used to select the first number of Gaussian primitives to be retained from the structure reference set based on the structure importance score in descending order of structure importance score, so as to obtain the structure skeleton set; The filling set determination unit is used to sequentially select Gaussian primitives that do not exist in the structural skeleton set from the ordered residual coverage candidate set to obtain the second number of retained Gaussian primitives, thereby obtaining a residual filling set; wherein, when the number of Gaussian primitives that can be used for filling in the ordered residual coverage candidate set is less than the second number of retained primitives, Gaussian primitives that do not exist in the structural skeleton set and the residual filling set are selected from the structural reference set based on the structural importance score to obtain the residual filling set; The target set generation unit is used to merge the structural skeleton set and the residual filling set to obtain the target retention set, wherein the number of Gaussian primitives in the target retention set is equal to the retention point budget; The attribute migration unit is used to determine the retention center position of each retained Gaussian primitive in the target retention set, and to use the retention center position to determine the spatially adjacent neighboring Gaussian primitives from the structural reference set. Based on the geometric properties, opacity properties and appearance properties of each neighboring Gaussian primitive, the attributes of the corresponding retained Gaussian primitives are aggregated and updated to obtain the target retention set after attribute migration. The optimization pruning unit is used to optimize the Gaussian primitive parameters in the target retention set after attribute transfer, using the multi-view images and the corresponding training camera set, while keeping the number of Gaussian primitives in the target retention set after attribute transfer equal to the budgeted number of retention points, to obtain an optimized retention set. Based on the optimized retention set, the dense 3D Gaussian scene representation is pruned to obtain a compressed 3D Gaussian scene representation.
[0071] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the video image compression method based on three-dimensional Gaussian pruning disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0072] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0073] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0074] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the video image compression method based on three-dimensional Gaussian pruning disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.
[0075] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned video image compression method based on three-dimensional Gaussian pruning. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0076] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0077] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0078] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0079] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0080] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A video image compression method based on three-dimensional Gaussian pruning, characterized in that, include: A dense 3D Gaussian scene representation is constructed based on several multi-view images corresponding to video images in a 3D Gaussian sputtering scene and training camera parameters. The dense 3D Gaussian scene representation includes several Gaussian primitives; Determine the budget for the number of retention points allowed after pruning, and determine the structural importance score of each Gaussian primitive based on the dense 3D Gaussian scene representation and the training camera parameters, and construct a structural reference set based on each structural importance score and the budget for the number of retention points; Based on the rendering error between the rendered images at each training viewpoint and the corresponding multi-view images of the dense 3D Gaussian scene representation, the error quality weight is determined; Based on the error quality weights and spatial locations of each Gaussian primitive, several candidate Gaussian primitives are determined, and the candidate Gaussian primitives are sorted to obtain an ordered residual cover candidate set. The layout features are determined based on the error quality distribution and spatial distribution corresponding to the structural reference set and the ordered residual coverage candidate set, respectively, and the skeleton budget ratio is determined based on the layout features. The number of elements to be retained is determined based on the skeleton budget ratio and the number of retained points budget. The dense 3D Gaussian scene representation is pruned using the skeleton set and residual fill set selected from the structural reference set based on the structural importance score and the number of elements to be retained, resulting in a compressed 3D Gaussian scene representation. The compressed 3D Gaussian scene representation is then rendered and video encoded to obtain the compressed video image.
2. The video image compression method based on three-dimensional Gaussian pruning according to claim 1, characterized in that, The construction of a dense 3D Gaussian scene representation based on several multi-view images corresponding to video images in a 3D Gaussian sputtering scene and trained camera parameters includes: Acquire several multi-view images corresponding to video images in a 3D Gaussian sputtering scene and training camera parameters corresponding to each multi-view image, and construct a dense 3D Gaussian scene representation based on the multi-view images and the training camera parameters; The dense 3D Gaussian scene representation includes several Gaussian primitives; each Gaussian primitive includes a center position, a geometric attribute used to characterize the spatial shape, an opacity attribute, and an appearance attribute.
3. The video image compression method based on three-dimensional Gaussian pruning according to claim 1, characterized in that, The process of determining the budgeted number of retention points allowed after pruning, determining the structural importance score of each Gaussian primitive based on the dense 3D Gaussian scene representation and the training camera parameters, and constructing a structural reference set based on each structural importance score and the budgeted number of retention points includes: Based on the dense 3D Gaussian scene representation, a training camera set and a dense Gaussian set including each Gaussian primitive and attribute information are constructed. The total number of Gaussian primitives in the dense Gaussian set is processed based on a preset pruning ratio and a preset target retention number to obtain the number of Gaussian primitives that can be retained after pruning, and the number of Gaussian primitives is set as the retention point budget. The structural contribution of each Gaussian primitive in the 3D Gaussian sputtering scene is evaluated based on the dense Gaussian set and the training camera set, and a structural importance score corresponding to each Gaussian primitive is obtained. The Gaussian primitives are sorted based on their structural importance scores to obtain a sorting result. Then, based on the retention point budget, a number of Gaussian primitives with high structural importance are selected from the sorting result to construct a structural reference set based on each of the Gaussian primitives to be processed.
4. The video image compression method based on three-dimensional Gaussian pruning according to claim 3, characterized in that, The determination of error quality weights based on the rendering error between the rendered images at each training viewpoint and the corresponding multi-view images based on the dense 3D Gaussian scene representation includes: Several training cameras are selected from the training camera set, and a training viewpoint set for error evaluation is constructed based on each training camera. Then, the training viewpoints in the training viewpoint set and the dense Gaussian set are used for image rendering to obtain the rendered images corresponding to each training viewpoint. The rendered image is compared pixel-by-pixel with the multi-view image corresponding to the training viewpoint to obtain a pixel error map; the pixel error map is used to determine the local reconstruction error of each Gaussian primitive under each training viewpoint. The center positions of each Gaussian primitive in the dense Gaussian set are projected onto the image planes corresponding to each training viewpoint in the training viewpoint set, thereby obtaining the effective projection positions of each Gaussian primitive in each training viewpoint. The pixel error corresponding to the effective projection position is read from the pixel error map, and the pixel error of the Gaussian primitive under each training viewpoint is accumulated and normalized to obtain the local error value; the local error value is used to characterize the degree of reconstruction error of the Gaussian primitive in the image plane region; The local error value and the corresponding opacity attribute corresponding to each Gaussian primitive are jointly calculated to obtain the corresponding error quality weight; the data size of the error quality weight is positively correlated with the data size of the local error; the data size of the error quality weight is positively correlated with the degree of opacity.
5. The video image compression method based on three-dimensional Gaussian pruning according to claim 4, characterized in that, Based on the error quality weights and spatial locations of each Gaussian primitive, several candidate Gaussian primitives are determined, and these candidate Gaussian primitives are sorted to obtain an ordered residual cover candidate set, including: Based on the center position of each Gaussian primitive in the dense Gaussian set, the three-dimensional space corresponding to the dense three-dimensional Gaussian scene representation is spatially partitioned to obtain a three-dimensional spatial partition. The error quality weights of the Gaussian primitives in each spatial partition are summarized to determine the residual quality corresponding to each spatial partition, and the corresponding candidate quota is determined based on the residual quality of each spatial partition. In each of the spatial partitions, candidate Gaussian primitives for the candidate slots are determined based on the error quality weights and the spatial positional relationships between the Gaussian primitives. The residual quality is then sorted and concatenated based on the selection order of the candidate Gaussian primitives to obtain an ordered residual coverage candidate set. The ordered residual coverage candidate set is used to evaluate the residual coverage capability.
6. The video image compression method based on three-dimensional Gaussian pruning according to claim 5, characterized in that, The step of determining layout features based on the error quality distribution and spatial distribution corresponding to the structural reference set and the ordered residual coverage candidate set, respectively, and determining the skeleton budget ratio based on the layout features, includes: Based on the error quality weight of each Gaussian primitive, the error quality of the Gaussian primitives in the structural reference set is summarized to obtain a first error quality characterization, and the error quality of the estimated number of retained points of the Gaussian primitives in the ordered residual cover candidate set is summarized to obtain a second error quality characterization. Error quality difference features are determined based on the difference between the first error quality characterization and the second error quality characterization; the error quality difference features are used to characterize the gain of the ordered residual coverage candidate set relative to the structural reference set in terms of high error region coverage that satisfies the preset high error condition. Based on the error quality weight of each Gaussian primitive, Gaussian primitives with error quality weights higher than the preset high weight condition are determined from the dense Gaussian set, so as to construct a set of residual hot zones based on each Gaussian primitive. The coverage of the structural reference set and the ordered residual coverage candidate set over the residual hot zone set is determined respectively, resulting in a first coverage situation and a second coverage situation respectively. The residual hot zone coverage difference feature is determined based on the difference between the first coverage situation and the second coverage situation. The residual hot zone coverage difference feature is used to reflect the degree of omission of the high error region by the structural reference set and the compensation ability of the ordered residual coverage candidate set for the omission region. Based on the error quality weights of each Gaussian primitive, spatial query positions are determined from the dense Gaussian set, and a first distance is determined from each spatial query position to the nearest Gaussian primitive in the structural reference set. Then, a second distance is determined from each spatial query position to the nearest Gaussian primitive in the ordered residual cover candidate set. Spatial coverage difference features are determined based on the difference between the first distance and the second distance corresponding to each of the spatial query locations; the spatial coverage difference features are used to reflect the difference in coverage density of the high-error spatial region by the structural reference set and the ordered residual coverage candidate set, respectively. The error quality difference feature is jointly calculated with the residual hot zone coverage difference feature and / or the spatial coverage difference feature to obtain the skeleton budget ratio; the skeleton budget ratio is used to determine the budget share allocated to the structural skeleton in the budget for the number of reserved points.
7. The video image compression method based on three-dimensional Gaussian pruning according to any one of claims 1 to 6, characterized in that, The process involves determining the number of elements to retain based on the skeleton budget ratio and the number of retained points budget, and then pruning the dense 3D Gaussian scene representation using a set of structural skeletons and a set of residual fills selected from the structural reference set based on the structural importance score and the number of elements to retain, to obtain a compressed 3D Gaussian scene representation, including: A first retention quantity is determined based on the skeleton budget ratio and the retention point budget, and a second retention quantity is determined based on the difference between the retention point budget and the first retention quantity; the first retention quantity is used to construct the structural skeleton set; the second retention quantity is used to construct the residual filling set. Based on the structural importance score, the first number of Gaussian primitives are selected from the structural reference set in descending order of structural importance score to obtain the structural skeleton set; From the ordered residual cover candidate set, Gaussian primitives that do not exist in the structural skeleton set are selected sequentially to obtain the second number of retained Gaussian primitives, and thus the residual filling set is obtained. Wherein, when the number of Gaussian primitives available for filling in the ordered residual cover candidate set is less than the second retention number, Gaussian primitives that do not exist in the structural skeleton set and the residual fill set are selected from the structural reference set based on the structural importance score to obtain the residual fill set; The structural skeleton set and the residual filling set are merged to obtain the target retention set, wherein the number of Gaussian primitives in the target retention set is equal to the retention point budget; The retention center position of each Gaussian primitive in the target retention set is determined, and the spatially adjacent Gaussian primitives in the structural reference set are determined using the retention center positions. Based on the geometric, opacity and appearance properties of each adjacent Gaussian primitive, the properties of the corresponding Gaussian primitives are aggregated and updated to obtain the target retention set after attribute migration. Using the multi-view images and the corresponding training camera set, while keeping the number of Gaussian primitives in the target retention set after attribute transfer equal to the budgeted number of retention points, the parameters of the Gaussian primitives in the target retention set after attribute transfer are optimized to obtain an optimized retention set. Based on the optimized retention set, the dense 3D Gaussian scene representation is pruned to obtain a compressed 3D Gaussian scene representation.
8. A video image compression device based on three-dimensional Gaussian pruning, characterized in that, include: A Gaussian scene representation generation module is used to construct a dense 3D Gaussian scene representation based on several multi-view images corresponding to video images in a 3D Gaussian sputtering scene and training camera parameters; the dense 3D Gaussian scene representation includes several Gaussian primitives. The importance score determination module is used to determine the budget of the number of retention points allowed to be retained after pruning, and to determine the structural importance score of each Gaussian primitive based on the dense 3D Gaussian scene representation and the training camera parameters, and to construct a structural reference set based on each structural importance score and the budget of the number of retention points; The error quality weight generation module is used to determine the error quality weight based on the rendering error between the rendered images of the dense 3D Gaussian scene representation under each training view and the corresponding multi-view images. The Gaussian primitive sorting module is used to determine several candidate Gaussian primitives based on the error quality weights and spatial positions of each Gaussian primitive, and sort each candidate Gaussian primitive to obtain an ordered residual cover candidate set. The layout feature generation module is used to determine the layout features based on the error quality distribution and spatial distribution corresponding to the structure reference set and the ordered residual coverage candidate set, respectively, and to determine the skeleton budget ratio based on the layout features. The video image compression module is used to determine the number of elements to be retained based on the skeleton budget ratio and the number of retained points budget, and to prune the dense 3D Gaussian scene representation using a set of structural skeletons and a set of residual fills selected from the structural reference set based on the structural importance score and the number of elements to be retained, thereby obtaining a compressed 3D Gaussian scene representation. The compressed 3D Gaussian scene representation is then rendered and video encoded to obtain the compressed video image.
9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the video image compression method based on three-dimensional Gaussian pruning as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the video image compression method based on three-dimensional Gaussian pruning as described in any one of claims 1 to 7.
