A three-dimensional gaussian sputtering model compression method based on semantic and geometric perception

Abstract

This invention discloses a 3D Gaussian sputtering model compression method based on semantic and geometric awareness, comprising: a semantically aware Gaussian unit importance evaluation mechanism to model the semantic importance of different Gaussian units in a scene and obtain a semantically aware importance score for each Gaussian unit; a geometrically aware Gaussian unit importance evaluation mechanism to obtain the geometric visibility count of each Gaussian unit in all training views and obtain a geometrically aware importance score; obtaining a comprehensive attention score based on the semantically and geometrically aware importance scores of the Gaussian units, and constructing a differentiated compression network for Gaussian unit importance awareness based on this score to obtain a semantically and geometrically aware 3DGS compression model; further fine-tuning the semantically and geometrically aware 3DGS compression model to output a compressed 3DGS representation model, and applying the compressed 3DGS representation model to a scene in 3D reconstruction of a museum or synthesis of a sports event scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision, 3D reconstruction and volumetric video processing, and in particular to a 3D Gaussian sputtering model compression method based on semantic and geometric awareness. Background Technology

[0002] With the rapid development of technologies such as virtual reality and augmented reality, volumetric video has gradually become an important form of 3D visual content expression. In recent years, the 3D Gaussian Splatting (3DGS) method, with its explicit Gaussian primitive representation and efficient rasterization rendering mechanism, has achieved a good balance between rendering quality and rendering efficiency, and has become an important research direction in the fields of 3D scene representation and new view synthesis. However, 3DGS usually relies on a large number of Gaussian primitives to model 3D scenes, resulting in a significant increase in the storage and transmission overhead of 3DGS-based scene representation models. Therefore, exploring efficient 3DGS compression methods is of great research significance.

[0003] To reduce the storage and transmission overhead of 3DGS representation models, researchers have proposed a series of 3DGS compression methods and achieved some progress. However, existing methods typically employ a uniform compression strategy for all regions in a scene, lacking differentiation based on the semantic importance of different regions. In practical applications, users usually pay more attention to the foreground subject region and its detailed information, while showing less concern for the background region.

[0004] Therefore, effectively preserving the reconstruction details of key main areas under limited storage overhead is the key to improving 3DGS compression performance. Summary of the Invention

[0005] To address the issue that existing 3DGS compression methods fail to adequately consider the differences in importance between different regions, leading to a decrease in the reconstruction quality of key subject regions in compressed 3D scenes, this invention proposes a 3D Gaussian sputtering model compression method based on semantic and geometric awareness. This invention constructs a semantically and geometrically aware Gaussian meta-importance evaluation mechanism by modeling Gaussian meta-semantic and geometrical visibility information, thereby achieving differentiated compression of key subject regions and background regions in 3D scenes. This method can significantly improve the detail reconstruction capability and view rendering quality of key subject regions in 3D scenes while maintaining the overall compression ratio, thus improving the compression performance of 3DGS representation models. Details are described below:

[0006] A semantically and geometrically aware method for compressing a 3D Gaussian sputtering model, the method comprising:

[0007] A 3DGS compression network is trained using multi-view images to obtain the original 3DGS representation model;

[0008] Based on the semantically aware Gaussian importance assessment mechanism, the semantic importance of different Gaussian elements in the scene is modeled to obtain the semantically aware importance score of each Gaussian element.

[0009] Based on the geometry-aware Gaussian importance evaluation mechanism, the geometric visibility number of each Gaussian element in all training views is obtained, and a geometry-aware importance score is obtained.

[0010] Based on the semantic and geometric importance scores of Gaussian units, a comprehensive attention score is obtained by combining the corresponding weight coefficients. Based on this score, a differentiated compression network for Gaussian unit importance perception is constructed to obtain a 3DGS compression model for semantic and geometric perception.

[0011] The semantic and geometric perception 3DGS compression model is further fine-tuned to output a compressed 3DGS representation model, which is then applied to scenes such as 3D reconstruction of museums or compositing of sports events.

[0012] The semantically aware Gaussian meta-importance evaluation mechanism is as follows:

[0013] The automatic mask generator in the pre-trained semantic segmentation SAM ViT-B model is used to perform forward reasoning for each view to obtain candidate mask maps representing different target regions in the scene.

[0014] A semantic label allocation strategy based on area priority is adopted, in which the semantic labels of the mask regions with smaller areas are used as the semantic labels of the overlapping regions, and each pixel is assigned a corresponding integer semantic label.

[0015] For each visible Gaussian element in the scene, based on the camera projection relationship of the current viewpoint, all training views are traversed, the visibility frequency vector of each Gaussian element under each semantic category stored in the semantic mask directory is counted, and its semantic label is determined by taking the semantic label with the maximum frequency.

[0016] This paper utilizes the Florence-2 graph-based text model to generate natural language semantic descriptions from candidate mask images in the semantic mask catalog generated by the SAM ViT-B semantic segmentation model. Furthermore, it combines a custom keyword-semantic importance mapping table to identify typical keywords from the language descriptions generated by the Florence-2 graph-based text model. Different keywords correspond to different semantic importance scores. Based on the correspondence set in the mapping table, the semantic importance score of each Gaussian unit is obtained. ;

[0017] When the number of pixels in a semantic label region is less than the threshold or the output of the Florence-2 graph-based text model is empty (i.e., the call fails), the semantic awareness importance score of the corresponding Gaussian unit is calculated based on the area size of each semantic category as a supplement.

[0018] The geometry-aware Gaussian meta-importance evaluation mechanism is as follows:

[0019] The input is the attributes of each Gaussian element and the camera projection matrix for each viewpoint; the output is the geometric visibility count for each Gaussian element.

[0020] For any Gaussian element via camera projection matrix center Project to the Image planes from multiple perspectives yield pixel coordinates. If a Gaussian element satisfies the geometric visibility condition after projection, it is determined that the Gaussian element is visible in the current viewpoint, and the geometric visibility count of the Gaussian element is obtained by counting all training viewpoints.

[0021] The 3DGS compression model for semantic and geometric perception is as follows:

[0022] Calculate the overall attention score for each Gaussian unit, and set the total target number of compressed Gaussian units;

[0023] Based on comprehensive attention score Sort all Gaussian elements and divide them into groups. There are several levels, and the set of all Gaussian elements within each level is called the Gaussian set for that level. Different retention rates are assigned to different levels. After determining the target number of Gaussian elements to be retained for each level, budget normalization is performed.

[0024] If the sum of the number of Gaussian elements for each level of target exceeds If the size is insufficient, the size is proportionally reduced; if it is insufficient, the size is proportionally enlarged. The Gaussian sub-models in each level are divided into K-dimensional tree spaces. All Gaussian elements in the local space after the division are called the local Gaussian element set. The aggregated and compressed local Gaussian element sets in each level are integrated to obtain the aggregated and compressed Gaussian sets of each level.

[0025] The aggregated and compressed Gaussian sets of each level are then merged again to output a semantically and geometrically aware 3DGS compressed model:

[0026]

[0027] in, This represents the complete Gaussian set after merging the aggregated and compressed Gaussian sets of each level. Indicates the number of levels. Indicates the first The Gaussian set after level aggregation and compression.

[0028] The beneficial effects of the technical solution provided by this invention are:

[0029] 1. This method constructs a Gaussian importance assessment mechanism based on semantic and geometric perception by modeling the semantic and geometric visibility information of Gaussian units, thereby achieving differentiated compression of key subject areas and background areas in the scene and improving the compression performance of 3DGS.

[0030] 2. This invention designs a semantically aware Gaussian importance evaluation mechanism. This mechanism makes full use of Gaussian semantic information, effectively improves the semantic expression priority of small targets and key regions, and provides fine-grained semantic basis for obtaining the comprehensive importance score.

[0031] 3. This invention designs a Gaussian primitive importance evaluation mechanism based on geometric visibility. This mechanism effectively identifies Gaussian primitives that are frequently visible in multiple views and contribute more to the overall rendering by statistically analyzing the number of times different Gaussian primitives are geometrically visible in all training views, thus providing a reliable geometric basis for obtaining the comprehensive importance score and resource allocation in the future.

[0032] 4. This invention constructs a differentiated compression model based on Gaussian importance perception. This model combines the semantic importance score of each Gaussian unit with the number of geometric visibilitys to obtain the comprehensive importance score of the Gaussian unit, thereby achieving differentiated compression of key subject areas and background areas in 3D scenes. This significantly improves the detail reconstruction capability and view rendering quality of key subject areas in 3D scenes, thereby improving the compression performance of the 3DGS representation model.

[0033] 5. This invention constructs a 3DGS model compression method based on semantic and geometric awareness. This method can significantly improve the synthesis quality of key subject areas while ensuring the compression rate and the overall image rendering quality, and provide better reconstruction results in minute details. Attached Figure Description

[0034] Figure 1 A flowchart of a semantically and geometrically aware compression method for a 3D Gaussian sputtering model;

[0035] Figure 2 This is a graph of experimental data. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0037] I. Constructing a 3DGS compression network based on semantic and geometry awareness

[0038] The 3DGS compression network proposed in this invention takes multi-view images as input and outputs a compressed 3DGS representation model. Specifically, the 3DGS compression network is first trained using multi-view images to obtain the original 3DGS representation model. Then, a semantically aware Gaussian unit importance evaluation mechanism is used to model the semantic importance of different Gaussian units in the scene, obtaining a semantically aware importance score for each Gaussian unit. Next, a geometrically aware Gaussian unit importance evaluation mechanism is used to obtain the geometric visibility count of each Gaussian unit across all training views, thus obtaining a geometrically aware importance score. Then, based on the semantically and geometrically aware importance scores of the Gaussian units, a differentiated compression network for Gaussian unit importance perception is constructed using corresponding weight coefficients, obtaining semantically and geometrically aware 3DGS compression models. Finally, the semantically and geometrically aware 3DGS compression models are further fine-tuned to output the compressed 3DGS representation model.

[0039] The Gaussian importance assessment mechanisms include: a semantically aware Gaussian importance assessment mechanism and a geometrically aware Gaussian importance assessment mechanism.

[0040] II. Constructing a semantically aware Gaussian meta-importance evaluation mechanism

[0041] The constructed semantically aware Gaussian importance evaluation mechanism takes multi-view images and the original 3DGS representation model as input, models the importance of different Gaussian elements in the scene, and outputs the semantically aware importance score of each Gaussian element.

[0042] Specifically, the automatic mask generator in the pre-trained semantic segmentation SAM ViT-B model is first used to perform forward reasoning for each view to obtain candidate mask maps representing different target regions in the scene.

[0043]

[0044] in, Indicates the first Input images from multiple perspectives This represents the set of candidate mask images to be output. This represents a pre-trained semantic segmentation model, SAM ViT-B. The automatic mask generator in this model consists of three functions: an image encoder, a cue encoder, and a mask decoder. The image encoder divides the input image into several segments. The image is divided into pixel-sized patches, and each patch is then converted into a 256-dimensional feature vector. The image is then divided into 16 equal parts (width and height) and fed into the cue encoder. The cue encoder maps the two-dimensional coordinates of the grid points where the horizontal and vertical lines intersect into 256-dimensional vectors using random Fourier position encoding. The mask decoder, based on the results from the image editor and the cue encoder, obtains the nth... The set of candidate mask images corresponding to each view Subsequently, the candidate mask image sets corresponding to all viewpoints were sorted according to the target region area, and combined with the background region not covered by any mask, they were collectively divided into... A semantic category, Indicates the number of semantic categories (default) =10).

[0045] Specifically, for overlapping regions between different masks, this embodiment of the invention employs a semantic label allocation strategy based on area priority. That is, it prioritizes the semantic labels of mask regions with smaller areas as the semantic labels for overlapping regions, thereby enhancing the priority and retention capability of small target objects in semantic modeling. Ultimately, each pixel is assigned a corresponding integer semantic label:

[0046]

[0047] in, Indicates the number of semantic categories. This represents a semantic label for the background area that is not covered by any mask. The semantic label representing the largest mask region. The semantic label represents the mask region with the smallest area. To preserve the semantic segmentation results, a semantic mask catalog was constructed to store subsequent 3D semantic mapping and semantic importance scores.

[0048] Subsequently, for each visible Gaussian element in the scene, based on the camera projection relationship of the current viewpoint, by traversing all training views, the visibility frequency vector of each Gaussian element under each semantic category stored in the semantic mask directory is counted. And its semantic tag is determined by taking the semantic tag with the highest frequency:

[0049]

[0050] in, Indicates the first Gao Siyuan Predicted as the first in all perspectives Number of class semantics.

[0051] Ultimately, each Gaussian element Each corresponds to a semantic tag Subsequently, the Florence-2 graph-to-text model was used to generate natural language semantic descriptions from candidate mask images in the semantic mask catalog generated by the semantic segmentation SAM ViT-B model. Florence-2 is a visual language generation model that first understands the visual content in an image and then determines the output based on task prompts.

[0052] The Florence-2 model first uses a visual encoder to extract features from the image, obtaining encoded semantic information. Then, the language generation module receives both visual features and task prompts. In this project, the task prompt is set to require the model to describe a specified region. Starting with the first text, the Florence-2 model repeatedly predicts the next most suitable text. Each time new text is generated, it integrates previously generated text, image visual features, and task requirements until a complete description or an end marker is generated.

[0053] The Florence-2 graph-based text model utilizes a custom keyword-semantic importance mapping table. This table identifies typical keywords from the language descriptions generated by the Florence-2 model, assigning different semantic importance scores to different keywords. Based on the mapping table's defined relationships, the semantic importance score of each Gaussian unit is obtained. For example, keywords like "person," "face," and "hand" indicate an image of a person and receive the highest score of 1.00, while keywords like "flower" and "bonsai" indicate an image of a plant and receive a slightly lower score of 0.90. Images of vehicles and goods might receive a score of 0.80, while images of buildings like windows and doors might only be assigned a score of 0.35. This mapping table is then used to assign semantic importance scores to semantic tags. Mapping to importance scores within the interval [0,1], thus obtaining each Gaussian element. semantic perceived importance score .

[0054] Specifically, if the number of pixels in a semantic label region is less than 100 or the output of the Florence-2 graph-based text model is empty, the call will fail. The semantic awareness importance score of the corresponding Gaussian unit will be calculated based on the area of ​​each semantic category. Thus, each Gaussian element Each received a semantic awareness importance score in the range [0,1]. The formula is as follows:

[0055]

[0056] in, Gausky element semantic tags, Indicates the number of semantic categories.

[0057] III. Constructing a Geometric Perception-Based Gaussian Unit Importance Assessment Mechanism

[0058] The constructed geometry-aware Gaussian element importance evaluation mechanism takes into input the attributes (position, covariance, color, and opacity) of each Gaussian element and the camera projection matrix for each viewpoint, and outputs the geometric visibility count for each Gaussian element. Specifically, for any Gaussian element... In this embodiment of the invention, a camera projection matrix is ​​used. center Project to the Image planes from multiple perspectives yield pixel coordinates. If a Gaussian primitive satisfies the geometric visibility criterion after projection, it is determined that the Gaussian primitive is visible in the current viewpoint. The geometric visibility count of the Gaussian primitive is then calculated across all training viewpoints. The formula for calculating the geometric visibility count is as follows:

[0059]

[0060] in, Indicates the first The first Gaussian Yuan in the Depth values ​​from each viewpoint Represents the projected pixel coordinates. This indicates the setting of the depth value (default). =0.2), and These represent the image width and height, respectively. Indicates an indicator function. The range of values ​​is This indicates that the Gaussian element is in Geometric visibility count in each training view .

[0061] IV. Constructing a Differentiated Compression Model for Gaussian Meta-Importance Perception

[0062] The input to the constructed Gaussian importance-aware differentially compressed network model is the original 3DGS representation model and the semantically perceived importance scores of each Gaussian unit. And geometric perception importance score (Represents the geometric visibility score after normalization for each Gaussian gramm), the output is a semantically and geometrically aware 3DGS compressed model. Specifically, it obtains the semantic awareness importance score for each Gaussian gramm. With geometric perception importance score Then, calculate the overall attention score for each Gaussian unit:

[0063]

[0064] in, This represents the semantic-aware importance score of each Gaussian unit. This represents the normalized geometric perception importance score for each Gaussian element. This represents the overall importance score for each Gaussian element. Adjustable mixing parameters (default) ), used to perform weighted fusion of Gaussian meta-semantic awareness importance scores and geometric awareness importance scores.

[0065] After obtaining the overall attention score of each Gaussian unit, the total target number of Gaussian units after compression is set as follows:

[0066]

[0067] in, This represents the number of Gaussian elements in the original 3DGS model. This represents the Gaussian cell retention rate, which is the ratio of the number of compressed Gaussian cells to the original number of Gaussian cells. Subsequently, it is calculated based on the overall attention score. Sort all Gaussian elements and divide them into groups. Levels (default) =5), the set of all Gaussian elements within each level is called the Gaussian set of that level. Then, different retention rates are assigned to different levels.

[0068]

[0069] In other words, the higher the overall attention score, the higher the Gaussian pixel retention rate. After determining the target Gaussian pixel retention count for each level, budget normalization is performed: if the sum of the target Gaussian pixel counts for each level exceeds... If the number of points is equal, the scale is reduced proportionally; if it is insufficient, the scale is increased proportionally. Since the number of points in each bucket is divided equally (e.g., all Gaussian points are divided into five equal parts according to five levels), the normalization scale is usually close to 1.0, which does not excessively distort the differential distribution.

[0070] Subsequently, the Gaussian sub-models within each level are partitioned into K-dimensional tree spaces. All Gaussian elements within the partitioned local space are called the local Gaussian element set. The Gaussian model reduction (GMR) method is used to aggregate and compress the local Gaussian element set. The aggregated and compressed local Gaussian element sets within each level are then integrated to obtain the aggregated and compressed Gaussian sets for each level.

[0071] Finally, the aggregated and compressed Gaussian sets of each level are merged again to output a semantically and geometrically aware 3DGS compressed model:

[0072]

[0073] in, This represents the complete Gaussian set after merging the aggregated and compressed Gaussian sets of each level. Indicates the number of levels. Indicates the first The Gaussian set after level aggregation and compression.

[0074] V. Training a differentiated compression model for Gaussian meta-importance perception

[0075] After obtaining the semantically and geometrically aware 3DGS compressed model, the geometric properties of each Gaussian primitive are frozen, including its position and covariance matrix. Then, using the 3DGS rendering loss, backpropagation optimization is performed on the color and opacity of each Gaussian primitive, thereby improving the rendering quality of the 3DGS compressed model while preserving the geometric properties of the Gaussian primitives.

[0076] VI. Model Application

[0077] The improvement of the current model lies in allocating a higher proportion of resources to the main subject of the image under limited resources, thereby reducing the loss of detail in the main subject while maintaining a relatively consistent overall image quality. Therefore, it can be applied to scenarios with a clearly defined subject, such as 3D reconstruction of museums and compositing of sporting event scenes. For scenarios like museums and art exhibitions, it allows exhibits that are the main focus to be allocated more Gaussian units during reconstruction, significantly improving the reconstruction quality of the subject. Meanwhile, scenarios like sporting events, if broadcast, also face bandwidth limitations. Therefore, even after compression, the current model can still meet the quality requirements for the main subject.

[0078] VII. Experimental Data

[0079] As can be seen from the images, in the three different perspectives of this scene, the main body area, due to the allocation of more Gaussian elements, exhibits excellent visualization of details in the composite view. Compared with the Ground Truth (real-world scene), it can be observed that the solution proposed in this embodiment can greatly restore the realistic state in details such as bicycle gears, grass around the seat, and the seat itself, meeting users' demands for higher quality views of the main body.

[0080] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for compressing a three-dimensional Gaussian sputtering model based on semantic and geometric awareness, characterized in that, The method includes: A 3DGS compression network is trained using multi-view images to obtain the original 3DGS representation model; Based on the semantically aware Gaussian importance assessment mechanism, the semantic importance of different Gaussian elements in the scene is modeled to obtain the semantically aware importance score of each Gaussian element. Based on the geometry-aware Gaussian importance evaluation mechanism, the geometric visibility number of each Gaussian element in all training views is obtained, and a geometry-aware importance score is obtained. Based on the semantic and geometric importance scores of Gaussian units, a comprehensive attention score is obtained by combining the corresponding weight coefficients. Based on this score, a differentiated compression network for Gaussian unit importance perception is constructed to obtain a 3DGS compression model for semantic and geometric perception. The semantic and geometric perception 3DGS compression model is further fine-tuned to output a compressed 3DGS representation model, which is then applied to scenes such as 3D reconstruction of museums or compositing of sports events.

2. The method for compressing a three-dimensional Gaussian sputtering model based on semantic and geometric awareness according to claim 1, characterized in that, The semantically aware Gaussian meta-importance evaluation mechanism is as follows: The automatic mask generator in the pre-trained semantic segmentation SAM ViT-B model is used to perform forward reasoning for each view to obtain candidate mask maps representing different target regions in the scene. A semantic label allocation strategy based on area priority is adopted, in which the semantic labels of the smaller mask areas are used as the semantic labels of the overlapping areas, and each pixel is assigned a corresponding integer semantic label. For each visible Gaussian element in the scene, based on the camera projection relationship of the current viewpoint, all training views are traversed, the visibility frequency vector of each Gaussian element under each semantic category stored in the semantic mask directory is counted, and its semantic label is determined by taking the semantic label with the maximum frequency. Natural language semantic descriptions are generated from candidate mask images in the semantic mask catalog generated by the semantic segmentation SAM ViT-B model using the Florence-2 graph-based text model. Typical keywords are identified from the language descriptions generated by the Florence-2 model, with different keywords corresponding to different semantic importance scores. Based on the correspondence set in a custom mapping table, the semantic importance scores of each Gaussian unit are obtained. ; If the number of pixels in a semantic label region is less than a threshold or the output of the Florence-2 graph-based text model is empty (indicating a failed call), then the semantic awareness importance score of the corresponding Gaussian unit is calculated based on the area of ​​each semantic category. As a supplement.

3. The method for compressing a three-dimensional Gaussian sputtering model based on semantic and geometric awareness according to claim 1, characterized in that, The geometry-aware Gaussian meta-importance evaluation mechanism is as follows: The input is the attributes of each Gaussian element and the camera projection matrix for each viewpoint; the output is the geometric visibility count for each Gaussian element. For any Gaussian element via camera projection matrix center Project to the Image planes from multiple perspectives yield pixel coordinates. If a Gaussian element satisfies the geometric visibility condition after projection, it is determined that the Gaussian element is visible in the current viewpoint, and the geometric visibility count of the Gaussian element is obtained by counting all training viewpoints.

4. The method for compressing a three-dimensional Gaussian sputtering model based on semantic and geometric awareness according to claim 1, characterized in that, The semantic and geometric perception 3DGS compression model is as follows: Calculate the overall attention score for each Gaussian unit, and set the total target number of compressed Gaussian units; Based on comprehensive attention score Sort all Gaussian elements and divide them into groups. There are several levels, and the set of all Gaussian elements in each level is called the Gaussian set of each level. Different retention rates are assigned to different levels. After determining the target number of Gaussian elements to be retained for each level, budget normalization is performed. If the sum of the number of Gaussian elements for each level of target exceeds If the size is insufficient, the size is proportionally reduced; if it is insufficient, the size is proportionally enlarged. The Gaussian sub-models in each level are divided into K-dimensional tree spaces. All Gaussian elements in the local space after the division are called the local Gaussian element set. The aggregated and compressed local Gaussian element sets in each level are integrated to obtain the aggregated and compressed Gaussian sets of each level. The aggregated and compressed Gaussian sets of each level are then merged again to output a semantically and geometrically aware 3DGS compressed model: in, This represents the complete Gaussian set after merging the aggregated and compressed Gaussian sets of each level. Indicates the number of levels. Indicates the first The Gaussian set after level aggregation and compression.

Images