Task-oriented 3DGS scene data compression method and device

Abstract

The application relates to a task-oriented 3DGS scene data compression method and device. The application extracts a first feature map from a rendering graph of a 3DGS based on a neural network, extracts a second feature map from a true value graph of the 3DGS, calculates a gradient of a Gaussian attribute corresponding to a Gaussian cell according to a feature relationship between the first feature map and the second feature map, and calculates the importance of the Gaussian cell to a task according to the gradient of the Gaussian attribute, so that a compression transformation from a human eye to the task is realized, the compression model can accurately identify and eliminate high-frequency texture details useless to the task, and only the geometric skeleton and semantic key points are reserved, and the compression performance is improved. Moreover, the application introduces a hyperparameter generation network, automatically infers the quantization step of each point according to the local features of the Gaussian cell and the task importance, the strategy can not only avoid tedious manual parameter searching, but also can more finely allocate code rate resources, and the extreme expression of scene information is realized.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and apparatus for task-aware 3DGS scene data compression. Background Technology

[0002] In recent years, with the rapid development of mobile robots, autonomous driving, and augmented reality (AR) technologies, extremely high demands have been placed on the high-fidelity reconstruction and real-time rendering of 3D environments. 3D Gaussian Splatting (3DGS), as an emerging explicit radiant field representation method, leverages the characteristic of using anisotropic Gaussian ellipsoids (3D Gaussians) to represent scene geometry and appearance, achieving real-time rendering speeds and photorealistic reconstruction quality that surpass traditional Neural Radiation Fields (NeRF). Therefore, 3DGS is gradually replacing point clouds and meshes, becoming a key environmental representation in fields such as robot simultaneous localization and mapping, path planning, and visual navigation.

[0003] While 3DGS offers superior rendering quality, its data storage overhead is enormous. A standard room-level scene often contains millions or even tens of millions of Gaussian pixels, with file sizes frequently reaching hundreds of megabytes or even gigabytes. For mobile robot platforms (such as drones and inspection robots) with limited computing resources, storage space, and communication bandwidth, directly storing or transmitting raw 3DGS models would impose a huge system burden, severely restricting their deployment and application on edge devices.

[0004] To address the issue of data volume, existing compression technologies primarily employ strategies such as pruning, quantization, and codebook indexing. However, these methods are generally designed based on the standard human visual system, focusing only on visually perceived redundancy and failing to identify key information differences within the task scenario. For example, in robot navigation tasks, geometrical details relied upon for path planning may be misjudged as redundant and over-compressed, while areas less sensitive to the human eye retain excessive data, leading to low compression efficiency and compromised task performance. This compression mechanism, detached from specific application scenarios, struggles to balance the conflict between data reduction and task requirements, limiting the practical development of 3DGS technology in resource-constrained environments. Summary of the Invention

[0005] The main objective of this disclosure is to propose a task-aware 3DGS scene data compression method that can achieve efficient lossy compression of massive Gaussian metadata. While reducing data storage and transmission costs, it can also ensure the performance of the compressed 3D scene model in downstream robot tasks such as synchronous localization and mapping, semantic navigation, and environmental perception.

[0006] A first aspect of this application provides a task-aware 3DGS scene data compression method, the method comprising: In response to the Gaussian compression command, determine the data compression model, the 3DGS model, and multiple Gaussian elements of the 3DGS model; The plurality of Gaussian primitives are compressed according to the compression model; wherein, the process of compressing the plurality of Gaussian primitives according to the compression model includes: Obtain the rendered image and ground truth image of the 3DGS model from the current viewpoint, and extract the first feature map of the rendered image and the second feature map of the ground truth image based on the neural network. Based on the feature relationship between the first feature map and the second feature map, calculate the gradient of the Gaussian property corresponding to each Gaussian element; Based on the gradient of the Gaussian property corresponding to each Gaussian element, the importance of each Gaussian element to the task of the 3DGS model is calculated. Based on the hyperparameter generation network and the importance level, predict the compression parameters corresponding to each Gaussian unit; Based on the compression parameters corresponding to each Gaussian element, the Gaussian properties corresponding to each Gaussian element are compressed.

[0007] The task-aware 3DGS scene data compression method provided in this application has at least the following beneficial effects: (1) By constructing the feature relationship between the first feature map and the second feature map, the gradient of the Gaussian attribute is calculated, and the importance of Gaussian primitives to the task is calculated based on the gradient of the Gaussian attribute. This realizes the compression transformation from human eye-oriented to task-oriented, and can accurately identify and remove high-frequency texture details that are useless to the task, while retaining only the geometric skeleton and semantic key points. (2) Introducing a hyperparameter generation network, based on the local features of Gaussian units and the importance of the task, automatically infers the quantization step size of each point. This non-uniform adaptive strategy not only avoids tedious manual parameter search, but also allocates bitrate resources more precisely, achieving the ultimate expression of scene information. (3) Feature matching is used as a task proxy scheme, which does not rely on specific and complex downstream algorithm source code or differentiable simulators. This framework can be easily migrated to different tasks by simply replacing the corresponding different neural networks, which reduces the threshold and cost of developing dedicated compression algorithms for new tasks.

[0008] A second aspect of this application provides a task-aware 3DGS scene data compression device, the device comprising: The instruction response module is used to respond to Gaussian compression instructions and determine the data compression model, the 3DGS model, and multiple Gaussian elements of the 3DGS model. A data compression module is used to compress the plurality of Gaussian primitives according to the compression model; wherein, the process of compressing the plurality of Gaussian primitives according to the compression model includes: Obtain the rendered image and ground truth image of the 3DGS model from the current viewpoint, and extract the first feature map of the rendered image and the second feature map of the ground truth image based on the neural network. Based on the feature relationship between the first feature map and the second feature map, calculate the gradient of the Gaussian property corresponding to each Gaussian element; Based on the gradient of the Gaussian property corresponding to each Gaussian element, the importance of each Gaussian element to the task of the 3DGS model is calculated. Based on the hyperparameter generation network and the importance level, predict the compression parameters corresponding to each Gaussian unit; Based on the compression parameters corresponding to each Gaussian element, the Gaussian properties corresponding to each Gaussian element are compressed.

[0009] A third aspect of this application provides an electronic device including at least one controller and a memory for communicatively connecting to the controller; the memory stores instructions executable by the at least one controller, the instructions being executed by the at least one controller to cause the at least one controller to perform a task-aware 3DGS scene data compression method as described in the first aspect of this application.

[0010] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a task-aware 3DGS scene data compression method as described in the first aspect of this application.

[0011] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of a task-aware 3DGS scene data compression method provided in an embodiment of this application; Figure 2This is a schematic diagram of a task-aware 3DGS scene data compression method provided in another embodiment of this application; Figure 3 This is a schematic diagram illustrating the generation of the quantization step size codebook and index value provided in an embodiment of this application; Figure 4 This is a schematic diagram of a task-aware 3DGS scene data compression method provided in an embodiment of this application; Figure 5 This is a schematic diagram of the experimental comparison provided in the embodiments of this application; Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0015] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0016] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed or function in a specific orientation, and therefore should not be construed as a limitation of this application.

[0017] like Figure 1 As shown, this application provides a task-aware 3DGS scene data compression method, the method including: Step S110: In response to the Gaussian compression command, determine the data compression model, the 3DGS model, and multiple Gaussian elements of the 3DGS model.

[0018] Step S120: Compress multiple Gaussian primitives according to the compression model; wherein, the process of compressing multiple Gaussian primitives according to the compression model includes: Step S210: Obtain the rendering map and ground truth map of the 3DGS model from the current viewpoint, and extract the first feature map of the rendering map and the second feature map of the ground truth map based on the neural network.

[0019] Step S220: Calculate the gradient of the Gaussian attribute corresponding to each Gaussian element based on the feature relationship between the first feature map and the second feature map.

[0020] Step S230: Calculate the importance of each Gaussian element to the task of the 3DGS model based on the gradient of the Gaussian property corresponding to each Gaussian element.

[0021] Step S240: Based on the hyperparameters, generate the network and predict the compression parameters corresponding to each Gaussian unit.

[0022] Step S250: Compress the Gaussian attribute corresponding to each Gaussian element according to the compression parameters corresponding to each Gaussian element.

[0023] For ease of understanding, the following explains some key terms in this embodiment: 3D Gaussian Splashing (3DGS) is an emerging explicit radiation field representation method that uses anisotropic Gaussian ellipsoids to represent scene geometry and appearance, enabling high-fidelity reconstruction and real-time rendering.

[0024] Gaussian elements are the basic building blocks of a 3DGS model. Each Gaussian element typically contains Gaussian properties such as position, covariance, color, and opacity, which are used to describe the local geometry and appearance information in the scene. Multiple Gaussian elements work together to construct a complete 3DGS scene.

[0025] Hyperparameter generation networks are a type of neural network whose input can be information such as the importance of Gaussian cells, and whose output is parameters used to guide Gaussian cell compression.

[0026] In step S110, the method includes responding to a Gaussian compression command and determining a data compression model, a 3DGS model, and multiple Gaussian elements of the 3DGS model.

[0027] In practical applications, Gaussian pixel compression commands can be manually triggered by the user through a graphical user interface, or automatically issued by the system based on preset conditions (such as insufficient storage space or limited network bandwidth). The data compression model can be a pre-trained general compression model capable of handling different types of 3DGS data. The 3DGS model and its constituent Gaussian pixels can be loaded from local storage devices or obtained from a remote server via a network. For example, in a mobile robot system, when the robot needs to send a 3DGS map of its current environment to a remote operator, a compression command is triggered. The system then loads the 3DGS model corresponding to the current map and all its Gaussian pixels, preparing the data compression model for compression.

[0028] In step S120, multiple Gaussian primitives are compressed according to a compression model. Specifically, the process of compressing multiple Gaussian primitives according to the compression model includes obtaining the rendered image and ground truth image of the 3DGS model from the current viewpoint in step S210, and extracting the first feature map of the rendered image and the second feature map of the ground truth image based on a neural network.

[0029] In this embodiment, a rendered image of the current viewpoint can be generated using a 3DGS differentiable rasterizer. The ground truth image can be a rendered image of the original uncompressed 3DGS model or an image captured from the real world. The neural network can be a pre-trained image feature extraction network, such as SuperPoint, DeepLabV3, and DINOv2. Notably, different neural networks can be selected to extract feature points from the image for different tasks.

[0030] In step S220, the gradient of the Gaussian property corresponding to each Gaussian element is calculated based on the feature relationship between the first feature map and the second feature map.

[0031] Feature relationships can be obtained by calculating the cosine similarity and / or L2 distance between the first and second feature maps. Gradient calculation can be implemented using the standard backpropagation algorithm to quantify the influence of the Gaussian properties of each Gaussian unit on the feature relationships. For example, if there is a large difference between the first and second feature maps, and the color attribute of a certain Gaussian unit contributes significantly to this difference, then the gradient value of that color attribute will be large.

[0032] In step S230, the importance of each Gaussian element to the task of the 3DGS model is calculated based on the gradient of the Gaussian property corresponding to each Gaussian element.

[0033] Importance can be calculated based on the absolute value of the Gaussian property gradient under a single viewpoint. For example, the larger the absolute value of the gradient of a Gaussian element, the greater its influence on the feature relationship under the current viewpoint, and therefore it is considered to have higher importance. Alternatively, it can be calculated as the cumulative magnitude of the gradient of a Gaussian element across all training viewpoints.

[0034] In step S240, the compression parameters corresponding to each Gaussian unit are predicted based on the hyperparameter generation network and importance.

[0035] The hyperparameter generation network can be a simple multilayer perceptron (MLP) or a HyperNet. Its input is the importance of the primitive, and its output is a uniform compression parameter for that primitive, such as a fixed quantization step size. For example, for primitives with high importance, the hyperparameter generation network may predict a smaller quantization step size to retain more details, while for primitives with low importance, it may predict a larger quantization step size.

[0036] In step S250, the Gaussian attribute corresponding to each Gaussian element is compressed according to the compression parameters corresponding to each Gaussian element.

[0037] Compression can be a simple uniform quantization of Gaussian properties, mapping property values ​​to a predefined set of discrete values. For example, if the predicted compression parameter is a quantization step size, the color or opacity attribute of each Gaussian cell will be quantized according to that step size, thereby reducing the number of bits required to represent it.

[0038] Here is a specific example: User A uses a mobile robot for autonomous navigation in an indoor environment. The robot needs to transmit its perceived 3DGS scene data to a remote control center in real time so that the operator can monitor and plan the path. However, the robot's communication bandwidth is limited, and the original 3DGS model data is huge. Direct transmission will cause serious latency and bandwidth consumption, affecting the efficiency of the navigation task. Traditional compression methods may only focus on visual fidelity, ignoring obstacle information or details of specific target objects that are crucial to the navigation task.

[0039] To address this issue, the robot employs the aforementioned task-aware 3DGS scene data compression method. First, when the robot receives an instruction to transmit 3DGS scene data, the system responds with a Gaussian cell compression instruction, determines the 3DGS model of the current environment and its millions of Gaussian cells, and loads a pre-defined data compression model.

[0040] The robot generates a rendered image from its current perspective and simultaneously acquires a ground truth image provided by a high-precision sensor. Subsequently, a pre-trained neural network is used to extract a first feature map and a second feature map from the rendered image and the ground truth image, respectively. These feature maps contain geometric and semantic information about the scene, such as walls, furniture, doors, and windows.

[0041] The system calculates the feature relationship based on the L2 distance and cosine similarity between the first feature map and the second feature map. Then, based on the feature relationship, the system calculates the gradient of the Gaussian attribute (such as position, color, opacity) corresponding to each Gaussian element through the backpropagation algorithm. These gradients reflect the contribution of each Gaussian element to the feature difference between the rendered map and the ground truth map.

[0042] The system calculates the importance of each Gaussian element to the robot navigation task based on the gradient of the Gaussian property corresponding to each Gaussian element. For example, the system can simply use the sum of the absolute values ​​of the gradients of all Gaussian properties of each Gaussian element under the current viewpoint as a measure of its importance. The larger the sum of the absolute values ​​of the gradients of a Gaussian element, the more important it is to the navigation task.

[0043] The hyperparameter generation network receives the importance of each Gaussian primitive as input and predicts the corresponding quantization step size. For Gaussian primitives that are determined to be very important to the navigation task (such as primitives that constitute obstacles), the hyperparameter generation network predicts a smaller quantization step size to ensure that their attributes are preserved with high accuracy. For primitives with lower importance (such as primitives that constitute background walls), a larger quantization step size is predicted, allowing for greater compression.

[0044] The system uniformly quantizes and compresses the Gaussian attributes of each Gaussian unit based on the prediction compression parameters corresponding to each Gaussian unit. For example, the Gaussian attributes of important obstacles are finely quantized, while the background Gaussian attributes are coarsely quantized. Through this process, the amount of data in the original 3DGS model is significantly reduced, forming a compressed 3DGS model. The compressed model is then transmitted to a remote control center, where operators can perform efficient monitoring and path planning without delays caused by excessive data volume.

[0045] Based on the above examples, this method, by introducing a task-aware compression strategy, has at least the following beneficial effects: (1) Existing technologies usually take human visual quality as the optimization goal, which makes it difficult to distinguish visual redundancy in the scene. This embodiment calculates the gradient of Gaussian attributes by constructing the feature relationship between the first feature map and the second feature map, and calculates the importance of Gaussian primitives to the task based on the gradient of Gaussian attributes. This realizes the compression transformation from human eye-oriented to machine-oriented, and can accurately identify and remove high-frequency texture details that are useless to the task, while retaining only the geometric skeleton and semantic key points. (2) This embodiment introduces a hyperparameter generation network, which automatically infers the quantization step size of each point based on the local features of Gaussian units and the importance of the task. This non-uniform adaptive strategy not only avoids tedious manual parameter search, but also allocates bitrate resources more precisely, achieving the ultimate expression of scene information. (3) This embodiment uses feature matching as a task proxy scheme, which does not rely on specific and complex downstream algorithm source code or differentiable simulators. This framework can be easily migrated to different tasks by simply replacing the corresponding different neural networks, which reduces the threshold and cost of developing dedicated compression algorithms for new tasks.

[0046] In some embodiments of this application, the compression parameters include quantization step size and retention probability; Step S250, which involves compressing the Gaussian attribute corresponding to each Gaussian element based on the compression parameters, includes: Step S2510: Select the corresponding Gaussian elements whose retention probability is greater than the preset value; Step S2520: Based on the quantization step size of the Gaussian property of the Gaussian cell with a retention probability greater than the preset value, quantize the corresponding Gaussian property of the Gaussian cell with a retention probability greater than the preset value to obtain quantization parameters.

[0047] Quantization step size is a key parameter in digital signal processing, used to define the precision when mapping continuous or high-precision discrete data to a finite set of discrete values. Its role is to control the granularity of data compression. A larger quantization step size leads to coarser quantization, thereby achieving a higher compression ratio, but may introduce greater distortion; a smaller quantization step size retains more details, but the compression ratio is relatively lower.

[0048] The retention probability refers to the probability that a data unit will be retained instead of discarded during data compression or sparsification. Its function is to achieve structural pruning or sparsification. By setting a threshold, only data units that contribute significantly to model performance or task completion are retained, thereby greatly reducing the amount of data while ensuring that core information is not lost. The retention probability can be a continuous value, determined by comparison with a preset threshold; or it can be binary, directly indicating whether to retain or discard.

[0049] In step S2510, Gaussian primitives with retention probabilities greater than a preset value are selected. This step serves to perform preliminary screening or pruning based on the importance assessment results of each Gaussian primitive, removing those that contribute little to the task and can be safely discarded. This is an importance-based sparsity strategy that effectively reduces the amount of data processed subsequently and concentrates resources on processing more critical information. This can be achieved by comparing the retention probability of each Gaussian primitive with a preset threshold (such as 0.5, 0.7, or 0.9), marking Gaussian primitives with retention probabilities higher than this threshold as retained, and the rest as discarded. Discarded Gaussian primitives do not participate in the subsequent compression process.

[0050] This embodiment introduces quantization step size and retention probability as specific compression parameters, and combines Gaussian primitives with personalized allocation based on task importance, making the compression process more flexible and targeted. First, using retention probability for filtering intelligently eliminates Gaussian primitives that contribute little to the task, thereby significantly reducing the amount of data that needs to be stored or transmitted without significantly impacting task performance. Second, for the retained Gaussian primitives, different quantization step sizes are assigned according to their importance, ensuring that critical information is retained with higher precision, while less important information is compressed with lower precision, achieving an optimal balance between compression ratio and task performance. This refined compression strategy effectively solves the challenge of balancing compression efficiency and task performance in traditional compression methods, significantly improving the efficiency and user experience of 3DGS scene data during storage, transmission, and rendering.

[0051] In some embodiments of this application, after selecting the corresponding Gaussian elements with a retention probability greater than a preset value in step S2520, the method further includes: Based on clustering algorithms or non-uniform quantizers, the quantization step size of the corresponding Gaussian elements whose retention probability is greater than a preset value is used to map and generate a quantization step size codebook and the index value corresponding to each element in the quantization step size codebook.

[0052] In step S2520, based on the quantization step size of the selected Gaussian elements with retention probabilities greater than a preset value, the corresponding Gaussian attributes of the selected Gaussian elements with retention probabilities greater than a preset value are quantized to obtain quantization parameters, including: Step S2521: Determine the target Gaussian element. The target Gaussian element is any one of the Gaussian elements whose retention probability is greater than a preset value.

[0053] Step S2522: Search for the corresponding target quantization step in the quantization step codebook based on the target index value corresponding to the target Gaussian element.

[0054] Step S2523: Quantize the Gaussian property corresponding to the target Gaussian element according to the target quantization step size to obtain the quantization parameters.

[0055] Clustering algorithms or non-uniform quantizers aim to group or optimize the representation of a large number of discrete quantization steps to reduce storage and transmission overhead. Clustering algorithms can use K-means clustering to group similar quantization steps into one class and use the center value of that class as a representative. Non-uniform quantizers can be based on algorithms such as Lloyd-Max, designing non-uniform quantization intervals and representative values ​​according to the distribution characteristics of the quantization steps to optimize quantization error. A quantization step codebook stores a set of representative quantization step values ​​to replace the original, potentially large number of discrete quantization steps. It can be a lookup table, array, or hash table, where each element is a quantization step value. Index values ​​are used to point to a specific quantization step in the codebook, usually represented as integers corresponding to their positions within the codebook.

[0056] In steps S2521 to S2523, the target Gaussian element refers to a single Gaussian element that needs to be quantized during the compression process. It can be any one of the selected Gaussian elements whose retention probability is greater than a preset value. The target index value is the index of the quantization step size corresponding to the target Gaussian element in the codebook, which is obtained through lookup or mapping. The target quantization step size is retrieved from the codebook and is the actual quantization step size used to quantize the attribute of the target Gaussian element. It is directly read from the quantization step size codebook according to the target index value. Quantizing the Gaussian attribute is to convert the floating-point attribute value of the Gaussian element into a discrete integer value to achieve data compression. Uniform quantization or non-uniform quantization can be used.

[0057] This embodiment can effectively solve the data redundancy problem caused by independently storing and transmitting quantization step sizes for each Gaussian element. By generating and utilizing a quantization step size codebook, the potentially large number of discrete quantization step sizes are mapped to a finite number of codebook elements and their indices, which greatly reduces the representation complexity of quantization parameters. This not only significantly reduces the bit rate of compressed data and improves the efficiency of data transmission and storage, but also achieves more efficient data compression while ensuring the quality of 3DGS scene reconstruction.

[0058] In some embodiments of this application, after compressing the Gaussian property corresponding to each Gaussian element according to the compression parameters corresponding to each Gaussian element in step S250, the method further includes: Step S310: Entropy encoding is performed on the index values ​​of the corresponding Gaussian units whose retention probabilities are greater than the preset values ​​to obtain the encoding parameters.

[0059] Step S320: Pack the quantization step size codebook, encoding parameters, and quantization parameters and send them to the client. The client decodes the encoding parameters to obtain the index value. The quantization step size of the corresponding Gaussian unit is read from the quantization step size codebook based on the index value. The quantization parameters are decompressed based on the quantization step size to reconstruct the 3DGS scene based on the decompressed Gaussian unit.

[0060] Entropy coding is a lossless data compression technique. Its core idea is to allocate different coding lengths based on the probability of symbols appearing in the data. Frequently occurring symbols are assigned short codes, and infrequently occurring symbols are assigned long codes, thereby reducing data redundancy. The coding parameters are the data after entropy coding processing. They contain information about the original Gaussian element index values, but are represented in a more compact form. After being transmitted to the client, these parameters need to be restored using a corresponding decoding algorithm to obtain the original index values. Packaging refers to the process of organizing multiple related data items (such as quantization step size codebook, coding parameters, and quantization parameters) into a unified data structure or file format. Its purpose is to ensure the integrity and consistency of data during transmission, facilitating the client to receive and process all necessary information at once. Sending to the client refers to transmitting the packaged compressed data to a remote client device via a network or other communication medium. This allows the client to obtain and use the compressed 3DGS scene data. Data transmission can be based on various network protocols, such as TCP / IP, UDP, HTTP / S, or WebSocket, depending on the application scenario's requirements for reliability, real-time performance, or bandwidth. After receiving the encoded parameters, the client needs to perform entropy decoding operations corresponding to the encoding process to restore the encoded parameters to the original Gaussian pixel index values. This process ensures that the client can identify and locate the compressed Gaussian pixels. After obtaining the index values ​​of the Gaussian pixels, the client uses these index values ​​as lookup keys to query the pre-received quantization step codebook to obtain the specific quantization step size used for each Gaussian pixel during compression. This is usually achieved through array lookup, hash table lookup, or binary search. Decompression, also known as inverse quantization, refers to the client using the quantization step size read from the quantization step codebook to perform the inverse operation on the received quantization parameters, thereby recovering an approximate Gaussian attribute value. For example, if the quantization parameter is an integer, the decompression process may involve multiplying the quantization parameter by the corresponding quantization step size and possibly adding a preset offset. After successfully decompressing and recovering the attributes of the Gaussian pixels (such as position, color, size, rotation, etc.), the client uses these decompressed Gaussian pixels to render through the 3DGS rendering pipeline, thereby reconstructing and displaying the 3DGS scene on the client device.

[0061] This embodiment effectively addresses the challenges faced by compressed 3DGS scene data during transmission and client reconstruction. By entropy encoding the selected Gaussian meta-index values, the amount of data required for transmission is significantly reduced, thereby improving data transmission efficiency. Especially in environments with limited network bandwidth, the unified packaging of the quantization step size codebook, encoding parameters, and quantization parameters ensures the integrity and consistency of data transmission, avoiding the problem of the client being unable to correctly reconstruct the scene due to missing or mismatched data. The client can accurately decode the original index values ​​based on the received encoding parameters and precisely decompress the quantization parameters using the quantization step size codebook, ultimately achieving high-quality reconstruction of the 3DGS scene. This enables the compressed 3DGS model to be presented efficiently and accurately on the client, greatly improving the user experience and expanding the deployment capabilities of 3DGS technology in various application scenarios.

[0062] In some embodiments of this application, the feature relationship includes the sum of the L2 distance and cosine similarity between the first feature map and the second feature map.

[0063] In this embodiment, feature relation refers to a metric used to quantify the similarity or difference between the first feature map and the second feature map. Its role is to provide a basis for subsequent calculation of the gradient of the Gaussian property corresponding to the Gaussian element, thereby accurately assessing the importance of the Gaussian element to the 3DGS model task.

[0064] L2 distance is a metric that measures the straight-line distance between two points in a multidimensional space. In the context of feature maps, L2 distance quantifies the numerical difference between the first and second feature maps, i.e., the absolute difference in pixel values ​​or feature vector element values. It is typically calculated as the square root of the sum of the squares of the differences between corresponding elements. Cosine similarity is a measure of the cosine of the angle between two non-zero vectors. It assesses the similarity of two vectors in a direction. In the context of feature maps, cosine similarity quantifies the similarity between the first and second feature maps in terms of pattern or direction, i.e., whether the shape or trend of the feature vectors is consistent. It is typically calculated as the dot product of the two vectors divided by the product of their respective magnitudes. This example combines L2 distance and cosine similarity to comprehensively evaluate the relationship between feature maps from both numerical difference and pattern similarity dimensions, thus providing richer and more robust feature matching information.

[0065] This embodiment defines the feature relationship between the first feature map and the second feature map as the sum of L2 distance and cosine similarity. This allows for a comprehensive and robust comparison of the feature maps from two dimensions: numerical difference and pattern similarity. This multi-dimensional feature relationship metric enables a more accurate capture of the impact on the rendering results when calculating the gradient of the Gaussian attribute corresponding to each Gaussian primitive, thereby significantly improving the accuracy of Gaussian primitive importance assessment. Based on a more accurate importance assessment, subsequent compression parameter prediction and Gaussian attribute compression can more effectively identify and retain information that is crucial to specific tasks, avoiding misjudgments and loss of key information that may be caused by a single feature relationship metric. Therefore, this embodiment can achieve higher quality and more task-aware 3DGS scene data compression, ensuring that it can still effectively support the execution of downstream tasks after compression.

[0066] In some embodiments of this application, the loss function of the compression model includes: ; in, The loss function for the compression model is... For task proxy loss function, Let be the bitrate loss function. The distortion loss function; It can be any value between 0 and 1. for The weight, for The weights; The task proxy loss function includes the loss of the sum of the L2 distance and cosine similarity between the first feature map and the second feature map; The bitrate loss function includes the bitrate loss of the corresponding Gaussian bits whose retention probability is greater than a preset value; wherein, the bitrate is calculated from the quantization parameters using an entropy model; The distortion loss function includes the distortion loss between the first feature map and the second feature map.

[0067] A loss function is a metric used during machine learning model training to measure the difference between the model's predictions and the true values, with the goal of minimizing this difference. In this embodiment, the loss function is designed as a weighted sum of the task proxy loss function, bitrate loss function, and distortion loss function, aiming to balance the task performance, data size, and visual reconstruction quality of the compressed 3DGS model. This can be achieved by calculating these three losses during training based on the model's output compression parameters and original Gaussian properties, and updating the model parameters using optimization algorithms such as gradient descent. Alternatively, the weights of each loss term can be dynamically adjusted according to the needs of the actual application scenario to adapt to different compression objectives.

[0068] The task proxy loss function is used to measure the performance of the compressed 3DGS model on a specific task. In this embodiment, it is reflected by comparing the sum of the L2 distance and cosine similarity between the first feature map and the second feature map. Its role is to ensure that the compressed model can maintain high accuracy when performing downstream tasks (such as SLAM). One implementation method is to pre-train a feature extraction network to map the rendered map and the ground truth map to a high-dimensional feature space, and then calculate the distance and similarity between these feature vectors. Another implementation method is to directly train the feature extraction network as part of the compressed model end-to-end, making its feature representation more adapted to the compression task.

[0069] The bitrate loss function measures the size of the compressed data, i.e., the compression efficiency. Its role is to guide the model to generate more compact compression parameters, thereby reducing the number of bits required for storage and transmission. In this embodiment, the bitrate is calculated from the quantization parameters using an entropy model. One approach is to estimate the entropy value of the quantization parameters using a learnable entropy model (such as a probability model based on a Gaussian or Laplace distribution) and use it as the loss term. Another approach is to use a context-based entropy encoder to dynamically adjust the bitrate estimate based on the statistical characteristics of the quantization parameters.

[0070] The distortion loss function measures the visual difference between the reconstructed image and the original image, i.e., the reconstruction quality. Its purpose is to ensure that the compressed 3DGS model still provides high-quality scene rendering. In this embodiment, the distortion loss function includes the distortion loss between the first feature map and the second feature map. This can be achieved by quantifying the distortion by calculating the L1 or L2 distance between the first and second feature maps. Another approach is to use perceptual loss (such as VGG loss), which better simulates the distortion perceived by the human eye by comparing the differences between feature maps extracted by the deep neural network.

[0071] in, and These two parameters are the weights of the bitrate loss and distortion loss in the loss function. Their role is to balance the contribution of different loss terms to the total loss, thereby making a trade-off between compression ratio, task performance, and visual quality. Furthermore, a condition variable is introduced. As a control signal, a single trained model can dynamically adjust the compression aggressiveness during inference based on bandwidth fluctuations or task accuracy requirements in actual applications, generating an optimal compressed bitstream located on the Pareto frontier.

[0072] The compression model in this embodiment optimizes task performance, data bitrate, and visual reconstruction quality simultaneously when compressing 3DGS scene data using a comprehensive loss function. Specifically, the introduction of the task proxy loss function allows the model to perceive and retain feature information crucial to specific downstream tasks (such as object detection and scene understanding) during compression, ensuring that the compressed 3DGS model maintains high accuracy and robustness when performing these tasks. The introduction of the bitrate loss function directly constrains the storage and transmission costs of the compressed data, achieving efficient data compression. The distortion loss function ensures that the compressed 3DGS scene still provides high-quality rendering effects. Here, through multi-objective joint optimization of bitrate-distortion-task (RD-Task), the model is forced to prioritize the protection of high-sensitivity regions during compression, maintaining the stability of feature point extraction and the integrity of geometric structure even at extremely high compression ratios, thereby ensuring the success rate and robustness of task execution.

[0073] In some embodiments of this application, step S230, which calculates the importance of each Gaussian element to the task of the 3DGS model based on the gradient of the Gaussian property corresponding to each Gaussian element, includes: Step S2310: Based on the gradient of the Gaussian property corresponding to each Gaussian element, calculate the cumulative magnitude of the gradient of each Gaussian element under all views.

[0074] Step S2320: Determine the importance of each Gaussian element to the task of the 3DGS model based on the cumulative modulus of each Gaussian element.

[0075] In this embodiment, the cumulative magnitude of the gradient of each Gaussian primitive (GMT) is calculated across all viewpoints. This aims to more comprehensively and robustly evaluate the importance of each GMT by aggregating gradient information from different viewpoints. A gradient from a single viewpoint may only reflect the local contribution of a GMT within that specific viewpoint, while the cumulative magnitude reflects its global contribution across the entire scene or task. The process can be achieved by iterating through all available training viewpoints or pre-defined representative viewpoints, calculating the gradient of the corresponding Gaussian property for each GMT at each viewpoint, and then summing the magnitudes of these gradients. Alternatively, multiple viewpoints can be randomly sampled, and the gradient magnitudes calculated and summed at each sampled viewpoint to achieve a balance between computational efficiency and evaluation accuracy.

[0076] In this embodiment, the importance of each Gaussian primitive (GMT) to the 3DGS model task is determined based on its cumulative modulus. This step transforms the previously calculated cumulative modulus into an importance index that can be used for subsequent compression decisions. A larger cumulative modulus generally means a greater contribution of the GMT to the 3DGS model task, thus indicating higher importance. This importance can be determined by directly using the cumulative modulus as the original value, or by normalizing it to a specific range. Alternatively, a threshold can be set, marking GMTs with cumulative modulus exceeding the threshold as important and those below as unimportant. A hierarchical strategy can also be used, classifying GMTs into different importance levels based on their cumulative modulus.

[0077] like Figure 2 , Figure 3 and Figure 4 The embodiments of this application provide a task-aware 3DGS scene data compression method.

[0078] This method primarily addresses the resource constraints of mobile robot platforms. Starting from a machine vision-oriented coding paradigm, it constructs a multi-objective joint optimization model for bitrate, distortion, and task requirements. Utilizing source coding techniques such as model pruning, adaptive quantization, and entropy coding, it achieves efficient lossy compression of massive Gaussian units. The aim is to reduce data storage and transmission costs while ensuring the performance of the compressed 3D scene model in downstream tasks such as simultaneous localization and mapping, semantic navigation, and environmental perception.

[0079] The specific technical approach includes the following four key steps: (1) Construct a differentiable task mapping mechanism: To address the challenge that downstream robotic tasks (such as SLAM and semantic navigation) often contain non-differentiable modules, making direct gradient optimization impossible, this embodiment introduces a pre-trained neural network (such as SuperPoint or DINOv2) as a teacher network to act as a task proxy. By constraining the consistency between the compressed 3DGS rendered image and the original ground truth image in the feature space, rather than just the pixel space, a differentiable task loss function is constructed, thereby explicitly incorporating task performance into the optimization loop.

[0080] (2) Establish a fine-grained feature evaluation system: By leveraging the backpropagation characteristics of the aforementioned task agent, the gradient of the task loss with respect to each Gaussian attribute in the scene is calculated. Then, the importance of each Gaussian attribute to the robot's downstream tasks is quantified by the magnitude of the gradient, generating a task sensitivity map. This step achieves a shift from uniform compression to on-demand bitrate allocation.

[0081] (3) Implement an adaptive strategy generation and clustering indexing mechanism based on hypernetworks: To address the challenge of the decoder's inability to obtain task gradient information, the generation of the adaptive strategy is strictly limited to the encoder. First, a hyperparameter generation network (HyperNet) is used to infer the ideal quantization step size and pruning probability for each Gaussian primitive based on its sensitivity and local properties. Then, to avoid the significant overhead of transmitting quantization parameters point-by-point, this embodiment introduces a parameter clustering mechanism. The quantization step sizes of the massive number of Gaussian primitives are clustered into K representative levels (constructing a quantization parameter codebook), and a corresponding index value is assigned to each Gaussian primitive. The decoder only needs to receive a compact index stream and the codebook to look up the table and recover the decompression strategy for each point, thereby achieving efficient non-uniform compression.

[0082] (4) RD-Task multidimensional joint optimization and dynamic adjustment: During the training phase, a joint objective function incorporating bitrate loss, distortion loss, and task loss is constructed to optimize the model parameters end-to-end. Simultaneously, a conditional variable is introduced as a control signal, enabling a single trained model to dynamically adjust the compression aggressiveness during inference based on bandwidth fluctuations or task accuracy requirements in real-world applications, generating an optimal compressed bitstream located on the Pareto front.

[0083] The method specifically includes the following steps: Step S710: Select a neural network and construct a differentiable teacher network. As an agent; Step S7110, 3DGS model parameters ,in, For location, For covariance (by scaling) and rotation constitute), For opacity, The spherical harmonic coefficients, The total number of Gaussian elements; Step S7120, Teacher Network Selection; wherein step S7120 includes: Step S7121: For the SLAM task, a pre-trained feature point extraction network (Super Point) is selected as the teacher network. .

[0084] Step S7122: For the semantic navigation task, select a pre-trained semantic segmentation network (DeepLabV3 or DINOv2) as the teacher network. .

[0085] Step S7130, Feature rendering and matching; Given camera pose RGB images are rendered using a 3DGS miniaturized rasterizer. ,Will Input Teacher Network Extract the first feature map of the intermediate layer Simultaneously, acquire the ground truth image (or the original uncompressed 3DGS rendering) from this perspective. Second feature map .

[0086] Step S7140, Task Agent Loss Function Defined as the distance between feature maps; ; By combining L2 distance and cosine similarity, we ensure that both the intensity and orientation of the features are preserved.

[0087] Step S720: Calculate the importance of each Gaussian unit to the performance of the robot's downstream tasks, and generate a sensitivity map to guide subsequent non-uniform quantization and pruning. Step S720 includes: Step S7210, gradient backpropagation; Calculate the task proxy loss using an automatic differentiation mechanism. For each Gaussian element Gaussian properties (mainly position) and opacity The gradient of ) ; Step S7220, Sensitivity Definition: Definition of the first The task sensitivity of each high-level element For all training perspectives The cumulative magnitude of the downgradient is calculated and normalized. ; in, In the first From one perspective, task agent loss To Gorsky The gradient of the Gaussian property.

[0088] Step S7221, The larger the value, the more important the Gaussian element is to maintaining the performance of downstream tasks of the robot (such as object edges and corners). Step S7222, The smaller the value, the less important the Gaussian element is to the task (such as the texture of a flat wall).

[0089] Step S730: Policy generation and parameter clustering based on HyperNet; Step S730 includes: Step S7310: Run at the encoding end, using the ideal compression parameters calculated by HyperNet, and convert them into a transferable index through clustering; Step S7311, for each Gaussian element This will include task sensitivity. eigenvectors Input HyperNet and output the ideal quantization step size at that point. and retention probability ; Step S7312, set the threshold ,like Then a pruning mask is generated. =0 (this point is discarded and will not proceed to the next step), otherwise =1; Step S7313, Quantize parameter clustering; Step S7320, collect all reserved points ( Ideal quantization step size = 1) ; Step S7330: Using the K-Means clustering algorithm or a non-uniform quantizer, the continuously changing... Mapped to A discrete representative value (e.g.) ), forming a quantization step size codebook ; Step S7331: Set the ideal step size for each Gaussian element. Mapped to the representative value of the nearest neighbor in the codebook And record its index. ; Step S7332: Use the actual quantization step size obtained from the lookup table. Quantifying Gaussian properties: ; in, This represents the minimum retention probability.

[0090] Step S740: Construct the RD-Task three-objective joint optimization model; Step S740 includes: Construct the final loss function, adjusting the HyperNet parameters. End-to-end training was performed using the finely tuned 3DGS parameters; Step S7410, bitrate loss: The bitrate of the quantization parameters is estimated using an entropy model, assuming... Given the probability distribution of quantized symbols, the bitrate loss is approximately: ; As a pruning mask, the pruned Gaussian does not contribute to the bitrate; Step S7420, Distortion loss: Using traditional or Loss, ensuring the basic visual structure for the human eye is preserved, and preventing excessive collapse; ; Step S7430, Overall optimization objective: ; in, It is an externally input condition variable; During training, Random sampling within the interval This forces the compressed network to learn all the optimal solutions on the Pareto Frontier; Step S750, bitstream encapsulation and decoding inference; Step S750 includes: Step S7510, coding and packaging; Step S7511: Generate the quantization step size codebook Write to the header of the bitstream; Step S7512, set the index value of each Gaussian cell. Entropy encoding is performed. To further compress the data, a block indexing strategy can be adopted, where a group of spatially adjacent Gaussian points share the same index. Step S7513 involves compressing and packaging the quantization parameters obtained after quantizing the Gaussian properties. Step S7520, Decoding and Reasoning; In step S7521, the decoder first reads the quantization step size codebook from the beginning of the bitstream. ; Step S7522: Decode the index stream to obtain each point. Through table lookup operation Directly obtain the inverse quantization step size at that point; Step S7523, using The Gaussian parameters are dequantized and reconstructed to recover the 3DGS scene for rendering.

[0091] This embodiment aims to address the technical problems of severe bitrate redundancy and loss of critical task features in robotic applications caused by existing 3D Gaussian sputtering compression technology, which focuses solely on optimizing human visual quality. This embodiment breaks through the traditional peak signal-to-noise ratio-oriented compression paradigm by constructing a multi-objective joint optimization model based on task agents and sensitivity maps, achieving a shift from preserving visual performance to preserving task performance. Simultaneously, it utilizes a hyperparameter generation network to generate adaptive quantization and pruning strategies for different downstream tasks and bandwidth constraints. This aims to significantly reduce data storage volume and transmission bandwidth usage while maximizing the preservation of geometric and semantic information crucial to machine vision algorithms, thereby effectively balancing compression ratio and task accuracy, and improving the applicability and operational efficiency of 3DGS technology on mobile robot platforms with limited computing power and communication.

[0092] Compared with the prior art, this embodiment has at least the following significant beneficial effects: (1) It breaks through the bottleneck of traditional visual compression and significantly improves the compression efficiency in machine task scenarios; Existing technologies typically optimize for human visual quality, making it difficult to distinguish visual redundancy in a scene. This invention, by constructing a task agent and sensitivity map, achieves a first-ever shift from human-eye-oriented to machine-oriented compression. This embodiment can accurately identify and remove high-frequency texture details useless for robotic tasks, retaining only the geometric skeleton and semantic key points. Experiments show that, under the same task accuracy requirements, this embodiment's method can further reduce the bitrate by 30%-50% compared to traditional PSNR-based compression methods, greatly alleviating the storage and transmission pressure on mobile robots.

[0093] (2) It solves the problem of key feature loss under low bit rate and ensures the robustness of downstream tasks; Under extremely low bandwidth conditions, traditional methods often employ uniform and aggressive compression, which can easily destroy features crucial to machine vision algorithms, such as object edges and corners, leading to SLAM tracking loss or semantic segmentation failure. This embodiment utilizes RD-Task multi-objective joint optimization to force the model to prioritize the protection of high-sensitivity regions during compression. Even at extremely high compression ratios, it maintains the stability of feature point extraction and the integrity of the geometric structure, thereby ensuring the success rate and robustness of downstream robot tasks.

[0094] (3) It realizes fine-grained adaptive generation of compression strategies, eliminating the limitations of manual parameter tuning; Existing technologies typically employ a uniform quantization step size or pruning threshold for the entire scene, lacking flexibility. This embodiment introduces a hyperparameter generation network, which automatically infers the optimal quantization step size and pruning probability point by point based on the local features and task importance of each Gaussian unit. This non-uniform adaptive strategy not only avoids tedious manual parameter search but also allocates bitrate resources more precisely, achieving the ultimate expression of scene information.

[0095] (4) It has flexible dynamic adjustment capabilities, and a single model can adapt to a variety of application needs; This embodiment introduces an adjustable condition variable into the optimization objective. After training, there is no need to retrain multiple models for different compression ratios. Instead, variables only need to be adjusted during the inference phase to dynamically generate various compression strategies located on the Pareto front. This enables the robotic system to seamlessly switch between high-precision and low-latency modes in real time, based on current real-time network bandwidth or task accuracy requirements, demonstrating significant engineering practical value.

[0096] (4) It has good versatility and scalability; This embodiment uses feature matching as a task proxy scheme, which does not rely on specific and complex downstream algorithm source code or differentiable simulators. This means that the framework can be easily migrated to different robotic tasks (only the corresponding teacher network needs to be replaced), reducing the threshold and cost of developing dedicated compression algorithms for new tasks.

[0097] The following is a more detailed explanation with reference to the accompanying drawings.

[0098] In the first example, this embodiment simulates a drone conducting inspections in a field or underground environment with limited communication bandwidth, which requires compressing and transmitting the three-dimensional environment it constructs back to the ground station for high-precision positioning.

[0099] Implementation environment hardware platform; Model training (ground station / cloud): A high-performance workstation equipped with an Intel Xeon Gold processor and an NVIDIA RTX 4090 graphics card (24GB VRAM) for handling Gaussian sparsity training and compression strategy generation. Inference and application (drone carrier): An NVIDIA Jetson Orin NX embedded computing module (16GB RAM) to simulate a resource-constrained edge computing environment.

[0100] Software and data environment: Operating system: Ubuntu 20.04LTS.

[0101] Development framework: PyTorch 1.13 + CUDA 11.7, integrated with gsplat or diff-gaussian-rasterization rendering library.

[0102] Downstream task algorithm: ORB-SLAM3 (used to verify feature point tracking performance).

[0103] Dataset: TUM-RGBD or Replica datasets were selected to simulate indoor and outdoor scenes with rich textures but complex geometry.

[0104] Specific implementation steps; Step S810: Obtain the baseline model; Using sequence images collected by drones and sparse point clouds calculated by COLMAP, an uncompressed raw 3DGS model was trained on the training end. At this time, the model size was about 500MB and contained millions of Gaussian primitives.

[0105] Step S820: Construct a feature point-aware proxy; To address the sensitivity of the ORB-SLAM3 algorithm to corner points and texture gradients, a pre-trained Super Point network was loaded as the teacher network.

[0106] The rendered image from the current perspective Truth Plot Enter Super Point; Calculate the L2 loss and cosine similarity loss of the two on the feature point heatmap and descriptor, and use them as the task proxy loss. .

[0107] Step S830: Generate SLAM sensitivity map; Using automatic differential calculation The position of each Gaussian element in the scene and opacity The gradient.

[0108] The system identifies high-frequency geometric areas such as corners, table edges, and keyboard elements as having high gradients (high sensitivity), while smooth white walls and ceilings have gradients approaching zero (low sensitivity).

[0109] Step S840: Adaptive compression training; HyperNet is introduced to dynamically assign quantization parameters to each Gaussian unit based on the sensitivity value of the input. For highly sensitive Gaussian points, allocate 16-bit precision; for low-sensitivity points, allocate 4-bit precision or even prune directly. Set weights Joint training of three objective functions is performed.

[0110] In the second example, this embodiment simulates an indoor service robot in a home or office setting, which needs to store a 3D map of the entire house and perform semantic understanding and obstacle avoidance.

[0111] Implementation environment hardware platform: Model training end (cloud server): Configure NVIDIA A100 GPU cluster.

[0112] Inference and Application End (Robot Chassis): Mobile robot chassis equipped with Qualcomm Snapdragon or NVIDIA Jetson AGX Orin (such as TurtleBot4 or self-developed AGV).

[0113] Software and data environment: Operating system: RobotOperatingSystem2 (ROS2Humble).

[0114] Simulation environment: Habitat-Sim or Isaac Sim simulator.

[0115] Downstream task algorithms: Mask2Former (for panorama segmentation) + ROSNav2NavigationStack (for path planning).

[0116] Dataset: Select Scan Net or Matterport 3D datasets, which contain rich semantic categories.

[0117] Specific implementation steps; Step S910: Obtain the baseline model; The indoor environment scanned by the service robot was reconstructed using 3DGS. The original scene contained complex interior textures.

[0118] Step S920: Construct a semantically aware agent; Considering the requirement for navigation tasks to distinguish between passable areas and obstacle boundaries, DINOv2 (ViT-L / 14) was selected as the teacher network. Leveraging DINOv2's powerful semantic clustering capabilities, deep semantic features (CLSToken and PatchTokens) are extracted from the rendered graph and the ground truth graph. Construct a semantic consistency loss function that primarily penalizes regions where object categories are confused.

[0119] Step S930: Generate semantic sensitivity map and enhance boundaries; Calculate the gradient distribution of the semantic loss; Findings: Texture variations inside an object have minimal impact on semantic classification, resulting in very low gradients; while gradients at the object's edges are extremely high. Strategy: Generate a sensitivity map centered on semantic boundaries.

[0120] Step S940: HyperNet policy generation; HyperNet executes an edge-protected compression strategy based on the semantic sensitivity map.

[0121] Operation: Aggressively prune redundant Gaussian points inside the object, but retain high-density, high-precision Gaussian points at the object's edges to prevent navigation misjudgments caused by jagged edges.

[0122] like Figure 5 As shown in one embodiment of this application, a task-aware 3DGS scene data compression device includes: The instruction response module 1001 is used to respond to Gaussian compression instructions and determine the data compression model, the 3DGS model, and multiple Gaussian elements of the 3DGS model; The data compression module 1002 is used to compress multiple Gaussian primitives according to a compression model; wherein, the process of compressing multiple Gaussian primitives according to the compression model includes: Obtain the rendered image and ground truth image of the 3DGS model from the current viewpoint, and extract the first feature map of the rendered image and the second feature map of the ground truth image based on the neural network. Based on the feature relationship between the first feature map and the second feature map, calculate the gradient of the Gaussian property corresponding to each Gaussian element; Based on the gradient of the Gaussian property corresponding to each Gaussian primitive, the importance of each Gaussian primitive to the task of the 3DGS model is calculated. Based on the hyperparameter generation network and its importance, predict the compression parameters corresponding to each Gaussian unit; Based on the compression parameters corresponding to each Gaussian element, the Gaussian properties corresponding to each Gaussian element are compressed.

[0123] It should be noted that the task-aware 3DGS scene data compression device provided in this embodiment is based on the same inventive concept as the task-aware 3DGS scene data compression method described above. Therefore, the content of the task-aware 3DGS scene data compression method described above is also applicable to the content of the task-aware 3DGS scene data compression device in this embodiment, and will not be repeated here.

[0124] like Figure 6One embodiment of this application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned task-aware 3DGS scene data compression method. The electronic device includes: At least one battery; At least one memory; At least one processor; At least one program; The program is stored in memory, and the processor executes at least one program to implement the above-described task-aware 3DGS scene data compression method of this disclosure.

[0125] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0126] The electronic devices according to embodiments of this application will now be described in detail.

[0127] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to implement a task-aware 3DGS scene data compression method according to an embodiment of this disclosure.

[0128] The input / output interface 1800 is used to implement information input and output. The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900); The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0129] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described task-aware 3DGS scene data compression method.

[0130] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0131] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0132] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0133] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated; that is, they may be located in one place or distributed across multiple network units. Some or all of the blocks can be selected to achieve the purpose of this embodiment according to actual needs.

[0134] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional blocks / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0135] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0136] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0137] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0138] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0139] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0140] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0141] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. A task-aware 3DGS scene data compression method, characterized in that, The method includes: In response to the Gaussian compression command, determine the data compression model, the 3DGS model, and multiple Gaussian elements of the 3DGS model; The plurality of Gaussian primitives are compressed according to the compression model; wherein, the process of compressing the plurality of Gaussian primitives according to the compression model includes: Obtain the rendered image and ground truth image of the 3DGS model from the current viewpoint, and extract the first feature map of the rendered image and the second feature map of the ground truth image based on the neural network. Based on the feature relationship between the first feature map and the second feature map, calculate the gradient of the Gaussian property corresponding to each Gaussian element; Based on the gradient of the Gaussian property corresponding to each Gaussian element, the importance of each Gaussian element to the task of the 3DGS model is calculated. Based on the hyperparameter generation network and the importance level, predict the compression parameters corresponding to each Gaussian unit; Based on the compression parameters corresponding to each Gaussian element, the Gaussian properties corresponding to each Gaussian element are compressed.

2. The task-aware 3DGS scene data compression method according to claim 1, characterized in that, The compression parameters include quantization step size and retention probability; The step of compressing the Gaussian attribute corresponding to each Gaussian element based on the compression parameters corresponding to each Gaussian element includes: Select the Gaussian elements whose retention probability is greater than a preset value; Based on the quantization step size of the selected Gaussian elements whose retention probabilities are greater than a preset value, the corresponding Gaussian attributes of the selected Gaussian elements with retention probabilities greater than a preset value are quantized to obtain quantization parameters.

3. The task-aware 3DGS scene data compression method according to claim 2, characterized in that, After selecting the Gaussian elements whose retention probability is greater than a preset value, the method further includes: Based on clustering algorithms or non-uniform quantizers, according to the quantization step size of the corresponding Gaussian elements whose retention probability is greater than a preset value, a quantization step size codebook and the index value corresponding to each element in the quantization step size codebook are mapped and generated. The step size of the selected Gaussian property corresponding to the Gaussian element whose retention probability is greater than the preset value is used to quantize the corresponding Gaussian property to obtain quantization parameters, including: Determine the target Gaussian element, which is any one of the Gaussian elements whose retention probability is greater than a preset value; Based on the target index value corresponding to the target Gaussian element, search for the corresponding target quantization step size in the quantization step size codebook; The Gaussian properties corresponding to the target Gaussian element are quantized according to the target quantization step size to obtain quantization parameters.

4. The task-aware 3DGS scene data compression method according to claim 3, characterized in that, After compressing the Gaussian property corresponding to each Gaussian element according to the compression parameters corresponding to each Gaussian element, the process further includes: The index values ​​of the selected Gaussian elements with retention probabilities greater than a preset value are entropy encoded to obtain the encoding parameters; The quantization step size codebook, the encoding parameters, and the quantization parameters are packaged and sent to the client. The client decodes the encoding parameters to obtain an index value. The quantization step size of the corresponding Gaussian unit is read from the quantization step size codebook based on the index value. The quantization parameters are decompressed based on the quantization step size to reconstruct the 3DGS scene based on the decompressed Gaussian unit.

5. A task-aware 3DGS scene data compression method according to claim 2, characterized in that, The feature relationship includes the sum of the L2 distance and cosine similarity between the first feature map and the second feature map.

6. The task-aware 3DGS scene data compression method according to claim 2, characterized in that, The loss function of the compression model includes: ； in, Let be the loss function of the compression model. For task proxy loss function, Let be the bitrate loss function. The distortion loss function; It can be any value between 0 and 1. for The weight, for The weights; The task proxy loss function includes the loss of the sum of the L2 distance and cosine similarity between the first feature map and the second feature map; The bitrate loss function includes the bitrate loss of the corresponding Gaussian units whose retention probability is greater than a preset value; wherein the bitrate is calculated from the quantization parameters using an entropy model; The distortion loss function includes the distortion loss between the first feature map and the second feature map.

7. The task-aware 3DGS scene data compression method according to claim 1, characterized in that, The step of calculating the importance of each Gaussian element to the task of the 3DGS model based on the gradient of the Gaussian property corresponding to each Gaussian element includes: Based on the gradient of the Gaussian property corresponding to each Gaussian element, calculate the cumulative magnitude of the gradient of each Gaussian element under all views; The importance of each Gaussian element to the task of the 3DGS model is determined based on the cumulative modulus of each Gaussian element.

8. A task-aware 3DGS scene data compression device, characterized in that, The device includes: The instruction response module is used to respond to Gaussian compression instructions and determine the data compression model, the 3DGS model, and multiple Gaussian elements of the 3DGS model. A data compression module is used to compress the plurality of Gaussian primitives according to the compression model; wherein, the process of compressing the plurality of Gaussian primitives according to the compression model includes: Obtain the rendered image and ground truth image of the 3DGS model from the current viewpoint, and extract the first feature map of the rendered image and the second feature map of the ground truth image based on the neural network. Based on the feature relationship between the first feature map and the second feature map, calculate the gradient of the Gaussian property corresponding to each Gaussian element; Based on the gradient of the Gaussian property corresponding to each Gaussian element, the importance of each Gaussian element to the task of the 3DGS model is calculated. Based on the hyperparameter generation network and the importance level, predict the compression parameters corresponding to each Gaussian unit; Based on the compression parameters corresponding to each Gaussian element, the Gaussian properties corresponding to each Gaussian element are compressed.

9. An electronic device, characterized in that, It includes at least one controller and a memory for communicatively connecting with the controller; the memory stores instructions executable by the at least one controller, which, when executed by the at least one controller, causes the at least one controller to perform a task-aware 3DGS scene data compression method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions for causing a computer to perform a task-aware 3DGS scene data compression method as described in any one of claims 1 to 7.

Images