A multi-modal three-dimensional gaussian attribute management method and system based on attribute decoupling

Abstract

The application discloses a kind of multi-modal three-dimensional Gaussian attribute management method and system based on attribute decoupling in the technical field of three-dimensional scene reconstruction and rendering under multi-modal supervision.The method comprises the following steps: based on the obtained scene initial Gaussian cell set, according to the conflict degree of the color loss gradient and the geometry loss gradient of each Gaussian cell, the scene initial Gaussian cell set is divided into three kinds of Gaussian sets;Among them, the three kinds of Gaussian sets include general Gaussian set, color activated Gaussian set and geometry activated Gaussian set;According to the three kinds of Gaussian sets and the preset training sample set, iterative optimization operation is performed, and the delete and division operation is performed once every preset number of iterations.After the iteration number reaches the preset iteration number threshold, the final Gaussian cell set is output.The method eliminates the gradient conflict under multi-modal supervision by attribute decoupling and separation rendering, and realizes compact scene representation and high-fidelity rendering by adapting the task decoupling deletion of separation rendering architecture.

Description

Technical Field

[0001] This invention relates to the field of 3D scene reconstruction and rendering technology under multimodal supervision, and particularly to a multimodal 3D Gaussian attribute management method and system based on attribute decoupling. Background Technology

[0002] Early 3D reconstruction methods typically relied on multi-view geometric consistency or the introduction of prior knowledge for scene completion. However, due to the ambiguity of image feature correspondences, the reconstruction accuracy could not meet high-fidelity requirements. Methods based on neural radiation fields utilize multilayer perceptrons to model radiation fields, demonstrating strong capabilities in novel perspective synthesis tasks. However, the implicit representation still limits the compactness and editability of scene representations. 3D Gaussian splashing methods instead employ anisotropic Gaussian primitives for explicit scene representation, combined with fast rasterization based on graphics processing units, achieving real-time, high-quality rendering. However, the overly simplistic management process also leads to a large number of Gaussian primitives generated during the fitting of scene details, resulting in high representation redundancy. Methods such as structured Gaussian representation based on sparse anchor points alleviate the redundancy problem to some extent, but their focus on color appearance during modeling, lacking effective utilization of geometric information, limits their scalability in applications requiring precise surface information.

[0003] In surface reconstruction and representation, early radiation field-based methods typically employed heuristic density thresholding for surface extraction, which easily introduced noise and artifacts. Methods based on signed distance fields, by modeling density as a signed distance function, can generate higher-fidelity surfaces; however, these methods still face high computational costs during rendering. Recently, surface reconstruction methods based on 3D Gaussian splashing have received widespread attention. Some methods constrain the shape of Gaussian primitives to align with the object's surface and achieve efficient extraction, while others flatten Gaussian primitives into unambiguous surface elements and use Poisson surface reconstruction to obtain surface results. Although these methods achieve high-fidelity geometric surfaces, they sacrifice some accuracy in appearance reconstruction.

[0004] In summary, existing 3D Gaussian splashing and its derivatives suffer from conflicting gradient update directions for Gaussian primitive attributes in multimodal supervised joint training scenarios, such as simultaneously using color images and geometric graphs (e.g., normal vector maps). This conflict disrupts the optimization process for Gaussian primitive attributes, leading to a decline in the quality of appearance or geometric reconstruction. Furthermore, due to the varying requirements for the number and distribution of Gaussian primitives across different supervisory modalities, simple global deletion strategies struggle to balance multimodal reconstruction accuracy with the compactness of scene representation. Therefore, a method is urgently needed to resolve the gradient conflict problem generated by 3D Gaussian primitives under multimodal supervision, thereby achieving high-fidelity appearance and geometric reconstruction while maintaining a compact scene representation. Summary of the Invention

[0005] The purpose of this invention is to overcome the technical problems of existing 3D Gaussian splashing methods, which suffer from conflicts in the gradient update directions of Gaussian meta-attributes by the loss functions of different modalities during multimodal supervised joint training, leading to a decline in the quality of appearance reconstruction or geometric reconstruction, and the difficulty of balancing multimodal reconstruction accuracy and scene representation compactness with simple global deletion strategies. This invention provides a multimodal 3D Gaussian attribute management method and system based on attribute decoupling. By decoupling attributes and separating rendering, gradient conflicts under multimodal supervision are eliminated, and compact scene representation and high-fidelity rendering are achieved through task-based decoupling deletion adapted to the separating rendering architecture.

[0006] To solve the above-mentioned technical problems, the present invention is implemented using the following technical solution:

[0007] In a first aspect, the present invention provides a multimodal 3D Gaussian attribute management method based on attribute decoupling, comprising:

[0008] Based on the obtained initial Gaussian set of the scene, according to the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian set, the initial Gaussian set of the scene is divided into three types of Gaussian sets; wherein, the three types of Gaussian sets include a general Gaussian set, a color activation Gaussian set, and a geometric activation Gaussian set.

[0009] Based on the three types of Gaussian sets and the preset training sample set, the optimization operation is performed iteratively. Each iteration performs a deletion and partitioning operation once after a preset number of iterations. After the number of iterations reaches the preset iteration threshold, the final Gaussian set is output.

[0010] The optimization operations include:

[0011] The general Gaussian set and the color-activated Gaussian set are used to control the rendering of the color image together, and the general Gaussian set and the geometric-activated Gaussian set are used to render the geometric image together, so as to obtain the color image and the geometric image of the rendering output.

[0012] The color loss is calculated using the rendered color image and the corresponding ground truth color image, the geometric loss is calculated using the rendered geometric image and the corresponding ground truth geometric image, and the attribute values ​​of each Gaussian unit are optimized based on the color loss and the geometric loss.

[0013] The deletion and partitioning operations include:

[0014] Calculate the task-specific contribution value of each Gaussian element on the pixels it participates in rendering. Sort the three types of Gaussian sets according to the task-specific contribution value. Delete the Gaussian elements with the lowest contribution value of the preset proportion in the three types of Gaussian sets and then merge them to obtain the Gaussian element set.

[0015] Based on the Gaussian set, the Gaussian set is divided into three types of Gaussian sets according to the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian element.

[0016] Optionally, the initial Gaussian set of the scene is obtained through the following steps:

[0017] Sparse point cloud data is obtained from multi-view images of a scene using the structure-reconstruction-motion method.

[0018] An initial set of Gaussian elements for the scene is constructed based on the sparse point cloud data; wherein each Gaussian element corresponds to a point in the sparse point cloud data.

[0019] Optionally, the Gaussian element includes basic attributes and task-specific attributes;

[0020] The basic attributes include mean, rotation component, scale component, and opacity; wherein, the mean is the position center of the Gaussian element in three-dimensional space, the rotation component is a quaternion representing the orientation of the Gaussian element, the scale component represents the scale of the two tangential principal axes of the Gaussian element, and the opacity represents the opacity of the Gaussian element.

[0021] The task-specific attributes include color attributes and geometric attributes; wherein, the color attributes are three-channel color values, and the geometric attributes are three-channel normal vector values.

[0022] Optionally, the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian unit is quantified by the following steps:

[0023] Calculate the gradient vector of the basic properties of each Gaussian unit with respect to the color loss function, and the gradient vector with respect to the geometric loss function;

[0024] Iterate through each Gaussian element in the initial Gaussian element set of the scene, and take the currently traversed Gaussian element as the first Gaussian element, and mark the first Gaussian element as the processed primitive.

[0025] For the first Gaussian primitive, traverse the other Gaussian primitives in the initial Gaussian primitive set of the scene that are not marked as processed primitives, and take the currently traversed unmarked Gaussian primitive as the second Gaussian primitive, and form a Gaussian primitive pair with the first Gaussian primitive and the second Gaussian primitive.

[0026] For any Gaussian pair, calculate the normalized inner product of the gradient vectors of the two Gaussian elements in the pair, and use this normalized inner product as the quantification value of the degree of conflict between the two Gaussian elements.

[0027] Optionally, the gradient vector of the basic properties of each Gaussian unit with respect to the color loss function is obtained by the following formula:

[0028] ,

[0029] in, Represents the color loss function. This represents the gradient vector of the basic properties of the Gaussian unit with respect to the color loss function. This represents the concatenation function. This represents the basic attribute index of the Gaussian meta-element. This represents the mean of the Gaussian elements. Represents the rotational component of the Gaussian element. Represents the scale component of the Gaussian element. This indicates the opacity of the Gaussian unit. Represents the basic properties of Gaussian elements The gradient relative to the color loss function;

[0030] The gradient vector of the basic properties of each Gaussian primitive with respect to the geometric loss function is obtained by the following formula:

[0031] ,

[0032] in, Represents the geometric loss function. This represents the gradient vector of the basic properties of the Gaussian primitives with respect to the geometric loss function. Represents the basic properties of Gaussian elements The gradient relative to the geometric loss function;

[0033] The normalized inner product is obtained using the following formula:

[0034] ,

[0035] in, Represents the normalized inner product. This indicates the operation of taking the modulus of a vector.

[0036] Optionally, based on the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian unit, the initial Gaussian set of the scene is divided into three types of Gaussian sets, including:

[0037] For any quantification value of conflict level satisfy Gaussian separation is performed on the Gaussian Gaussian units. Based on the separation results, the general Gaussian units are divided into the general Gaussian set, the color-activated Gaussian units are divided into the color-activated Gaussian set, and the geometrically activated Gaussian units are divided into the geometrically activated Gaussian set.

[0038] For any quantification value of conflict level satisfy Furthermore, Gaussian elements that do not satisfy the compaction condition are classified into the universal Gaussian set;

[0039] For Gaussian elements in the general Gaussian set, color-activated Gaussian set, and geometrically activated Gaussian set that satisfy the compaction condition, perform cloning and splitting operations, and assign the Gaussian elements obtained by cloning and splitting to the set where the original Gaussian element was located.

[0040] The Gaussian separation operation includes:

[0041] If the color loss gradient magnitude of the Gaussian unit is satisfy And the magnitude of the geometric loss gradient satisfy The Gaussian element is separated into a general Gaussian element and a color-activated Gaussian element; where, This represents the operation of taking the modulus of a vector. To preset the first threshold, To preset the second threshold, the general Gaussian element retains the basic attributes, color attributes, and geometric attributes of the Gaussian element, and the color-activated Gaussian element retains the basic attributes and color attributes of the Gaussian element;

[0042] If the color loss gradient magnitude of the Gaussian unit is satisfy And the magnitude of the geometric loss gradient satisfy The Gaussian element is separated into a general Gaussian element and a geometrically activated Gaussian element; wherein the geometrically activated Gaussian element retains the basic and geometric properties of the Gaussian element.

[0043] If the color loss gradient magnitude of the Gaussian unit is satisfy And the magnitude of the geometric loss gradient satisfy The Gaussian element is separated into a color-activated Gaussian element and a geometric-activated Gaussian element.

[0044] Optionally, the rendering of the color image by controlling the general Gaussian set and the color-activated Gaussian set together using the indicator function is achieved through the following formula:

[0045] ,

[0046] in, This represents the color image of the rendered output. This represents the total number of Gaussian elements in the general Gaussian set and the color-activated Gaussian set. Indicates the first The color attribute of each high-order element. Indicates the first Alpha weights of Gaussian elements, Indicates the first The transmittance of each high-speed element, , Indicates the first Alpha weights of Gaussian elements, Indicates the first A high-ranking dollar, Indicates an indicator function, express Participating in the current rendering, express Not involved in the current rendering. Represents the universal Gaussian set, Indicates the color activation Gaussian set;

[0047] The rendering of the geometric image by controlling the general Gaussian set and the geometrically activated Gaussian set together using an indicator function is achieved through the following formula:

[0048] ,

[0049] in, This represents the geometric image of the rendered output. This represents the total number of Gaussian elements in the universal Gaussian set and the geometrically activated Gaussian set. express The cumulative transmittance of each Gaussian element, Indicates the first Geometric properties of Gaussian elements, Indicates the first Alpha weights of Gaussian elements, Indicates the first The transmittance of each high-speed element, Represents the geometrically activated Gaussian set. Indicates the first A high-ranking dollar, express Participating in the current rendering, express It is not involved in the current rendering.

[0050] Optionally, the task-specific contribution value is achieved using the following formula:

[0051] ,

[0052] in, Indicates Gaussian element, Gausky element The set of all pixels involved in rendering. express The total number of pixels in Gausky element In pixels Alpha weights on Gausky element In pixels Transmittance on.

[0053] Optionally, the color loss is obtained using the following color loss function:

[0054] ,

[0055] in, Represents the color loss function. This represents the color image of the rendered output. For true color images, Describes the L1 loss function. Represents the D-SSIM loss function;

[0056] The geometric loss is obtained through the following geometric loss function:

[0057] ,

[0058] in, Represents the geometric loss function. This represents the geometric image of the rendered output. Represents the true geometric graph, Represents the cosine operation;

[0059] The optimization of the attribute values ​​of each Gaussian unit based on the color loss and the geometric loss is achieved by the following formula:

[0060] ,

[0061] in, Indicates the first The attribute values ​​of Gaussian elements in the next iteration. Indicates the first The attribute values ​​of Gaussian elements in the next iteration. This refers to the Adam optimizer. This represents the attribute values ​​of Gaussian elements relative to the loss function. gradient vector, .

[0062] Secondly, this invention provides a multimodal 3D Gaussian attribute management system based on attribute decoupling, comprising:

[0063] The partitioning module is used to divide the initial Gaussian set of the scene into three types of Gaussian sets based on the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian set; wherein the three types of Gaussian sets include a general Gaussian set, a color activation Gaussian set, and a geometric activation Gaussian set.

[0064] The core iteration module is used to iteratively perform optimization operations based on the three types of Gaussian sets and the preset training sample set. Each iteration performs a deletion and partitioning operation once after a preset number of iterations. After the number of iterations reaches a preset iteration threshold, the final Gaussian element set is output.

[0065] The optimization operation module is used to control the general Gaussian set and the color-activated Gaussian set to jointly perform color image rendering and the general Gaussian set and the geometric-activated Gaussian set to jointly perform geometric image rendering, so as to obtain the rendered color image and geometric image; calculate the color loss using the rendered color image and the corresponding ground truth color image, calculate the geometric loss using the rendered geometric image and the corresponding ground truth geometric image, and optimize the attribute values ​​of each Gaussian unit based on the color loss and the geometric loss;

[0066] The deletion and partitioning module is used to calculate the task-specific contribution value of each Gaussian element on the pixels it participates in rendering, sort them according to the task-specific contribution value within the three types of Gaussian sets, delete the Gaussian elements with the lowest contribution value of a preset proportion in the three types of Gaussian sets, and then merge them to obtain a Gaussian element set; based on the Gaussian element set, according to the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian element, the Gaussian element set is divided into three types of Gaussian sets.

[0067] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0068] 1. This invention separates Gaussian primitives into three types with different attributes—general Gaussian primitives, color-activated Gaussian primitives, and geometric-activated Gaussian primitives—through attribute decoupling, thereby enabling them to participate in different rendering processes. By fully utilizing the characteristics of Gaussian primitives, the gradient conflict problem caused by dual supervision during the optimization process is eliminated.

[0069] 2. The attribute decoupling proposed in this invention results in the existence of three types of Gaussian primitives in the scene, including general, color-activated, and geometric-activated primitives. The separation rendering method based on indicator functions ensures that different types of Gaussian primitives only participate in reasonable forward rendering and receive gradient signals from their corresponding modalities. Its advantage lies in the strict consistency between the gradient backpropagation path and the rendering calculation process graph. If the rendering contribution of a certain type of Gaussian primitive in a certain modal rendering is zero, then the gradient of the loss function of that modality for that Gaussian primitive is always zero. Thus, the attribute decoupling process is effectively executed and guaranteed.

[0070] 3. The decoupled deletion strategy proposed in this invention performs deletion operations independently within each Gaussian set. Specifically, the color-activated Gaussian set only serves color rendering, and its deletion strategy is based solely on color rendering contributions, unaffected by geometric requirements. The same applies to the geometry-activated Gaussian set. The general Gaussian set serves bimodality, and its deletion decision is based on comprehensive contributions, retaining Gaussian primitives that are important in both color and geometry rendering. The deletion ratio of each set can be set independently, allowing different representational resources to be allocated to appearance and geometry according to actual application requirements. Through the task-based decoupled deletion strategy, the total number of Gaussian primitives is effectively controlled while ensuring reconstruction accuracy, achieving a compact scene representation. Attached Figure Description

[0071] Figure 1 The flowchart illustrates a multimodal 3D Gaussian attribute management method based on attribute decoupling, as provided in an embodiment of the present invention. Detailed Implementation

[0072] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments and specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations thereof. In the absence of conflict, the embodiments and technical features in the embodiments can be combined with each other.

[0073] It should be noted that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0074] Example 1

[0075] This invention discloses a multimodal 3D Gaussian attribute management method based on attribute decoupling, referencing... Figure 1 As shown, the specific steps include the following:

[0076] S1. Based on the obtained initial Gaussian set of the scene, the initial Gaussian set of the scene is divided into three types of Gaussian sets according to the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian set; wherein, the three types of Gaussian sets include a general Gaussian set, a color activation Gaussian set, and a geometric activation Gaussian set.

[0077] S2, based on the three types of Gaussian sets and the preset training sample set, perform optimization operations iteratively. Perform deletion and partitioning operations once every preset number of iterations. After the number of iterations reaches the preset iteration threshold, output the final Gaussian element set.

[0078] S2.1, the optimization operation includes:

[0079] S2.1.1, using an indicator function to control the general Gaussian set and the color-activated Gaussian set to jointly execute the rendering of the color image, and the general Gaussian set and the geometric-activated Gaussian set to jointly execute the rendering of the geometric image, to obtain the rendered color image and geometric image;

[0080] S2.1.2, calculate the color loss using the color image output by the rendering and the corresponding ground truth color image, calculate the geometric loss using the geometric image output by the rendering and the corresponding ground truth geometric image, and optimize the attribute values ​​of each Gaussian unit based on the color loss and the geometric loss.

[0081] S2.2, the deletion and partitioning operations include:

[0082] S2.2.1 Calculate the task-specific contribution value of each Gaussian element on the pixels it participates in rendering. Sort the three types of Gaussian sets according to the task-specific contribution value. Delete the Gaussian elements with the lowest contribution value in the three types of Gaussian sets and then merge them to obtain a set of Gaussian elements.

[0083] S2.2.2 Based on the Gaussian set, the Gaussian set is divided into three types of Gaussian sets according to the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian element.

[0084] In step S1, sparse point cloud data is obtained from multi-view images of the scene using the structure-of-motion method, with each point in the sparse point cloud data serving as the mean of the initial Gaussian unit. And initialize basic and task-specific properties for each Gaussian meta-element.

[0085] The basic properties of each Gaussian unit include: mean. , represents the center of the Gaussian element in three-dimensional space. Represents the real number field; rotational components Orientation of Gaussian elements is represented in quaternion form; scale components , representing the scaling of the Gaussian element along its two principal axes; opacity , representing the opacity of the Gaussian element.

[0086] Each Gaussian unit's task-specific attributes include: color attribute , which are three-channel color values; geometric properties , where is the three-channel normal vector value. The aforementioned task-specific attributes will be selectively activated or deactivated in subsequent steps based on the degree of gradient conflict.

[0087] In the multimodal supervised joint training process, the appearance loss function and geometric loss function Gradients are generated for the basic properties of Gaussian elements respectively; when the gradients are in opposite directions, sharing the same Gaussian element will cause the optimization directions to cancel each other out, resulting in a decrease in reconstruction quality; among them, the appearance loss function With geometric loss function These represent the fitting error loss between the rendered result and the true value, respectively.

[0088] To quantify the degree of gradient conflict, this embodiment calculates the basic properties of each Gaussian unit relative to the appearance loss function. gradient vector and relative to the geometric loss function gradient vector Among them, the gradient vector The acquisition process is as follows: extract each component from the basic attributes. , , , Relative to loss The gradients are extracted separately, concatenated, and flattened into a one-dimensional vector, i.e.:

[0089] ,

[0090] in, Represents the color loss function. This represents the gradient vector of the basic properties of the Gaussian unit with respect to the color loss function. This represents the concatenation function. This represents the basic attribute index of the Gaussian meta-element. This represents the mean of the Gaussian elements. Represents the rotational component of the Gaussian element. Represents the scale component of the Gaussian element. This indicates the opacity of the Gaussian unit. Represents the basic properties of Gaussian elements The gradient relative to the color loss function;

[0091] Subsequently, the normalized inner product of the two gradient vectors is calculated. As a quantitative indicator of the degree of gradient conflict:

[0092] ,

[0093] in, Represents the normalized inner product. This represents the operation of taking the modulus of a vector;

[0094] when When the gradient of appearance loss is opposite to that of geometric loss, it is determined that there is a gradient conflict in the Gaussian unit.

[0095] Based on the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian element, the initial Gaussian element set of the scene is divided into three types of Gaussian sets, including:

[0096] For any quantification value of conflict level satisfy Gaussian elements perform Gaussian separation operations:

[0097] If the color loss gradient magnitude of the Gaussian unit is satisfy And the magnitude of the geometric loss gradient satisfy This indicates that the Gaussian primitive is primarily driven by appearance loss. Therefore, the Gaussian primitive is separated into a general Gaussian primitive and a color-activated Gaussian primitive; among them, This represents the operation of taking the modulus of a vector. To preset the first threshold, In this embodiment, a second threshold is preset. and The specific value can be set according to the complexity of the scenario and the training strategy; the default value is [value to be filled in]. The general Gaussian unit retains the basic, color, and geometric attributes of the Gaussian unit; the color-activated Gaussian unit retains the basic and color attributes of the Gaussian unit; the separated general Gaussian unit retains the basic, color, and geometric attributes; the color-activated Gaussian unit retains the basic and color attributes, and the geometric attributes are set to an inactive state.

[0098] If the color loss gradient magnitude of the Gaussian unit is satisfy And the magnitude of the geometric loss gradient satisfy This indicates that the Gaussian primitive is mainly driven by geometric loss. The Gaussian primitive is then separated into a general Gaussian primitive and a geometrically activated Gaussian primitive. The geometrically activated Gaussian primitive retains the basic and geometric properties of the original Gaussian primitive. The separated general Gaussian primitive retains the basic, color, and geometric properties. The geometrically activated Gaussian primitive retains the basic and geometric properties, while the color property is set to an inactive state.

[0099] If the color loss gradient magnitude of the Gaussian unit is satisfy And the magnitude of the geometric loss gradient satisfy This indicates that the Gaussian primitive is strongly driven by both appearance loss and geometric loss. The Gaussian primitive is then separated into a color-activated Gaussian primitive and a geometric-activated Gaussian primitive. The separated color-activated Gaussian primitive retains the basic and color attributes, while the geometric attributes are set to an inactive state. The geometric-activated Gaussian primitive retains the basic and geometric attributes, while the color attributes are set to an inactive state.

[0100] Based on the separation results, the general Gaussian unit is assigned to the general Gaussian set, the color-activated Gaussian unit is assigned to the color-activated Gaussian set, and the geometric-activated Gaussian unit is assigned to the geometric-activated Gaussian set.

[0101] For any quantification value of conflict level satisfy Furthermore, Gaussian elements that do not satisfy the compaction condition are classified into the universal Gaussian set;

[0102] For Gaussian elements in the general Gaussian set, color-activated Gaussian set, and geometrically activated Gaussian set that satisfy the compaction condition, perform cloning and splitting operations, and assign the Gaussian elements obtained by cloning and splitting to the set where the original Gaussian element was located.

[0103] After the above partitioning operation, the scene is represented by the following three disjoint Gaussian sets:

[0104] Universal Gaussian set , which includes Gaussian elements that possess both color and geometric attributes;

[0105] Color-activated Gaussian set It contains Gaussian elements that only have color attributes, while geometric attributes are inactive.

[0106] Geometrically activated Gaussian sets It contains Gaussian elements that only have geometric properties, while the color property is inactive.

[0107] In step S2.1.1, during the rendering stage, to adapt to the aforementioned attribute decoupling and Gaussian set partitioning, this embodiment adopts a separate rendering architecture by introducing an indicator function. Controlling each Gaussian set to participate only in the rendering process of its corresponding modality, the indicator function... Its function is: when Gausky element The value is 1 if it belongs to the Gaussian set participating in the current rendering mode, and 0 otherwise; thus, the color-activated Gaussian set... Gaussian elements in the geometric normal map are excluded when rendering the geometric activation Gaussian set. In this model, Gaussian elements are excluded when rendering color images and depth maps, thus structurally ensuring the independence of each modal rendering process.

[0108] The rendering of a color image is controlled by an indicator function using the general Gaussian set and the color-activated Gaussian set through the following formula:

[0109] ,

[0110] in, This represents the color image of the rendered output. This represents the total number of Gaussian elements in the general Gaussian set and the color-activated Gaussian set. Indicates the first The color attribute of each high-order element. Indicates the first Alpha weights of Gaussian elements, Indicates the first The transmittance of each high-speed element, , Indicates the first Alpha weights of Gaussian elements, Indicates the first A high-ranking dollar, Indicates an indicator function, express Participating in the current rendering, express Not involved in the current rendering. Represents the universal Gaussian set, Indicates the color activation Gaussian set;

[0111] The rendering of the geometric image is controlled by the indicator function using the general Gaussian set and the geometrically activated Gaussian set through the following formula:

[0112] ,

[0113] in, This represents the geometric image of the rendered output. This represents the total number of Gaussian elements in the universal Gaussian set and the geometrically activated Gaussian set. express The cumulative transmittance of each Gaussian element, Indicates the first Geometric properties of Gaussian elements, Indicates the first Alpha weights of Gaussian elements, Indicates the first The transmittance of each high-speed element, Represents the geometrically activated Gaussian set. Indicates the first A high-ranking dollar, express Participating in the current rendering, express It is not involved in the current rendering.

[0114] In step S2.1.2, the loss is calculated based on the rendered color image and geometric image, respectively, using the ground truth color image and ground truth geometric image. The ground truth color image is an RGB image, and the ground truth geometric image is a normal vector map.

[0115] The color loss is obtained by combining L1 loss and D-SSIM loss, and is derived using the following color loss function:

[0116] ,

[0117] in, Represents the color loss function. This represents the color image of the rendered output. For true color images, Describes the L1 loss function. Represents the D-SSIM loss function;

[0118] The geometric loss uses the cosine similarity loss of the normal vector, and is obtained through the following geometric loss function:

[0119] ,

[0120] in, Represents the geometric loss function. This represents the geometric image of the rendered output. Represents the true geometric graph, Represents the cosine operation;

[0121] The aforementioned loss is backpropagated to optimize and update the properties of each Gaussian element:

[0122] ,

[0123] in, Indicates the first The attribute values ​​of Gaussian elements in the next iteration. Indicates the first The attribute values ​​of Gaussian elements in the next iteration. This refers to the Adam optimizer. This represents the attribute values ​​of Gaussian elements relative to the loss function. gradient vector, .

[0124] In step S2.2.1, since each Gaussian set serves a different rendering modality after attribute decoupling, a unified sorting and deletion across sets cannot meet the differentiated requirements of different modalities for Gaussian primitive distribution. Therefore, in this embodiment, the deletion operation is performed independently within each Gaussian set.

[0125] First, calculate the task-specific contribution value of each Gaussian element. This is used to measure the contribution of the Gaussian pixel to the pixels it participates in rendering:

[0126] ,

[0127] in, Indicates Gaussian element, Gausky element The set of all pixels involved in rendering. express The total number of pixels in Gausky element In pixels Alpha weights on Gausky element In pixels Transmittance on.

[0128] Then, in the universal Gaussian set respectively Color-activated Gaussian set and geometrically activated Gaussian sets Internally, based on task-specific contribution values Sort the Gaussian elements and remove the preset proportions with the lowest contribution values ​​from each Gaussian set. Gao Siyuan.

[0129] Through the above steps, this embodiment achieves the organic unity of attribute decoupling, separate rendering, and task-specific decoupling and deletion; it separates Gaussian primitives for different modal tasks through gradient conflict detection and attribute decoupling; it ensures that each Gaussian set only participates in the corresponding modal rendering through indicator functions; it uses the Gaussian set partitioning results brought about by attribute decoupling to independently evaluate and delete low-contribution Gaussian primitives within each Gaussian set; under the combined effect of these three, high-fidelity reconstruction and compact scene representation under multimodal supervision are achieved.

[0130] To verify the effectiveness of the method in this embodiment, Table 1 below shows a comparison of the experimental results of the method in this embodiment with the prior art on two public benchmark datasets, BlenderMVS and DTU. Table 2 below shows a comparison of the implementation results of the method in this embodiment with the prior art on two open scene benchmark datasets, DeepBlending and MipNeRF360.

[0131] Existing technologies include SuGaR (Surface-Aligned Gaussian Splatting for Efficient 3DMesh Reconstruction and High-Quality Mesh Rendering), GaussianSurfels (High-quality Surface Reconstruction using Gaussian Surfels), GSDF (3DGS Meets SDF for Improved Rendering and Reconstruction, combining 3D Gaussian splatting with SDF to improve rendering and reconstruction), and ProbeSDF (Light Field Probes for Neural Surface Reconstruction). Experimental comparison metrics include appearance rendering quality metrics such as peak signal-to-noise ratio (PSNR), structural similarity (SSIM), geometric reconstruction metrics such as chamfer distance (CD), learned perceptual image patch similarity (LPIPS), and model size (SIZE). The multimodal 3D Gaussian attribute management method based on attribute decoupling proposed in this embodiment fully demonstrates its effectiveness on the above four datasets.

[0132] Table 1: Comparison of experimental results between the method of this embodiment and existing technologies on two public benchmark datasets, BlenderMVS and DTU.

[0133]

[0134] Table 2: Comparison of the implementation results of the method in this embodiment with existing technologies on two open-scene benchmark datasets, DeepBlending and MipNeRF360.

[0135]

[0136] Example 2

[0137] Based on the same inventive concept as Embodiment 1, this embodiment of the invention discloses a multimodal 3D Gaussian attribute management system based on attribute decoupling, comprising:

[0138] The partitioning module is used to divide the initial Gaussian set of the scene into three types of Gaussian sets based on the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian set; wherein the three types of Gaussian sets include a general Gaussian set, a color activation Gaussian set, and a geometric activation Gaussian set.

[0139] The core iteration module is used to iteratively perform optimization operations based on the three types of Gaussian sets and the preset training sample set. Each iteration performs a deletion and partitioning operation once after a preset number of iterations. After the number of iterations reaches a preset iteration threshold, the final Gaussian element set is output.

[0140] The optimization operation module is used to control the general Gaussian set and the color-activated Gaussian set to jointly perform color image rendering and the general Gaussian set and the geometric-activated Gaussian set to jointly perform geometric image rendering, so as to obtain the rendered color image and geometric image; calculate the color loss using the rendered color image and the corresponding ground truth color image, calculate the geometric loss using the rendered geometric image and the corresponding ground truth geometric image, and optimize the attribute values ​​of each Gaussian unit based on the color loss and the geometric loss;

[0141] The deletion and partitioning module is used to calculate the task-specific contribution value of each Gaussian element on the pixels it participates in rendering, sort them according to the task-specific contribution value within the three types of Gaussian sets, delete the Gaussian elements with the lowest contribution value of a preset proportion in the three types of Gaussian sets, and then merge them to obtain a Gaussian element set; based on the Gaussian element set, according to the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian element, the Gaussian element set is divided into three types of Gaussian sets.

[0142] The specific functions of each module described above are explained in the relevant content of the method in Embodiment 1, and will not be repeated here.

[0143] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0144] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0145] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0146] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0147] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A multimodal 3D Gaussian attribute management method based on attribute decoupling, characterized in that, include: Based on the obtained initial Gaussian set of the scene, according to the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian set, the initial Gaussian set of the scene is divided into three types of Gaussian sets; wherein, the three types of Gaussian sets include a general Gaussian set, a color activation Gaussian set, and a geometric activation Gaussian set. Based on the three types of Gaussian sets and the preset training sample set, the optimization operation is performed iteratively. Each iteration performs a deletion and partitioning operation once after a preset number of iterations. After the number of iterations reaches the preset iteration threshold, the final Gaussian set is output. The optimization operations include: The general Gaussian set and the color-activated Gaussian set are used to control the rendering of the color image together, and the general Gaussian set and the geometric-activated Gaussian set are used to render the geometric image together, so as to obtain the color image and the geometric image of the rendering output. The color loss is calculated using the rendered color image and the corresponding ground truth color image, the geometric loss is calculated using the rendered geometric image and the corresponding ground truth geometric image, and the attribute values ​​of each Gaussian unit are optimized based on the color loss and the geometric loss. The deletion and partitioning operations include: Calculate the task-specific contribution value of each Gaussian element on the pixels it participates in rendering. Sort the three types of Gaussian sets according to the task-specific contribution value. Delete the Gaussian elements with the lowest contribution value of the preset proportion in the three types of Gaussian sets and then merge them to obtain the Gaussian element set. Based on the Gaussian set, the Gaussian set is divided into three types of Gaussian sets according to the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian element.

2. The multimodal 3D Gaussian attribute management method based on attribute decoupling according to claim 1, characterized in that, The initial Gaussian set of the scenario is obtained through the following steps: Sparse point cloud data is obtained from multi-view images of a scene using the structure-reconstruction-motion method. An initial set of Gaussian elements for the scene is constructed based on the sparse point cloud data; wherein each Gaussian element corresponds to a point in the sparse point cloud data.

3. The multimodal 3D Gaussian attribute management method based on attribute decoupling according to claim 1, characterized in that, The Gaussian element includes basic attributes and task-specific attributes; The basic attributes include mean, rotation component, scale component, and opacity; wherein, the mean is the position center of the Gaussian element in three-dimensional space, the rotation component is a quaternion representing the orientation of the Gaussian element, the scale component represents the scale of the two tangential principal axes of the Gaussian element, and the opacity represents the opacity of the Gaussian element. The task-specific attributes include color attributes and geometric attributes; wherein, the color attributes are three-channel color values, and the geometric attributes are three-channel normal vector values.

4. The multimodal 3D Gaussian attribute management method based on attribute decoupling according to claim 1, characterized in that, The degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian unit is quantified by the following steps: Calculate the gradient vector of the basic properties of each Gaussian unit with respect to the color loss function, and the gradient vector with respect to the geometric loss function; Iterate through each Gaussian element in the initial Gaussian element set of the scene, and take the currently traversed Gaussian element as the first Gaussian element, and mark the first Gaussian element as the processed primitive. For the first Gaussian primitive, traverse the other Gaussian primitives in the initial Gaussian primitive set of the scene that are not marked as processed primitives, and take the currently traversed unmarked Gaussian primitive as the second Gaussian primitive, and form a Gaussian primitive pair with the first Gaussian primitive and the second Gaussian primitive. For any Gaussian pair, calculate the normalized inner product of the gradient vectors of the two Gaussian elements in the pair, and use this normalized inner product as the quantification value of the degree of conflict between the two Gaussian elements.

5. The multimodal 3D Gaussian attribute management method based on attribute decoupling according to claim 4, characterized in that, The gradient vector of the basic properties of each Gaussian unit relative to the color loss function is obtained by the following formula: ， in, Represents the color loss function. This represents the gradient vector of the basic properties of the Gaussian unit with respect to the color loss function. This represents the concatenation function. This represents the basic attribute index of the Gaussian meta-element. This represents the mean of the Gaussian elements. Represents the rotational component of the Gaussian element. Represents the scale component of the Gaussian element. This indicates the opacity of the Gaussian unit. Represents the basic properties of Gaussian elements The gradient relative to the color loss function; The gradient vector of the basic properties of each Gaussian primitive with respect to the geometric loss function is obtained by the following formula: ， in, Represents the geometric loss function. This represents the gradient vector of the basic properties of the Gaussian primitives with respect to the geometric loss function. Represents the basic properties of Gaussian elements The gradient relative to the geometric loss function; The normalized inner product is obtained using the following formula: ， in, Represents the normalized inner product. This indicates the operation of taking the modulus of a vector.

6. The multimodal 3D Gaussian attribute management method based on attribute decoupling according to claim 4, characterized in that, Based on the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian element, the initial Gaussian element set of the scene is divided into three types of Gaussian sets, including: For any quantification value of conflict level satisfy Gaussian separation is performed on the Gaussian Gaussian units. Based on the separation results, the general Gaussian units are divided into the general Gaussian set, the color-activated Gaussian units are divided into the color-activated Gaussian set, and the geometrically activated Gaussian units are divided into the geometrically activated Gaussian set. For any quantification value of conflict level satisfy Furthermore, Gaussian elements that do not satisfy the compaction condition are classified into the universal Gaussian set; For Gaussian elements in the general Gaussian set, color-activated Gaussian set, and geometrically activated Gaussian set that satisfy the compaction condition, perform cloning and splitting operations, and assign the Gaussian elements obtained by cloning and splitting to the set where the original Gaussian element was located. The Gaussian separation operation includes: If the color loss gradient magnitude of the Gaussian unit is satisfy And the magnitude of the geometric loss gradient satisfy The Gaussian element is separated into a general Gaussian element and a color-activated Gaussian element; where, This represents the operation of taking the modulus of a vector. To preset the first threshold, To preset the second threshold, the general Gaussian element retains the basic attributes, color attributes, and geometric attributes of the Gaussian element, and the color-activated Gaussian element retains the basic attributes and color attributes of the Gaussian element; If the color loss gradient magnitude of the Gaussian unit is satisfy And the magnitude of the geometric loss gradient satisfy The Gaussian element is separated into a general Gaussian element and a geometrically activated Gaussian element; wherein the geometrically activated Gaussian element retains the basic and geometric properties of the Gaussian element. If the color loss gradient magnitude of the Gaussian unit is satisfy And the magnitude of the geometric loss gradient satisfy The Gaussian element is separated into a color-activated Gaussian element and a geometric-activated Gaussian element.

7. The multimodal 3D Gaussian attribute management method based on attribute decoupling according to claim 1, characterized in that, The rendering of the color image by controlling the general Gaussian set and the color-activated Gaussian set together using the indicator function is achieved through the following formula: ， in, This represents the color image of the rendered output. This represents the total number of Gaussian elements in the general Gaussian set and the color-activated Gaussian set. Indicates the first The color attribute of each high-order element. Indicates the first Alpha weights of Gaussian elements, Indicates the first The transmittance of each high-speed element, , Indicates the first Alpha weights of Gaussian elements, Indicates the first A high-ranking dollar, Indicates an indicator function, express Participating in the current rendering, express Not involved in the current rendering. Represents the universal Gaussian set, Indicates the color activation Gaussian set; The rendering of the geometric image by controlling the general Gaussian set and the geometrically activated Gaussian set together using an indicator function is achieved through the following formula: ， in, This represents the geometric image of the rendered output. This represents the total number of Gaussian elements in the universal Gaussian set and the geometrically activated Gaussian set. express The cumulative transmittance of each Gaussian element, Indicates the first Geometric properties of Gaussian elements, Indicates the first Alpha weights of Gaussian elements, Indicates the first The transmittance of each high-speed element, Represents a geometrically activated Gaussian set. Indicates the first A high-ranking dollar, express Participating in the current rendering, express It is not involved in the current rendering.

8. The multimodal 3D Gaussian attribute management method based on attribute decoupling according to claim 1, characterized in that, The specific contribution value for the task is achieved through the following formula: ， in, Indicates Gaussian element, Gausky element The set of all pixels involved in rendering. express The total number of pixels in Gausky element In pixels Alpha weights on Gausky element In pixels Transmittance on.

9. The multimodal 3D Gaussian attribute management method based on attribute decoupling according to claim 1, characterized in that, The color loss is obtained through the following color loss function: ， in, Represents the color loss function. This represents the color image of the rendered output. For true color images, Describes the L1 loss function. Represents the D-SSIM loss function; The geometric loss is obtained through the following geometric loss function: ， in, Represents the geometric loss function. This represents the geometric image of the rendered output. Represents the true geometric graph, Represents the cosine operation; The optimization of the attribute values ​​of each Gaussian unit based on the color loss and the geometric loss is achieved by the following formula: ， in, Indicates the first The attribute values ​​of Gaussian elements in the next iteration. Indicates the first The attribute values ​​of Gaussian elements in the next iteration. This refers to the Adam optimizer. This represents the attribute values ​​of Gaussian elements relative to the loss function. gradient vector, .

10. A multimodal 3D Gaussian attribute management system based on attribute decoupling, characterized in that, include: The partitioning module is used to divide the initial Gaussian set of the scene into three types of Gaussian sets based on the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian set; wherein the three types of Gaussian sets include a general Gaussian set, a color activation Gaussian set, and a geometric activation Gaussian set. The core iteration module is used to iteratively perform optimization operations based on the three types of Gaussian sets and the preset training sample set. Each iteration performs a deletion and partitioning operation once after a preset number of iterations. After the number of iterations reaches a preset iteration threshold, the final Gaussian element set is output. The optimization operation module is used to control the general Gaussian set and the color-activated Gaussian set to jointly perform color image rendering and the general Gaussian set and the geometric-activated Gaussian set to jointly perform geometric image rendering, so as to obtain the rendered color image and geometric image; calculate the color loss using the rendered color image and the corresponding ground truth color image, calculate the geometric loss using the rendered geometric image and the corresponding ground truth geometric image, and optimize the attribute values ​​of each Gaussian unit based on the color loss and the geometric loss; The deletion and partitioning module is used to calculate the task-specific contribution value of each Gaussian element on the pixels it participates in rendering, sort them according to the task-specific contribution value within the three types of Gaussian sets, delete the Gaussian elements with the lowest contribution value of a preset proportion in the three types of Gaussian sets, and then merge them to obtain a Gaussian element set; based on the Gaussian element set, according to the degree of conflict between the color loss gradient and the geometric loss gradient of each Gaussian element, the Gaussian element set is divided into three types of Gaussian sets.

Images