Semantic payload gaussian spatter modeling method, apparatus, device and storage medium

Abstract

This application discloses a method, apparatus, device, and storage medium for Gaussian splash modeling of semantic payloads, relating to the field of computer technology. The method includes: constructing a set of 3D Gaussian primitives based on a sparse point cloud of a scene; initializing the geometric parameters of each 3D Gaussian primitive in the set; iteratively updating the geometric parameters of the 3D Gaussian primitives through a differentiable rendering process until convergence, updating the 3D Gaussian primitives to geometrically optimal primitives; acquiring sample points of the scene and their semantic supervision information; calculating the interpretation weight of the geometrically optimal primitives for the sample points; combining the interpretation weight with the semantic supervision information of the sample points to obtain the semantic payload distribution of the geometrically optimal primitives; acquiring spatial points of the scene; calculating the interpretation weight of the geometrically optimal primitives for the spatial points; combining the interpretation weight with the semantic payload distribution of the geometrically optimal primitives to obtain and output the semantic posterior probability and semantic entropy of the spatial points.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a semantic load Gaussian splash modeling method, apparatus, device, and storage medium. Background Technology

[0002] In recent years, 3D reconstruction and scene modeling technologies have been widely applied in fields such as digital twins, autonomous driving, high-precision map construction, industrial inspection, power line inspection, and virtual reality. With the development of multi-view reconstruction and neural rendering technologies, 3D representation methods based on Gaussian mixture models, especially 3D Gaussian Splatting (3DGS) modeling, have gradually become an important technical route in the field of real-time 3D reconstruction due to their high rendering efficiency, high representation accuracy, and good differentiability. Existing methods mainly focus on improving geometric reconstruction accuracy, optimizing rendering efficiency, and enhancing visualization quality, while paying relatively little attention to the expression and utilization of semantic information within the model.

[0003] However, in practical applications, geometric structural information alone is often insufficient to meet the needs of industrial applications. For example, in power digital twin systems, it is not only necessary to construct high-precision 3D equipment models, but also to distinguish and identify semantic information such as equipment category, functional area, and risk level. In autonomous driving scenarios, semantic labels for vehicles, pedestrians, and road surfaces play a decisive role in the decision-making system. Therefore, how to enable the 3D model itself to have semantic expressive capabilities while maintaining the accuracy of geometric modeling has become an important issue in current technological development.

[0004] The following processing methods are commonly found in existing technologies.

[0005] Post-processing classification method: After completing geometric modeling, an independent classification network is trained to perform semantic prediction on spatial points or voxels. This method increases model complexity, and the decoupling of the semantic module from the geometric module leads to the need for dual-model collaboration during the inference stage, resulting in high computational costs.

[0006] Joint training coupling method: Semantic supervision signals are introduced during the geometric modeling stage, and geometric and semantic parameters are jointly optimized through a multi-task loss function. Although this method achieves semantic embedding, the simultaneous action of geometric optimization and semantic objectives on shared parameters often alters the optimal geometric solution, leading to a shift in the geometric edge distribution and affecting modeling accuracy.

[0007] Direct label attachment method: Simply binds existing labels to corresponding 3D Gaussian primitives. This method lacks probabilistic expressive power, cannot characterize semantic uncertainty, and does not establish a unified probabilistic structure of semantics and geometry from the perspective of generative models, resulting in insufficient innovation and theoretical completeness.

[0008] None of the above-mentioned methods provide a systematic solution to the problem of "geometry and semantics decoupling and strict losslessness" from the perspective of probabilistic generative models. Summary of the Invention

[0009] In view of this, the technical problem to be solved by this application is: how to achieve semantic embedding of 3DGS models without changing their geometric edge distribution.

[0010] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0011] Firstly, this application provides a semantic load Gaussian splashing modeling method, including:

[0012] Based on the sparse point cloud of the scene, a set of 3D Gaussian primitives is constructed, and the geometric parameters of each 3D Gaussian primitive in the set of 3D Gaussian primitives are initialized.

[0013] The geometric parameters of the 3D Gaussian primitive are iteratively updated through a differentiable rendering process until convergence, thereby updating the 3D Gaussian primitive to a geometrically optimal primitive.

[0014] Obtain sample points and their semantic supervision information in the scene, calculate the interpretation weight of the geometrically optimal primitive on the sample points, and process the interpretation weight and the semantic supervision information of the sample points to obtain the semantic load distribution of the geometrically optimal primitive.

[0015] Obtain spatial points of the scene, calculate the explanatory weight of the geometrically optimal primitive for the spatial points, and process the explanatory weight with the semantic load distribution of the geometrically optimal primitive to obtain the semantic posterior probability and semantic entropy of the spatial points and output them.

[0016] In one alternative embodiment of the first aspect, the iterative update of the geometric parameters of the 3D Gaussian primitive through a differentiable rendering process until convergence, thereby updating the 3D Gaussian primitive to a geometrically optimal primitive, includes:

[0017] The 3D Gaussian primitive is projected onto the image plane through a differentiable rendering process to generate a rendered image, and the gradient of the rendered image relative to the 3D Gaussian primitive is obtained.

[0018] Using the gradient, with the goal of minimizing the rendering error of the rendered image, the geometric parameters of the 3D Gaussian primitive are iteratively updated until convergence, and the 3D Gaussian primitive is updated to a geometrically optimal primitive.

[0019] In one alternative of the first aspect, the steps of acquiring sample points of the scene and their semantic supervision information, calculating the explanatory weight of the geometrically optimal primitive on the sample points, and combining the explanatory weight with the semantic supervision information of the sample points to obtain the semantic load distribution of the geometrically optimal primitive include:

[0020] Obtain sample points of the scene and their semantic supervision information, and calculate the explanatory weight of the geometrically optimal primitive on the sample points based on the contribution value of the geometrically optimal primitive to the sample points;

[0021] The semantic soft count of the geometrically optimal primitive is obtained by combining the explanatory weight of the geometrically optimal primitive on the sample point with the semantic supervision information of the sample point.

[0022] The semantic soft count of the geometrically optimal primitive is processed to obtain the normalized semantic load distribution of the geometrically optimal primitive.

[0023] In one alternative embodiment of the first aspect, the formula for calculating the explanatory weight of the geometrically optimal primitive for the sample point is:

[0024] ;

[0025] In the formula: Indicates the first One sample point; This represents the number of geometrically optimal primitives in the set of 3D Gaussian primitives; In the set of 3D Gaussian primitives, the first... The set of geometric parameters of a geometrically optimal primitive; Indicates the first The geometrically optimal primitive for the first Explanatory weights for each sample point; Indicates the first The geometrically optimal primitive for the first The contribution value of each sample point; For the first The contribution function of a geometrically optimal primitive to the sample point, variable The sample point is referred to here.

[0026] In one alternative to the first aspect, the formula for calculating the semantic soft count of the geometrically optimal primitive is:

[0027] ;

[0028] In the formula: This indicates the number of sample points; For the first The sample point about the th Semantic supervision information for various semantic categories; Indicates the first The geometrically optimal primitive for the first Semantic soft counts for each semantic category.

[0029] In one alternative of the first aspect, the semantic supervision information of the sample points is a hard label; if the first The semantics of the sample point belong to the first... semantic categories, then ;otherwise, .

[0030] In one alternative embodiment of the first aspect, the semantic supervision information of the sample points is a soft label; the soft label is the... The semantics of the nth sample point belong to the th The probability of a semantic category.

[0031] In one alternative to the first aspect, the formula for calculating the semantic load distribution of the geometrically optimal primitive is:

[0032] ;

[0033] In the formula: Indicates the first The geometrically optimal primitive for the first The distribution of semantic loads for each semantic category; Indicates the total number of semantic categories;

[0034] The semantic load distribution of the geometrically optimal primitive satisfies the normalization condition:

[0035] .

[0036] In one alternative to the first aspect, after processing the semantic soft count based on the geometrically optimal primitive to obtain a normalized semantic load distribution of the geometrically optimal primitive, the method further includes:

[0037] Acquire new sample points in the scene and their semantic supervision information, and calculate the explanatory weight of the geometrically optimal primitive on the new sample points based on the contribution value of the geometrically optimal primitive to the new sample points;

[0038] The explanatory weight of the geometrically optimal primitive for the newly added sample points is combined with the semantic supervision information of the newly added sample points to obtain the new semantic soft count of the geometrically optimal primitive;

[0039] The semantic soft count of the geometrically optimal primitive and the cumulative value of the newly added semantic soft count are processed to obtain the normalized semantic load distribution of the geometrically optimal primitive.

[0040] In one alternative of the first aspect, the steps of acquiring spatial points of the scene, calculating the explanatory weight of the geometrically optimal primitive for the spatial points, and combining the explanatory weight with the semantic load distribution of the geometrically optimal primitive to obtain and output the semantic posterior probability and semantic entropy of the spatial points include:

[0041] Obtain spatial points of the scene, calculate the explanatory weight of the geometrically optimal primitive for the spatial points, and make it satisfy non-negativity and normalization constraints;

[0042] The semantic load distribution of the geometrically optimal primitive is attached to the explanatory weight of the geometrically optimal primitive for the spatial point to obtain the semantic posterior probability of the spatial point;

[0043] The semantic posterior probability of the spatial point is processed to obtain the semantic entropy of the spatial point, and the semantic posterior probability and semantic entropy of the spatial point are output.

[0044] In one alternative embodiment of the first aspect, the formula for calculating the interpretation weight of the geometrically optimal primitive for the spatial point is as follows:

[0045] ;

[0046] In the formula: Represents the spatial point; This represents the number of geometrically optimal primitives in the set of 3D Gaussian primitives; In the set of 3D Gaussian primitives, the first... The set of geometric parameters of a geometrically optimal primitive; In the set of 3D Gaussian primitives, the first... The explanatory weights of the geometrically optimal primitives for the points in the space; Indicates the first The contribution value of each geometrically optimal primitive to the spatial point; For the first The contribution function of a geometrically optimal primitive to the spatial point, variable For the aforementioned spatial point;

[0047] The nonnegativity constraint is:

[0048] ;

[0049] The normalization constraint is:

[0050] .

[0051] In one alternative embodiment of the first aspect, the formula for calculating the semantic posterior probability of the spatial point is:

[0052] ;

[0053] In the formula: The semantics of the spatial point belong to the first... The probability of a semantic category; Indicates the first The semantic load distribution of a geometrically optimal primitive;

[0054] The formula for calculating the semantic entropy of the spatial point is:

[0055] .

[0056] Secondly, this application provides a semantic load Gaussian splash modeling apparatus, comprising:

[0057] The primitive initialization unit is used to construct a 3D Gaussian primitive set based on the sparse point cloud of the scene, and initialize the geometric parameters of each 3D Gaussian primitive in the 3D Gaussian primitive set.

[0058] The geometric training unit is used to iteratively update the geometric parameters of the 3D Gaussian primitive through a differentiable rendering process until convergence, and update the 3D Gaussian primitive to a geometrically optimal primitive.

[0059] The semantic learning unit is used to acquire sample points of the scene and their semantic supervision information, calculate the interpretation weight of the geometrically optimal primitive on the sample points, and process the interpretation weight and the semantic supervision information of the sample points to obtain the semantic load distribution of the geometrically optimal primitive.

[0060] The reasoning and output unit is used to acquire spatial points of the scene, calculate the explanatory weight of the geometrically optimal primitive for the spatial points, and process the explanatory weight with the semantic load distribution of the geometrically optimal primitive to obtain the semantic posterior probability and semantic entropy of the spatial points and output them.

[0061] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method provided by the first aspect of this application or any implementation thereof.

[0062] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method provided by the first aspect of this application or any implementation thereof.

[0063] The present application adopts the above technical solution, and compared with the prior art, it mainly has the following technical effects:

[0064] Semantic embedding does not change the geometric optimal solution of the 3D Gaussian primitives and the geometric edge distribution of the 3DGS model, ensuring that the accuracy of 3D modeling is not affected;

[0065] Semantics are attached to 3D Gaussian primitives in the form of a probability distribution through semantic payload distribution, eliminating the need to train an additional semantic training network and avoiding the complexity brought about by multi-model collaboration.

[0066] Semantics is statistically analyzed using interpretation weights consistent with geometry, allowing semantic load distribution to propagate naturally along the geometric structure and avoiding the problem of misalignment between semantics and geometric boundaries;

[0067] Semantic supervision information in the form of hard or soft labels can be compatible with multi-perspective and multi-source semantic supervision, and conflict information can be smoothly fused through a semantic soft counting mechanism to improve system stability.

[0068] Expressing semantics through semantic posterior probability and semantic entropy instead of a single label can simultaneously reflect semantic category and semantic uncertainty, making it more suitable for decision-making and querying in complex scenarios.

[0069] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart illustrating the semantic load Gaussian splash modeling method provided in the embodiments of this application;

[0072] Figure 2 This is a flowchart illustrating step S200 of the method provided in the embodiments of this application;

[0073] Figure 3 This is a flowchart illustrating step S300 of the method provided in the embodiments of this application;

[0074] Figure 4 This is another schematic diagram of S300 in the method provided in the embodiments of this application;

[0075] Figure 5 This is a flowchart illustrating step S400 of the method provided in the embodiments of this application;

[0076] Figure 6 This is a schematic diagram of the semantic load Gaussian splash modeling device provided in the embodiments of this application;

[0077] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0079] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or modules is not limited to the steps or modules listed, but may optionally include steps or modules not listed, or may optionally include other steps or modules inherent to such process, method, product, or apparatus.

[0080] In existing technologies, standard 3DGS models explicitly represent scene geometry and appearance by arranging several Gaussian primitives with attribute parameters in three-dimensional space, and achieve new perspective image synthesis through differentiable rendering. During training, 3DGS models typically minimize the reconstruction error between the rendered image and the real observed image through a differentiable rendering process, thereby jointly optimizing the position, covariance, opacity, and color or spherical harmonic sparsity of the 3D Gaussian primitives, gradually forming a high-quality representation of scene geometry and appearance.

[0081] From a spatial representation perspective, the distribution of each 3D Gaussian primitive in three-dimensional space can be represented as a three-dimensional Gaussian function:

[0082] ;

[0083] In the formula: Indicates the first The mean of the 3D Gaussian primitives, used to describe the 3D Gaussian primitive. The central location of a 3D Gaussian primitive; For the first The covariance matrix of the nth 3D Gaussian primitive is used to describe the... The spatial scale and orientation of a 3D Gaussian primitive.

[0084] This expression describes the spatial distribution characteristics of a single 3D Gaussian primitive. By combining multiple 3D Gaussian primitives, the continuous geometric structure of the scene and the contribution relationship between pixels and each 3D Gaussian primitive can be characterized.

[0085] The aforementioned 3DGS model is applicable to, but not limited to, the following scenarios: power digital twins (semantic differentiation of objects such as poles, insulators, switchgear, and conductor corridors), 3D asset modeling of industrial parks / factories, indoor and outdoor mobile mapping, robot environmental understanding, and AR / VR scene reconstruction. The input data for the 3DGS model can come from dense point clouds obtained through laser point cloud, structured light, photogrammetry, or multi-view image reconstruction, or from sampling points and pixel supervision signals generated during the 3DGS training process.

[0086] For example, when a 3DGS model is applied to a digital twin modeling scenario for a substation, data is acquired on-site through multi-view images or laser scanning to construct a high-precision 3D Gaussian model. The model typically includes various types of equipment and structures such as conductors, insulators, switchgear, supports, and the ground. If the model relies solely on geometric information, although the system can accurately represent the spatial shape, it is difficult to reliably answer semantically related questions such as "what type of equipment does the structure belong to?", "what functional area is the equipment or structure located in?", and "is the equipment or structure a key inspection target?".

[0087] As described in the background section, existing post-processing classification methods, joint training coupling methods, and direct label attachment methods all have significant limitations. To address these shortcomings, the purpose of this application is to achieve effective embedding of semantic information while maintaining the geometric expressive power of the 3DGS model. This enables the model to not only describe "what shapes exist in space" but also answer "what are the shapes in space," thereby significantly enhancing its practical value in scenarios such as digital twins, robot perception, and autonomous driving.

[0088] The present application will now be described in detail with reference to specific embodiments.

[0089] See Figure 1 This application provides a semantic load Gaussian splash modeling method, including the following steps:

[0090] S100, based on the sparse point cloud of the scene, constructs a set of 3D Gaussian primitives and initializes the geometric parameters of each 3D Gaussian primitive in the set of 3D Gaussian primitives;

[0091] S200 iteratively updates the geometric parameters of the 3D Gaussian primitives through a differentiable rendering process until convergence, thus updating the 3D Gaussian primitives to the geometrically optimal primitives.

[0092] S300: Obtain the sample points of the scene and their semantic supervision information, calculate the explanatory weight of the geometrically optimal primitive on the sample points, and process the explanatory weight with the semantic supervision information of the sample points to obtain the semantic load distribution of the geometrically optimal primitive.

[0093] S400: Obtain spatial points of the scene, calculate the explanatory weight of the geometrically optimal primitive for the spatial points, and process the explanatory weight with the semantic load distribution of the geometrically optimal primitive to obtain the semantic posterior probability and semantic entropy of the spatial points and output them.

[0094] In this embodiment, sample points are the observed locations of the scene, spatial points are the actual locations of the scene, and the semantic entropy of spatial points is used to characterize the semantic uncertainty of spatial points. Through S100-S400, the pure geometric modeling of the 3DGS model is first completed and converged to obtain the geometric optimal solution of the 3D Gaussian primitives. Then, without changing the geometric optimal solution, a semantic load distribution is attached to each 3D Gaussian primitive, so that the 3DGS model ontology can output the semantic posterior probability and semantic entropy of spatial points, while strictly ensuring that the geometric edge distribution of the 3DGS model does not change.

[0095] For example, the specific process of S100 is as follows: Based on multi-view image data, sparse point clouds of the scene are obtained through structure restoration methods such as SfM (Structure from Motion) or COLMAP, and an initial set of 3D Gaussian primitives is constructed on this basis to obtain the first... The set of initial geometric parameters for a 3D Gaussian primitive:

[0096] ;

[0097] In the formula: Indicates the first The initial mean of a 3D Gaussian primitive can be obtained by sampling points in the coefficient point cloud of the scene; Indicates the first The initial covariance matrix of a 3D Gaussian primitive can be obtained through local neighborhood statistical estimation. Indicates the first The initial opacity of a 3D Gaussian primitive; Indicates the first The initial color (or spherical harmonic) coefficients of a 3D Gaussian primitive.

[0098] The purpose of S100 is to construct an initial 3D Gaussian representation that can cover the scene's geometry, providing a stable starting point for subsequent differentiable rendering training. It should be understood that if the initial distribution of geometric parameters is too sparse or overlaps too much, it will lead to difficulties in the convergence of the differentiable rendering optimization process, affecting the final geometric representation quality of the 3DGS model.

[0099] In some embodiments, such as Figure 2 As shown, S200 specifically includes the following steps:

[0100] S201, through a differentiable rendering process, projects a 3D Gaussian primitive onto an image plane to generate a rendered image and obtains the gradient of the rendered image relative to the 3D Gaussian primitive;

[0101] S202, using this gradient, with the goal of minimizing the rendering error of the rendered image, the geometric parameters of the 3D Gaussian primitive are iteratively updated until convergence, and the 3D Gaussian primitive is updated to the geometrically optimal primitive.

[0102] The input to S201 is the initial 3D Gaussian primitive obtained in S100, and the optimization objective of S202 can be established accordingly as follows:

[0103] ;

[0104] In the formula: This represents the rendered image obtained by projecting the currently input 3D Gaussian primitive. The first term on the right-hand side of the equation represents the pixel-level L1 error, and the second term represents the structural similarity loss between the currently rendered image and the actual observed image. These are the weighting coefficients.

[0105] Using the aforementioned optimization objective as the geometric training objective for the 3D Gaussian primitive, the mean, covariance matrix, opacity, and color coefficients of the 3D Gaussian primitive can be jointly optimized. In other words, the purpose of establishing the above optimization objective is to drive the geometric parameter update of the 3D Gaussian primitive through image reconstruction error, so that the 3D Gaussian primitive gradually forms a continuous three-dimensional representation that can accurately express the geometry and appearance of the scene.

[0106] Subsequently, using the gradient of the rendered image obtained from the differentiable rendering process relative to the 3D Gaussian primitives, the geometric parameters of the 3D Gaussian primitives are updated through gradient descent or adaptive optimization algorithms. During the update process, the density of the 3D Gaussian primitives can be dynamically adjusted according to the 3DGS training strategy, for example, by adding new 3D Gaussian primitives in high-error regions and merging or deleting 3D Gaussian primitives in low-contribution regions. This process continues iteratively until the rendering error converges, thereby gradually approximating the true geometric structure of the scene and enabling the 3D Gaussian primitives to form a stable three-dimensional representation.

[0107] When the rendering error decreases to a preset threshold or the number of iterations reaches the upper limit, the geometric training is considered to have converged, at which point the set of geometric parameters of the geometrically optimal primitive is obtained:

[0108] ;

[0109] In the formula: Indicates the first The set of geometric parameters of a geometrically optimal primitive, and , , and They represent the first The mean, covariance matrix, opacity, and color coefficient of each geometrically optimal primitive. These geometric parameters constitute what is referred to as the geometrically optimal solution in this application.

[0110] It should be noted that before entering S300, the geometrically optimal solution needs to be frozen, that is, the geometric parameters of the geometrically optimal primitive will not be updated in the subsequent semantic learning process.

[0111] From the perspective of existing technology, the geometric training process involved in S200 can include common 3DGS geometric training methods, which will not be explained in detail here. The purpose of S201-S202 is to provide a stable and fixed geometric representation foundation for the subsequent semantic learning stage, thereby ensuring that semantic learning will not change the already converged scene geometry.

[0112] In some embodiments, such as Figure 3 As shown, S300 specifically includes the following steps:

[0113] S301, Obtain the sample points of the scene and their semantic supervision information, and calculate the explanatory weight of the geometrically optimal primitives on the sample points by using the contribution value of the geometrically optimal primitives to the sample points.

[0114] S302, combine the explanatory weight of the geometrically optimal primitive to the sample point with the semantic supervision information of the sample point to obtain the semantic soft count of the geometrically optimal primitive;

[0115] S303, based on the semantic soft count of the geometrically optimal primitive, is processed to obtain the normalized semantic load distribution of the geometrically optimal primitive.

[0116] The input to S301 is the geometrically optimal solution obtained from S200, the sample points of the scene, and the semantic supervision information of the sample points.

[0117] Since the geometric parameters of the geometrically optimal primitives have been frozen, the purpose of S301 is to recalculate the contribution relationship between the sample points and the 3D Gaussian primitives based on the geometrically optimal solution, thereby determining the proportion of the sample points explained by each 3D Gaussian primitive.

[0118] In some embodiments, the formula for calculating the explanatory weight of the geometrically optimal primitive for sample points is as follows:

[0119] ;

[0120] In the formula: Indicates the first One sample point; This represents the number of geometrically optimal primitives in the 3D Gaussian primitive set; Represents the first in the set of 3D Gaussian primitives. The set of geometric parameters of a geometrically optimal primitive, and , , and They represent the first The mean, covariance matrix, opacity, and color coefficient of a geometrically optimal primitive; Indicates the first The geometrically optimal primitive pair for the first Explanatory weights for each sample point; Indicates the first The geometrically optimal primitive pair for the first The contribution value of each sample point; For the first The contribution function of a geometrically optimal primitive to the sample points, variables These are sample points.

[0121] It should be understood that contribution value By the first sample points Assign contribution function variables in get.

[0122] For example, the contribution function in S301 can be defined in the following form:

[0123] Based on the definition of 3D Gaussian distribution:

[0124] ;

[0125] In the formula: Representing variables Follow the mean The covariance matrix is The Gaussian distribution; this form is suitable for point cloud or 3D spatial semantic supervision scenarios;

[0126] Based on the definition of a differentiable rendering process:

[0127] ;

[0128] In the formula: Represents the first in the set of 3D Gaussian primitives. Geometric optimal primitives along variables Cumulative transmittance along the line of sight; Represents the first in the set of 3D Gaussian primitives. A geometrically optimal primitive in variables The projection response at the location; this form is suitable for semantic supervision scenarios based on multi-view images;

[0129] Based on geometrically optimal primitives and variables Definition of feature consistency between:

[0130] ;

[0131] In the formula: Represents the first in the set of 3D Gaussian primitives. Features of a geometrically optimal primitive Representing variables Features for and The similarity function between them; this form can be used to fuse color information or semantic features to improve the discriminative ability of contribution evaluation;

[0132] Based on the formal definition of distance attenuation:

[0133] ;

[0134] In the formula: Represents the first in the set of 3D Gaussian primitives. Gaussian scaling parameters of a geometrically optimal primitive; this form is suitable for resource-constrained or simplified modeling scenarios;

[0135] And a weighted combination definition of the contributions of each perspective in a multi-perspective semantic fusion scenario:

[0136] ;

[0137] In the formula: Indicates the first Weighting coefficients for each perspective Indicates the first A perspective on variables The contribution function.

[0138] In practical applications, the various forms of contribution functions described above can be used individually or combined as needed. Furthermore, regardless of the form used, the contribution function is determined by the geometrically optimal solution and its response relationship in space or viewpoint, and remains fixed during the semantic learning phase, not participating in updates as an optimization variable.

[0139] In some embodiments, the formula for calculating the semantic soft count of the geometrically optimal primitive in S302 is:

[0140] ;

[0141] In the formula: Indicates the number of sample points; For the first The sample point about the th Semantic supervision information for various semantic categories; Indicates the first The geometrically optimal primitive pair for the first Semantic soft counts for each semantic category.

[0142] In this embodiment, semantic soft counting of geometrically optimal primitives is used. This can be understood as: if a sample point is geometrically more "belonging" to the first... The geometrically optimal primitive, and semantically more biased towards the first. If a semantic category is defined, then the sample point pair... Their contribution is even greater.

[0143] The purpose of S302 is to convert sample-level semantic supervision information into 3D Gaussian primitive-level statistics, providing a statistical basis for solving the subsequent semantic payload distribution.

[0144] In many engineering scenarios, semantic supervision information often comes from two-dimensional images and is mapped to three-dimensional space through camera pose and depth or reconstruction relationships. For example, in applications such as robot environment mapping, autonomous driving scene understanding, AR / VR spatial annotation, and digital twin modeling, the system usually first performs two-dimensional semantic segmentation on the acquired multi-view images, and then associates the segmentation results with the three-dimensional model.

[0145] Of course, semantic supervision information can also come directly from 3D spatial data (such as point clouds with semantic labels, LiDAR annotations, or manual 3D annotations), without relying on 2D image projection processes. For example, in applications such as semantic mapping of point clouds for autonomous driving, 3D environment understanding for robots, and digital twin modeling, the system can directly acquire 3D points or spatial sampling points with semantic information and establish a correspondence with the 3D geometric model through spatial registration.

[0146] In some embodiments, the semantic supervision information of the sample points is a hard label; if the first The semantics of the nth sample point belong to the th semantic categories, then ;otherwise, .

[0147] The existing direct label attachment method simply binds existing labels to the corresponding 3D Gaussian primitives. Here, simple binding means that after geometric training is completed, a single semantic category ID or artificial label field is directly written into a certain 3D Gaussian primitive, point or voxel record, forming only a static correspondence.

[0148] In this embodiment, hard labels are used only for observation supervision, not for single-value label binding. Specifically, hard-label inputs differ from the simple binding described above in at least the following ways:

[0149] Firstly, simple binding is usually a one-to-one or forcibly specified relationship, which cannot express the mixed structure where a sample point is interpreted by multiple 3D Gaussian primitives at the same time;

[0150] Secondly, simple binding does not produce a normalizable conditional probability distribution, making it impossible to output semantic uncertainty in the future;

[0151] Third, simple binding lacks the basis for geometric edge invariance derived from joint distribution, and cannot prove that the binding action will not destroy the statistical meaning of the original geometric model;

[0152] Fourth, simple binding lacks a buffering mechanism for multi-view conflict labels, boundary regions, and noise annotations, while hard labels can achieve smooth absorption through interpreting weights and semantic soft counting.

[0153] In some embodiments, the semantic supervision information for sample points is a soft label; the soft label is the first... The semantics of the nth sample point belong to the th Probability of semantic categories

[0154] For example, the process of forming soft labels based on two-dimensional images is as follows: semantic segmentation is performed on the multi-view image using a 2D semantic segmentation network to obtain the class probability of each pixel. The pixels are then mapped to three-dimensional points or sampling points through depth or reprojection to obtain the semantic distribution of that point. If multiple views generate multiple semantic distributions for the same point, they can be weighted and fused according to viewpoint, confidence level, or visibility to obtain the final semantic distribution. Finally, the semantic distribution will be... Imparting semantic supervision information, that is This creates a soft label input.

[0155] It should also be noted that during the semantic mapping process of the above 2D semantic segmentation network, it is necessary to ensure that the semantic category IDs are consistent, the ignore class is treated consistently, and invalid labels are excluded during statistics.

[0156] As can be seen from the above, using 2D semantic segmentation networks as a source of soft labels allows for the flexible introduction of semantic information after the 3DGS model has been established, without altering the existing geometric structure. This aligns with application scenarios such as robot environment understanding, autonomous driving 3D semantic map construction, and digital twin scene annotation.

[0157] In some embodiments, the formula for calculating the semantic load distribution of the geometrically optimal primitive in S303 is as follows:

[0158] ;

[0159] In the formula: Indicates the first The geometrically optimal primitive pair for the first The distribution of semantic loads for each semantic category; Indicates the total number of semantic categories;

[0160] The semantic load distribution of the geometrically optimal primitive satisfies the normalization condition:

[0161] .

[0162] get After that, semantic learning is complete.

[0163] The purpose of S303 is to convert semantic soft counting into a semantic payload distribution that satisfies probabilistic constraints, serving as a closed-loop update result for the semantic learning phase. Since S303 only updates the semantic payload distribution without writing back the geometric parameters of the geometrically optimal primitive, the difference between this application and existing joint training methods is that the semantic learning result does not back-perturb the geometrically optimal solution.

[0164] To address the issue of sparse semantic categories or insufficient sample points, and to ensure numerical stability, the semantic load distribution of the geometrically optimal primitive in S303 can also be obtained using the following formula:

[0165] ;

[0166] In the formula: It is a minimal smooth term;

[0167] The semantic load distribution of geometrically optimal primitives still needs to satisfy the normalization condition:

[0168] .

[0169] Introducing a minimal smoothing term The purpose is to avoid the denominator of the term on the right-hand side of the equation in the above formula for calculating semantic load distribution being zero. Therefore, when a geometrically optimal primitive receives a small number of soft counts of a certain type of semantic data, this minimal smoothing term can prevent the probability from degenerating to zero, thereby improving the stability of the model under sparse sample or noisy labeling conditions. Generally, Desirable to Quantity, preferred about.

[0170] In actual deployment, semantic annotation may be gradually improved or new semantic samples may be continuously collected. For example, in the long-term environmental mapping of robots, the continuous collection of autonomous driving scenario data, and the long-term maintenance of digital twin systems, the three-dimensional geometric model is often established stably, but the semantic information may continue to increase or be updated over time.

[0171] In some embodiments, such as Figure 4 As shown, after S303, S300 further includes the following steps:

[0172] S304, obtain the newly added sample points in the scene and their semantic supervision information, and calculate the explanatory weight of the geometrically optimal primitive to the newly added sample points by the contribution value of the geometrically optimal primitive to the newly added sample points.

[0173] S305, combine the explanatory weight of the geometrically optimal primitive for the new sample points with the semantic supervision information of the new sample points to obtain the new semantic soft count of the geometrically optimal primitive;

[0174] S306. Based on the semantic soft count of the geometrically optimal primitive and the cumulative value of the newly added semantic soft count, the normalized semantic load distribution of the geometrically optimal primitive is obtained again.

[0175] As can be seen from the above, if it is necessary to maintain historical semantic statistics, the semantic soft count can be persistently stored. When new data arrives, it can be accumulated on the basis of the original semantic soft count, thereby continuously improving the semantic quality without retraining the 3DGS model. This supports the updating of semantic models in long-term operating systems, such as the continuous improvement of robot environment semantic maps, the expansion of semantic libraries for autonomous driving scenarios, and the maintenance of semantic information in digital twin systems.

[0176] In this embodiment, under the condition of freezing the geometric parameters of the geometrically optimal primitive, the interpretation weight of the corresponding geometrically optimal primitive is calculated for each newly added sample point, and its semantic soft count is accumulated, thereby recalculating the normalized semantic load distribution and realizing incremental update of sample points and multi-batch semantic attachment.

[0177] From the coupling relationship between the two stages S200 and S300, S200 addresses the problem of how to optimally fit the observed distribution geometrically, while S300 addresses the problem of how to carry semantic probabilities on the existing optimal geometric interpretation structure. The two stages are coupled through the bridge variable of the relationship between sample points and Gaussian contributions, and decoupled to update degrees of freedom through a parameter freezing mechanism. It is this statistically coupled and optimizationally isolated architecture that makes this application different from simple binding and end-to-end joint training.

[0178] In some embodiments, such as Figure 5 As shown, S400 specifically includes the following steps:

[0179] S401, obtain the spatial points of the scene, calculate the interpretation weight of the geometric optimal primitive for the spatial points and make it satisfy the non-negativity and normalization constraints;

[0180] S402, attach the semantic load distribution of the geometrically optimal primitive to the explanatory weight of the geometrically optimal primitive for the spatial point, and obtain the semantic posterior probability of the spatial point.

[0181] S403 processes the semantic posterior probability of spatial points to obtain the semantic entropy of spatial points, and outputs the semantic posterior probability and semantic entropy of spatial points.

[0182] In this embodiment, after semantic attachment is completed, geometric and semantic information can be output simultaneously at any spatial point in the scene without relying on an additional classification network. For example, in robot environmental perception or autonomous driving scenarios, the system can query the semantic posterior probability of a spatial point in three-dimensional space to determine whether the location is more likely to belong to categories such as roads, pedestrians, vehicles, or buildings, and thereby assist in path planning or behavior decision-making.

[0183] In some embodiments, the formula for calculating the interpretation weight of spatial points by the geometrically optimal primitive in S401 is as follows:

[0184] ;

[0185] In the formula: Represents a point in space; This represents the number of geometrically optimal primitives in the 3D Gaussian primitive set; Represents the first in the set of 3D Gaussian primitives. The set of geometric parameters of a geometrically optimal primitive, and , , and They represent the first The mean, covariance matrix, opacity, and color coefficient of a geometrically optimal primitive; Indicates the first The explanatory weights of a geometrically optimal primitive for a point in space; Indicates the first The contribution of each geometrically optimal primitive to a point in space; For the first The contribution function of a geometrically optimal primitive to a point in space, variables For spatial points;

[0186] The nonnegativity constraint is:

[0187] ;

[0188] The normalization constraint is:

[0189] .

[0190] It should be understood that contribution value The contribution function in S301 mentioned above can be reused. Obtain, that is, by using spatial points Assign contribution function variables in get.

[0191] In some embodiments, the formula for calculating the semantic posterior probability of a spatial point in S402 is:

[0192] ;

[0193] In the formula: The semantics of representing a spatial point belong to the first The probability of a semantic category; Indicates the first The semantic load distribution of a geometrically optimal primitive; the above formula represents the weighted average of the semantic load distribution of all geometrically optimal primitives according to their interpretation weights for the current spatial point; the semantic posterior probability can be interpreted from the perspective of semantic distribution as follows: when the spatial point is mainly interpreted by a few 3D Gaussian primitives, the semantic distribution is relatively concentrated; when the spatial point is located at the structural boundary or mixed region, multiple 3D Gaussian primitives contribute together, and the semantic distribution is more dispersed;

[0194] The formula for calculating the semantic entropy of a spatial point in S403 is as follows:

[0195] ;

[0196] The higher the semantic entropy, the more uncertain the semantics of a spatial point. When the semantic probability of a spatial point is concentrated and the semantic entropy is small, the semantic judgment can be considered relatively reliable. When the semantic entropy is large, it indicates that there is category mixing or insufficient observation of the spatial point.

[0197] For example, in robot navigation, when a location is judged to be a walkable area with a high semantic posterior probability and low semantic entropy, the system can prioritize selecting that area as the path; when the semantic entropy of a location is high, it indicates that there is category mixing or insufficient observation in that area, which can trigger further observation or avoidance strategies; in digital twin or 3D scene query systems, users can also directly query the semantic category and semantic posterior probability of a certain spatial area to achieve semantic-based spatial retrieval and analysis.

[0198] It should be noted that the semantic posterior probability is directly calculated from the semantic payload within the 3DGS model, without the need for additional training of a semantic classification network; the semantic entropy is intrinsically calculated from the posterior probability distribution of spatial points, without the need for an additional uncertainty modeling module.

[0199] The following section uses the 3D Gaussian distribution to illustrate that semantic embedding of a 3DGS model using S401-S403 does not change its geometric edge distribution.

[0200] Based on the semantic load distribution of geometrically optimal primitives, a joint distribution is constructed in the 3DGS model:

[0201] ;

[0202] In the formula: Representing a spatial point Follow the mean The covariance matrix is Gaussian distribution;

[0203] Since the semantic load distribution of the geometrically optimal primitive satisfies the normalization condition:

[0204] ;

[0205] Therefore, spatial points can be obtained. Geometric edge distribution:

[0206] .

[0207] As shown above, the semantic load distribution is attached to the geometrically optimal primitive only in the form of a conditional distribution, without changing its geometric parameters, and therefore does not alter the geometric edge distribution of the 3DGS model. Semantics are statistically analyzed using interpretation weights consistent with geometry, allowing the semantic load distribution to propagate naturally along the geometric structure, avoiding the problem of misalignment between semantics and geometric boundaries.

[0208] See Figure 6 This application also provides a semantic load Gaussian splash modeling device. This device can be implemented as all or part of a terminal through software, hardware, or a combination of both, or it can be integrated as an independent module on a server. Specifically, the device includes a primitive initialization unit, a geometric training unit, a semantic learning unit, and a reasoning and output unit, wherein:

[0209] The primitive initialization unit is used to construct a 3D Gaussian primitive set based on the sparse point cloud of the scene and initialize the geometric parameters of each 3D Gaussian primitive in the 3D Gaussian primitive set.

[0210] The geometry training unit is used to iteratively update the geometric parameters of the 3D Gaussian primitives through a differentiable rendering process until convergence, thus updating the 3D Gaussian primitives to the geometrically optimal primitives.

[0211] The semantic learning unit is used to acquire sample points of the scene and their semantic supervision information, calculate the explanatory weight of the geometrically optimal primitive on the sample points, and process the explanatory weight and the semantic supervision information of the sample points to obtain the semantic load distribution of the geometrically optimal primitive.

[0212] The reasoning and output unit is used to acquire spatial points of the scene, calculate the explanatory weight of the geometrically optimal primitive for the spatial points, and process the explanatory weight with the semantic load distribution of the geometrically optimal primitive to obtain the semantic posterior probability and semantic entropy of the spatial points and output them.

[0213] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the functional modules described above when executing the semantic load Gaussian splash modeling method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus provided in the above embodiments and the semantic load Gaussian splash modeling method embodiments belong to the same concept, and its implementation process is detailed in the method embodiments, which will not be repeated here.

[0214] See Figure 7 This application provides an electronic device 700, including a processor 701 and a memory 702.

[0215] In this embodiment, the processor 701 is the control center of the computer system, and can be a processor of a physical machine or a processor of a virtual machine. The processor 701 may include one or more processing cores, such as a 4-core processor or an 8-core processor. The processor 701 can be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array).

[0216] Processor 701 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake-up state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state.

[0217] Memory 702 may include one or more computer-readable storage media. Memory 702 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments of this application, the computer-readable storage media in memory 702 are used to store at least one instruction, which is executed by processor 701 to implement the methods in the embodiments of this application.

[0218] In some embodiments, the electronic device 700 further includes a peripheral device interface 703 and at least one peripheral device 704. The processor 701, memory 702, and peripheral device interface 703 can be connected via a bus or signal line. Each peripheral device 704 can be connected to the peripheral device interface 703 via a bus, signal line, or circuit board. Specifically, the peripheral device 704 includes: a display screen, a camera, and audio circuitry. The peripheral device interface 703 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 701 and memory 702.

[0219] In some embodiments of this application, the processor 701, memory 702, and peripheral device interface 703 are integrated on the same chip or circuit board; in other embodiments of this application, any one or two of the processor 701, memory 702, and peripheral device interface 703 can be implemented on separate chips or circuit boards. This application does not specifically limit the implementation in this regard.

[0220] The schematic diagram of the electronic device shown in the embodiments of this application does not constitute a limitation on the electronic device 700. The electronic device 700 may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0221] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the methods in any of the foregoing embodiments. The computer-readable storage medium may include, but is not limited to, any type of disk, including floppy disks, optical disks, DVDs, CD-ROMs, microdrives, as well as magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic cards or optical cards, nanosystems (including molecular memory ICs), or any type of medium or device suitable for storing instructions and / or data.

[0222] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0223] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for modeling Gaussian splashing of semantic loads, characterized in that, The method includes: Based on the sparse point cloud of the scene, a set of 3D Gaussian primitives is constructed, and the geometric parameters of each 3D Gaussian primitive in the set of 3D Gaussian primitives are initialized. The geometric parameters of the 3D Gaussian primitive are iteratively updated through a differentiable rendering process until convergence, thereby updating the 3D Gaussian primitive to a geometrically optimal primitive. Obtain sample points and their semantic supervision information in the scene, calculate the interpretation weight of the geometrically optimal primitive on the sample points, and process the interpretation weight and the semantic supervision information of the sample points to obtain the semantic load distribution of the geometrically optimal primitive. Obtain spatial points of the scene, calculate the explanatory weight of the geometrically optimal primitive for the spatial points, and process the explanatory weight with the semantic load distribution of the geometrically optimal primitive to obtain the semantic posterior probability and semantic entropy of the spatial points and output them.

2. The semantic load Gaussian splashing modeling method according to claim 1, characterized in that, The step of iteratively updating the geometric parameters of the 3D Gaussian primitive through a differentiable rendering process until convergence, and updating the 3D Gaussian primitive to a geometrically optimal primitive, includes: The 3D Gaussian primitive is projected onto the image plane through a differentiable rendering process to generate a rendered image, and the gradient of the rendered image relative to the 3D Gaussian primitive is obtained. Using the gradient, with the goal of minimizing the rendering error of the rendered image, the geometric parameters of the 3D Gaussian primitive are iteratively updated until convergence, and the 3D Gaussian primitive is updated to a geometrically optimal primitive.

3. The semantic load Gaussian splashing modeling method according to claim 1, characterized in that, The process involves acquiring sample points and their semantic supervision information from the scene, calculating the explanatory weight of the geometrically optimal primitive on the sample points, and combining this explanatory weight with the semantic supervision information of the sample points to obtain the semantic load distribution of the geometrically optimal primitive, including: Obtain sample points of the scene and their semantic supervision information, and calculate the explanatory weight of the geometrically optimal primitive on the sample points based on the contribution value of the geometrically optimal primitive to the sample points; The semantic soft count of the geometrically optimal primitive is obtained by combining the explanatory weight of the geometrically optimal primitive on the sample point with the semantic supervision information of the sample point. The semantic soft count of the geometrically optimal primitive is processed to obtain the normalized semantic load distribution of the geometrically optimal primitive.

4. The semantic load Gaussian splashing modeling method according to claim 3, characterized in that, The formula for calculating the explanatory weight of the geometrically optimal primitive for the sample point is as follows: ； In the formula: Indicates the first One sample point; This represents the number of geometrically optimal primitives in the set of 3D Gaussian primitives; In the set of 3D Gaussian primitives, the first... The set of geometric parameters of a geometrically optimal primitive; Indicates the first The geometrically optimal primitive for the first Explanatory weights for each sample point; Indicates the first The geometrically optimal primitive for the first The contribution value of each sample point; For the first The contribution function of a geometrically optimal primitive to the sample point, variable The sample point is referred to here.

5. The semantic load Gaussian splashing modeling method according to claim 4, characterized in that, The formula for calculating the semantic soft count of the geometrically optimal primitive is as follows: ； In the formula: This indicates the number of sample points; For the first The sample point about the th Semantic supervision information for various semantic categories; Indicates the first The geometrically optimal primitive for the first Semantic soft counts for each semantic category.

6. The semantic load Gaussian splashing modeling method according to claim 5, characterized in that, The semantic supervision information of the sample points is a hard label; if the first The semantics of the sample point belong to the first... semantic categories, then ;otherwise, .

7. The semantic load Gaussian splashing modeling method according to claim 5, characterized in that, The semantic supervision information of the sample points is a soft label; the soft label is the first... The semantics of the sample point belong to the first... The probability of a semantic category.

8. The semantic load Gaussian splashing modeling method according to claim 5, characterized in that, The formula for calculating the semantic load distribution of the geometrically optimal primitive is as follows: ； In the formula: Indicates the first The geometrically optimal primitive for the first The distribution of semantic loads for each semantic category; Indicates the total number of semantic categories; The semantic load distribution of the geometrically optimal primitive satisfies the normalization condition: 。 9. The semantic load Gaussian splashing modeling method according to claim 3, characterized in that, After processing the semantic soft count based on the geometrically optimal primitive to obtain the normalized semantic load distribution of the geometrically optimal primitive, the method further includes: Acquire new sample points in the scene and their semantic supervision information, and calculate the explanatory weight of the geometrically optimal primitive on the new sample points based on the contribution value of the geometrically optimal primitive to the new sample points; The explanatory weight of the geometrically optimal primitive for the newly added sample points is combined with the semantic supervision information of the newly added sample points to obtain the new semantic soft count of the geometrically optimal primitive; The semantic soft count of the geometrically optimal primitive and the cumulative value of the newly added semantic soft count are processed to obtain the normalized semantic load distribution of the geometrically optimal primitive.

10. The semantic load Gaussian splashing modeling method according to claim 1, characterized in that, The process of acquiring spatial points in the scene, calculating the explanatory weight of the geometrically optimal primitive for the spatial points, and combining the explanatory weight with the semantic load distribution of the geometrically optimal primitive to obtain and output the semantic posterior probability and semantic entropy of the spatial points includes: Obtain spatial points of the scene, calculate the explanatory weight of the geometrically optimal primitive for the spatial points, and make it satisfy non-negativity and normalization constraints; The semantic load distribution of the geometrically optimal primitive is attached to the explanatory weight of the geometrically optimal primitive for the spatial point to obtain the semantic posterior probability of the spatial point; The semantic posterior probability of the spatial point is processed to obtain the semantic entropy of the spatial point, and the semantic posterior probability and semantic entropy of the spatial point are output.

11. The semantic load Gaussian splashing modeling method according to claim 10, characterized in that, The formula for calculating the interpretation weight of the geometrically optimal primitive for the spatial point is as follows: ； In the formula: Represents the spatial point; This represents the number of geometrically optimal primitives in the set of 3D Gaussian primitives; In the set of 3D Gaussian primitives, the first... The set of geometric parameters of a geometrically optimal primitive; Indicates the first The explanatory weights of the geometrically optimal primitives for the points in the space; Indicates the first The contribution value of each geometrically optimal primitive to the spatial point; For the first The contribution function of a geometrically optimal primitive to the spatial point, variable For the aforementioned spatial point; The nonnegativity constraint is: ； The normalization constraint is: 。 12. The semantic load Gaussian splashing modeling method according to claim 11, characterized in that, The formula for calculating the semantic posterior probability of the spatial point is: ； In the formula: The semantics of the spatial point belong to the first... The probability of a semantic category; Indicates the first The semantic load distribution of a geometrically optimal primitive; The formula for calculating the semantic entropy of the spatial point is: 。 13. A semantic load Gaussian splash modeling apparatus, based on the method as described in any one of claims 1-12, characterized in that, The device includes: The primitive initialization unit is used to construct a 3D Gaussian primitive set based on the sparse point cloud of the scene, and initialize the geometric parameters of each 3D Gaussian primitive in the 3D Gaussian primitive set. The geometric training unit is used to iteratively update the geometric parameters of the 3D Gaussian primitive through a differentiable rendering process until convergence, and update the 3D Gaussian primitive to a geometrically optimal primitive. The semantic learning unit is used to acquire sample points of the scene and their semantic supervision information, calculate the interpretation weight of the geometrically optimal primitive on the sample points, and process the interpretation weight and the semantic supervision information of the sample points to obtain the semantic load distribution of the geometrically optimal primitive. The reasoning and output unit is used to acquire spatial points of the scene, calculate the explanatory weight of the geometrically optimal primitive for the spatial points, and process the explanatory weight with the semantic load distribution of the geometrically optimal primitive to obtain the semantic posterior probability and semantic entropy of the spatial points and output them.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-12.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-12.

Images