Scene semantic understanding system based on 3D gaussian splash model

Abstract

This application relates to the fields of computer vision and computer graphics technology, and discloses a scene semantic understanding system based on a 3D Gaussian splash model. This system establishes Gaussian particle indices and obtains physical neighborhoods through a spatial topology construction subsystem; a physical state perception subsystem calculates the ray opacity entropy value during rasterization; a semantic feedback control subsystem determines the physical mode of pixel rays based on the entropy value and generates a gradient receiving mask to adaptively adjust the depth propagation path of the semantic gradient; and a geometric evolution subsystem performs orthogonalization correction on the semantic gradient based on the photometric convergence state and performs structured splitting using tangent plane constraints. This invention effectively solves the problem of semantic information ambiguity in the depth direction and gradient conflicts in multi-task optimization, achieving high-precision scene semantic parsing and geometric reconstruction.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and computer graphics, specifically to a scene semantic understanding system based on a 3D Gaussian splash model. Background Technology

[0002] 3D Gaussian splashing technology, with its explicit scene representation and efficient real-time rendering capabilities, has gradually become the mainstream method in the fields of 3D reconstruction and new perspective compositing. As application demands deepen, simple geometric and appearance reconstruction is no longer sufficient to meet the needs of high-level tasks such as robot navigation and virtual reality interaction. Imbuing scenes with fine semantic information during the reconstruction process has become a critical issue that urgently needs to be addressed.

[0003] However, existing semantic understanding methods based on Gaussian splashing suffer from significant technical bottlenecks when handling complex scenes. Since rasterization rendering projects three-dimensional space onto a two-dimensional plane, directly using the semantic labels of the two-dimensional image as supervisory signals for backpropagation often lacks effective perception of the physical structure in the depth direction. This leads to semantic gradients failing to correctly distinguish foreground from background when dealing with sparse occlusion structures such as object edges, fences, or semi-transparent media like smoke, easily resulting in semantic leakage. This causes occluded object surfaces to incorrectly acquire the semantic labels of the foreground.

[0004] Furthermore, gradient conflicts frequently arise between different task objectives during the joint optimization of geometric and semantic parameters. To fit high-dimensional semantic features, strong semantic gradients often interfere with the normal convergence of Gaussian particles to geometric positions and shapes, leading to non-physical distortions or drifts in the object's geometry. Simultaneously, traditional Gaussian particle splitting strategies typically rely solely on isotropic refinement based on positional gradients, lacking constraints on local surface topology. This can easily cause inappropriate thickening and volume expansion of the geometric surface in textured regions, reducing the geometric accuracy and realism of the reconstructed model.

[0005] Therefore, this invention proposes a scene semantic understanding system based on a 3D Gaussian splash model to address the shortcomings of existing technologies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a scene semantic understanding system based on a 3D Gaussian splash model. This system solves the problems of semantic information being confused and leaked in the depth direction due to the lack of depth physical constraints when performing scene semantic understanding based on the 3D Gaussian splash model, as well as the gradient conflict between semantic optimization and geometric reconstruction tasks that undermines the stability of the scene structure.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a scene semantic understanding system based on a 3D Gaussian splash model, configured to synchronously parse scene semantic information during the 3D scene reconstruction process, including:

[0008] The spatial topology construction subsystem is configured to establish a spatial index structure for a Gaussian particle set and obtain the physical neighborhood set of the target particle based on the spatial index structure.

[0009] The physics state awareness subsystem is configured to calculate ray opacity entropy values ​​based on a particle contribution list in the rasterization rendering pipeline, and to track the historical visibility state of Gaussian particles and manage exploration budget values.

[0010] A semantic feedback control subsystem is configured to determine the physical mode of a pixel ray based on the ray opacity entropy value, and generate a gradient receiving mask based on the physical mode to adjust the semantic gradient propagation path.

[0011] The geometric evolution subsystem is configured to orthogonalize the semantic gradient based on the photometric convergence state to update the Gaussian particle parameters and perform structured splitting based on the tangent plane.

[0012] This invention provides a scene semantic understanding system based on a 3D Gaussian splash model. It has the following beneficial effects:

[0013] 1. This invention calculates ray opacity entropy values ​​through a physical state perception subsystem, enabling adaptive differentiation of physical modes such as hard surfaces, transition layers, and volumetric media. Combined with the gradient gating mechanism of the semantic feedback control subsystem, differentiated gradient masks are applied to different modes. For example, a narrow window restricts gradient propagation in the depth direction in hard surface regions, while allowing full-flow propagation in volumetric media regions. This design effectively solves the problem of semantic information ambiguity and leakage in the depth direction during 3D reconstruction, ensuring that semantic labels are accurately attached to effective physical structures when dealing with complex occlusion relationships and semi-transparent objects, thus improving the spatial accuracy of scene semantic understanding.

[0014] 2. This invention utilizes the orthogonalized gradient update module within the geometric evolution subsystem to dynamically determine the geometric convergence state by evaluating the photometric gradient magnitude of a region, and adjusts the degree of orthogonal projection of the semantic gradient relative to the geometric gradient accordingly. In regions where the geometric structure is not yet stable, the system forcibly removes parallel components in the semantic gradient that may cause positional drift, eliminating interference from the semantic optimization task on the geometric reconstruction task. This mechanism solves the gradient conflict problem commonly encountered in multi-task joint optimization, ensuring the stability of the 3D geometric structure while endowing the scene with high-dimensional semantic features, and avoiding geometric artifacts or texture collapse caused by forced semantic fitting.

[0015] 3. This invention implements a tangential structured splitting strategy based on local covariance analysis. The structured splitting module identifies the normal direction and tangential plane direction of Gaussian particles, and constrains the newly generated particles to only undergo displacement and anisotropic scaling within the tangential plane. Combined with a photometric residual locking mechanism, the system only densifies particles in areas with significant visual reconstruction errors and strictly limits volume expansion along the surface normal. This technical feature ensures that the Gaussian particle cloud maintains a close fit to the object surface during the densification process, avoiding the non-physical surface thickening phenomenon caused by traditional splitting algorithms, and improving the geometric reconstruction sharpness of object edges and fine structures. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the system structure of the present invention;

[0017] Figure 2 This is a schematic diagram of the method flow of the present invention;

[0018] Figure 3 This is a flowchart of the gradient gating and transient energy injection logic of the present invention;

[0019] Figure 4 This is a time-series diagram illustrating the changes in the exploratory budget value of the present invention;

[0020] Figure 5 This is a histogram illustrating the statistical distribution of ray opacity entropy values ​​in a typical scenario of the present invention.

[0021] Among them, 100 is the spatial topology construction subsystem; 200 is the physical state perception subsystem; 300 is the semantic feedback control subsystem; and 400 is the geometric evolution subsystem. Detailed Implementation

[0022] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] See attached document Figure 1 This invention provides a scene semantic understanding system based on a 3D Gaussian splash model, configured to synchronously parse scene semantic information during 3D scene reconstruction. The system includes:

[0024] The spatial topology construction subsystem 100 is configured to establish a spatial index structure for the Gaussian particle set. The spatial topology construction subsystem 100 includes a Morton coding module, a grid indexing module, and a neighborhood filtering module. The Morton coding module obtains the coordinates of the Gaussian particle center, calculates the Morton code, performs radix sorting, and generates an ordered particle index list. The grid indexing module establishes a mapping relationship between spatial grid cells and the particle index list, constructing a grid range lookup table. The neighborhood filtering module queries the grid range lookup table based on the target particle position, obtains candidate neighborhoods, filters them based on Euclidean distance, and outputs a physical neighborhood set.

[0025] The physics state-aware subsystem 200 is configured to calculate scene physical statistics in the rendering pipeline. The physics state-aware subsystem 200 includes a rasterization rendering module, an entropy field calculation module, and a state tracking module. The rasterization rendering module projects Gaussian particles into screen space, performs tiled blending rendering, and outputs a list of particle contributions corresponding to screen pixels; the entropy field calculation module calculates ray opacity entropy values ​​based on the particle contribution list; the state tracking module counts the cumulative visibility frequency of particles and updates historical visibility states, while also managing particle exploration budget values.

[0026] The semantic feedback control subsystem 300 is configured to adjust the semantic gradient propagation path based on physical statistical characteristics. The semantic feedback control subsystem 300 includes a modality analysis module, a gradient gating module, and an energy injection module. The modality analysis module receives the ray opacity entropy value and classifies pixel rays into hard surface modes, transition layer modes, or volumetric medium modes; the gradient gating module generates a corresponding gradient receiving mask based on the classification results; the energy injection module monitors the semantic gradient strength and direction, and generates a bypass control signal when orthogonality constraints are satisfied and the budget is sufficient.

[0027] The geometric evolution subsystem 400 is configured to update Gaussian particle parameters and perform topology adjustments. The geometric evolution subsystem 400 includes an orthogonal update module, a photometric locking module, and a structure splitting module. The orthogonal update module calculates the photometric gradient and semantic gradient, adaptively adjusts the orthogonal projection component of the semantic gradient based on the photometric convergence state, and updates the particle geometric parameters. The photometric locking module calculates the region color rendering residual, and generates allow or prohibit splitting signals accordingly. The structure splitting module responds to the allow splitting signal, performs eigenvalue decomposition on the covariance matrix of particles that meet the conditions, and performs particle splitting along the tangent plane direction.

[0028] See attached document Figure 2 This invention provides a scene semantic understanding method based on a 3D Gaussian splash model, comprising:

[0029] S100, Construct a Gaussian particle spatial topological index. Obtain the center coordinates of Gaussian particles in the scene, calculate the corresponding Morton codes and sort them to generate a particle index list; establish a mapping relationship between spatial grid cells and the particle index list; retrieve the real physical neighborhood of the target particle based on the mapping relationship and physical distance;

[0030] S200 calculates the statistical characteristics of the scene's physical state. Gaussian particles are projected onto screen space for rasterized rendering, generating a particle contribution list; ray opacity entropy is calculated based on the weight distribution in the particle contribution list; and the historical visibility state and exploration budget of the Gaussian particles are updated.

[0031] S300 performs adaptive semantic gradient gating. It acquires the semantic gradient generated by the semantic loss; determines the physical mode of the pixel ray based on the ray opacity entropy value; generates a gradient receiving mask based on the physical mode to control the distribution of the semantic gradient in the depth direction; and bypasses the mask constraint when the energy injection condition is met.

[0032] S400 performs orthogonalization update of geometric parameters. It calculates the color gradient generated by the photometric loss; determines a relatively stable threshold based on the magnitude of the color gradient; performs adaptive orthogonalization correction on the semantic gradient based on the relatively stable threshold; and updates the position, rotation, and scaling parameters of the Gaussian particles using the corrected gradient.

[0033] S500 performs structured density control and splitting. It detects the anisotropy index and semantic gradient variance of Gaussian particles; calculates the color rendering residual of the region where the particle is located to determine the photometric locking state; for particles that meet the splitting conditions and are not in the locking state, it calculates the covariance matrix tangent plane and generates new particles along the tangent plane direction.

[0034] To further clarify the implementation of each technical aspect of the present invention, the following will provide a detailed description of the implementation of each functional module involved above and its internal processing flow.

[0035] During the operation of the spatial topology construction subsystem, the Merton coding module, grid indexing module, and neighborhood filtering module work together. First, a linearized spatial index based on Merton codes is established, and then adaptive grid mapping and real physical neighborhood retrieval are performed based on this index. This process aims to solve the problem of inefficient spatial querying of three-dimensional Gaussian particles in an unordered storage state. By mapping three-dimensional spatial coordinates to one-dimensional ordered codes, physical memory continuity of particle data is achieved, thereby improving the memory access hit rate of subsequent calculations.

[0036] Specifically, the implementation process of spatial topology construction may include:

[0037] S110, the Morton coding module generates Morton codes and constructs linear indices. The system obtains the center coordinate data of all Gaussian particles in the current scene; let the first... The center coordinates of the Gaussian particles are: The system normalizes and quantizes the floating-point center coordinates into integer coordinates based on the bounding box range of the scene. During this process, the system performs bit interleaving on the binary representation of the three-dimensional integer coordinates (assuming each dimension is quantized to a b-bit binary integer). This involves sequentially extracting bits b-1 to 0 of the x, y, and z coordinates and combining them to generate a 3b-bit binary code string, thus generating the corresponding Morton code. Morton code is a Z-order curve code that exhibits local order preservation, meaning that spatially adjacent particles typically have similar Morton code values. After generating the Morton codes for all particles, the system uses a radix sorting algorithm to sort the particle set. Radix sorting is based on the Morton codes. The numerical value is used to rearrange the particle data in memory, generating a list of particle indices ordered by spatial location. This sorting list makes spatially adjacent particles tend to be contiguous in memory address, which facilitates parallel reading by the graphics processing unit.

[0038] S120 involves constructing a grid range mapping table using the grid index module. To quickly locate particles within a specific spatial region in an ordered list, the grid index module constructs a grid range lookup table; the system then traverses the sorted particle index list. Compare the high-order grid identifiers corresponding to the Morton codes of adjacent particles; when a change in the grid identifier of an adjacent particle is detected, record the starting index of the current grid cell in the list. and end index In this way, the system establishes a system based on grid cell identifiers. To the particle list index range mapping relationship For empty grid cells that do not contain any particles, the corresponding entries in the mapping table are marked as invalid or zero range.

[0039] S130, perform adaptive mesh size setting. To adapt to 3D scenes of different scales and densities, the size of the mesh cells is not fixed; before building the index, the system calculates the scale parameters of all Gaussian particles in the current scene; let the... The scaling vector of the Gaussian particles is The system calculates the average scale statistics of all particles in the scene. ; Side length of the grid cell Dynamic settings are made based on this statistic to meet the requirements. ,in The preset scaling factor (e.g., 1.0 to 2.0) ensures that the grid resolution is high enough in dense detail areas and the grid span is large enough in sparse background areas, avoiding problems such as overly fine indexing or excessively large query range caused by fixed grid size.

[0040] S140, a two-stage neighborhood filtering process is performed by the neighborhood filtering module. This is for the target Gaussian particle. The system executes a neighborhood acquisition process from coarse screening to fine screening.

[0041] In the coarse screening stage, the system determines the target particles... The spatial location is used to calculate the identifier of the target mesh cell to which it belongs; the mapping relationship constructed in step S120 is then used. The system not only queries the target grid cell, but also simultaneously queries the twenty-six neighboring grid cells surrounding the target grid cell; the system extracts all particle indices within the range of these twenty-seven grid cells to form a candidate neighborhood set. This step utilizes the grid aggregation properties of Morton codes to quickly eliminate the vast majority of irrelevant particles in the scene.

[0042] In the fine screening stage, the system selects candidate neighborhood sets. Perform geometric authenticity verification; since the Z-order curve corresponding to the Morton code has abrupt changes at the spatial boundary, the candidate set may contain pseudo-neighbors with similar codes but large actual spatial distances; systematically traverse the set. Each candidate particle in Calculate its relationship with the target particle Euclidean distance : In the formula, and They represent particles respectively and particles The three-dimensional center coordinate vector is in the form of ; This represents the L2 norm operation of a vector.

[0043] The system will calculate the distance With the preset physical search radius Compare; only if satisfied Under certain conditions, the candidate particle Only then are they recognized as real physical neighbors and added to the final physical neighborhood set. The physical search radius Compared with the grid side length in step S130 There is a correlation, typically set to a scale that covers adjacent grid regions to ensure the completeness of neighborhood lookup. Through this two-stage filtering mechanism, the system can accurately obtain the local topology for subsequent density control and splitting calculations, eliminating spatial errors introduced by the index structure.

[0044] In the operational logic of the physical state perception subsystem, the system utilizes the rasterization rendering pipeline not only to generate visual images but also to simultaneously capture deep physical statistical features of the scene. By constructing a ray opacity entropy field, the system can quantify the medium obstruction patterns and complexity during the line of sight's movement in three-dimensional space, providing physical prior data for subsequent semantic understanding. This process extends the traditional 3D reconstruction process into a process of physical attribute perception.

[0045] In practice, the physical state perception processing flow is mainly executed by the rasterization rendering module, the entropy field calculation module, and the state tracking module, including:

[0046] S210, the rasterization rendering module captures tiled rasterized data. The rasterization rendering module employs a tiled rendering architecture, dividing the 2D screen space into several fixed-size tiles (e.g., 16×16 pixels). For each tile, the system calculates its view frustum extent and filters out 3D Gaussian particles whose spatial location is within that view frustum. The system sorts the filtered particles from nearest to farthest based on the vertical distance from the particle center to the camera plane (i.e., the depth value).

[0047] During rasterization, a ray is applied to each pixel on the screen. The system iterates through the sorted particle list sequentially and calculates the color contribution of each particle to the pixel. For rays... The first A Gaussian particle, with a mixed weight Due to the opacity of the particle It is determined together with cumulative transmittance. Among them, opacity It is the product of the Gaussian distribution function value of the particle covariance matrix projected onto a two-dimensional screen and the particle's own opacity parameter. Mixed weights The calculation formula is as follows:

[0048] ;

[0049] In the formula, Indicates the first The opacity of each particle is specifically calculated as follows: ,in For particle opacity parameters, The projected 2D covariance matrix. The deviation vector between the pixel coordinates and the particle projection center; This represents a series of multiplication operations. This indicates that the ray reaches the first... Before the particles, passing through the front Residual transmittance after each particle; Indicates the order along the ray depth at the position of the first The first particle before The opacity of each particle (how it is calculated) same).

[0050] During the rendering process, the system maintains a particle contribution list for each pixel ray. This list records all valid contributions on the ray (i.e., Particle indices and their corresponding mixing weights that are greater than a preset small threshold (e.g., 0.0001). This list is used not only for the weighted synthesis of the final pixel RGB colors, but also as the basic data source for subsequent calculations of ray entropy.

[0051] S220, the ray opacity entropy is calculated by the entropy field calculation module. The entropy field calculation module is based on the particle contribution list obtained in step S210. Calculate the opacity entropy of each ray. The entropy value is a dimensionless engineering statistic used to quantify the mixed uncertainty of the medium distribution in the current line of sight.

[0052] In the specific calculation, the system first processes the list. The mixed weights in the data are normalized to eliminate the influence of residual background color or transmittance, resulting in normalized weights. : It should be noted that this normalization operation is only used to calculate statistical features and does not participate in the calculation of the backpropagation gradient (i.e., gradient truncation is performed on the denominator term); subsequently, the system calculates the ray entropy using the definition of Shannon entropy:

[0053] ;

[0054] In the formula, Represents pixel rays The opacity entropy value; Represents the normalized i-th The mixed weight of each particle; To prevent the use of extremely small constants (e.g., 10⁻⁷) that are meaningless in logarithmic operations.

[0055] The opacity entropy of the ray The magnitude of the value directly reflects the physical properties of the medium through which the rays penetrate:

[0056] A lower entropy value corresponds to a single-modal central tendency in the weight distribution (i.e., weight variance). This indicates that the rays are mainly blocked by one or a very small number of opaque objects (such as hard surfaces like walls and the ground), and the uncertainty of line-of-sight penetration is low in this case.

[0057] A higher entropy value corresponds to a multimodal discrete characteristic of the weight distribution (i.e., weight variance). ) or uniform distribution characteristics (i.e., entropy value close to ,in (Effective particle number) indicates that the ray has passed through a semi-transparent object, a volume medium (such as smoke or clouds), or there is a complex sparse occlusion relationship (such as a fence or leaves), in which case the uncertainty of line-of-sight penetration is high.

[0058] By calculating this entropy value, the system can distinguish different physical modes without relying on prior semantic labels, thus providing a probabilistic basis for subsequent adaptive gating strategies.

[0059] The S230 uses a state tracking module to track the historical states of particles and manage budget. In addition to instantaneous rendering features, the state tracking module also maintains the long-term historical state of each Gaussian particle to assess its stability and importance. The system processes each particle... Assign a visibility counter And an exploration budget value .

[0060] In each frame of the rendering iteration, the system checks all particles involved in the rendering. If the particles... Mixing weights on arbitrary rays If the visibility exceeds a preset threshold, the system will update the visibility counter for that particle. Perform an accumulation operation. This counter is used to reflect the participation of particles throughout the entire training process and is not visible for a long period of time (i.e., ...). Particles with values ​​that are too low will be considered redundant and may be removed in subsequent optimizations.

[0061] Meanwhile, the state tracking module calculates the exploration budget value for each particle. Dynamic management is implemented; a budget value is explored to control the frequency at which particles break through conventional limits during gradient backpropagation; at the end of each iteration, the system recovers at a preset rate. Increase the budget value for all particles. However, its total value does not exceed the preset upper limit. ,Right now When a particle triggers a specific optimization operation (such as forced gradient injection), the system deducts the corresponding budget value. This "saving-consuming" mechanism ensures that the system can concentrate resources to make breakthroughs in specific difficult areas (such as semantically ambiguous areas) while avoiding global unstable oscillations.

[0062] In the operational logic of the semantic feedback control subsystem, the generation mechanism of the semantic gradient first needs to be clarified. This system integrates or connects a pre-trained two-dimensional semantic segmentation network (e.g., DeepLab series models based on CNN architecture or Segment-Anything-Model based on Transformer architecture). This two-dimensional semantic segmentation network is configured to receive a two-dimensional training image from the current viewpoint as input and output the corresponding pixel-level semantic label map or high-dimensional semantic feature map as the baseline ground truth. Simultaneously, the system projects the learnable semantic feature parameters carried by each Gaussian particle into the screen space through the same pipeline as rasterization rendering, generating a rendered semantic map. The system calculates the semantic loss (such as cross-entropy loss or cosine similarity loss) between the rendered semantic map and the baseline ground truth, and backpropagates this loss through an automatic differentiation mechanism, thereby generating the original semantic gradient acting on each Gaussian particle. Building upon this foundation, the modal analysis module, gradient gating module, and energy injection module utilize the ray opacity entropy output by the physical state perception module to construct a three-state physical mode determination mechanism. This mechanism aims to identify the type of physical medium penetrated by different pixel rays, mapping continuously changing entropy values ​​to discrete control states, thereby providing a decision-making basis for subsequent differentiated gradient backpropagation strategies. By dividing the ray entropy values ​​into intervals, the system can effectively distinguish between hard surface objects, complex edge transition regions, and semi-transparent volumetric media.

[0063] Specifically, the implementation process of three-state physical mode determination includes:

[0064] S310: Acquire ray entropy field data and set thresholds. The modal analysis module receives ray opacity entropy field data from the physical state sensing subsystem at the current viewpoint. The data is represented as a two-dimensional matrix with the same resolution as the screen, where each element... Corresponding pixel ray The entropy value. To classify physical modes, the system sets two entropy thresholds: a low entropy threshold and a low entropy threshold. and high entropy threshold And satisfy These two thresholds are set based on the global statistical characteristics of the scene; the system calculates the average entropy value of all rays in the current frame. and standard deviation , usually Set as ,Will Set as ,in and This is a preset scaling factor. This threshold setting method based on statistical distribution ensures that the system can adapt to scenes with different lighting conditions and geometric complexities, avoiding classification bias caused by a fixed threshold.

[0065] See attached document Figure 5 , Figure 5 This diagram presents a histogram showing the statistical distribution of ray opacity entropy values ​​in a typical scenario. The horizontal axis represents the magnitude of the entropy value, and the vertical axis represents the frequency of pixel rays corresponding to that entropy value. The system calculates the mean and standard deviation based on the statistical distribution presented in this diagram and sets a low-entropy threshold accordingly. With high entropy threshold (As shown by the dashed line in the figure). This diagram is used to illustrate the distribution of ray opacity entropy values ​​in order to understand how physical modes are classified based on statistical characteristics.

[0066] S320 performs ray-by-ray modal classification. The system traverses every pixel ray in the entropy field. its entropy value With the set threshold and The system performs a comparison; based on the comparison results, it uses a piecewise function to divide the current ray... physical modes It was determined to be a hard surface mode, a transition layer mode, or a volume medium mode.

[0067] The decision logic is as follows:

[0068] ;

[0069] In the formula, Represents pixel rays Physical mode identifier; Represents hard surface modes; Indicates the transition layer mode; This indicates the volumetric medium mode.

[0070] S330, Generate a modality identifier mapping map. The system generates a modality identifier mapping map based on the above classification results. This mapping map records the physical modality label corresponding to each pixel in the screen space.

[0071] For the hard surface mode The ray exhibits low entropy, indicating a significant unimodal characteristic in its weight distribution, with the line of sight primarily blocked by a single opaque physical surface. In this mode, the semantic information along the ray path is highly concentrated on this dominant surface, exhibiting extremely low uncertainty.

[0072] For the mode identified as a transition layer The ray's entropy value is in the middle range, indicating that the ray's weight distribution exhibits multi-peak or broad-peak characteristics. This corresponds to object edges, fine geometric structures (such as hair or vegetation), or areas with partial occlusion. In this modality, the line of sight passes through multiple contributing media layers, and semantic information needs to be distributed among these layers.

[0073] For the determined volume medium mode The ray with a high entropy value indicates that the weight distribution of the ray tends to be flat or extremely discrete; this situation corresponds to the line of sight passing through a volume medium such as smoke, flame, glass or translucent liquid; in this mode, not only the surface layer needs semantic information, but the deep medium also contributes to the final imaging, and the semantic gradient needs to have full-depth penetration.

[0074] The system generates this modal identifier mapping map and passes the physical prior information to the subsequent gradient gating module to guide the generation of semantic gradient masks for each ray. This classification mechanism avoids using a uniform gradient update strategy for all pixels, ensuring that while maintaining the sharpness of hard surfaces, it also provides sufficient optimization freedom for complex media.

[0075] See attached document Figure 3 After determining the physical mode of the pixel ray, the gradient gating module generates a corresponding gradient receiving mask based on different mode characteristics. This mask is a scalar weight sequence for each Gaussian particle on the ray, used to spatially modulate the amplitude of the semantic gradient during backpropagation. Through this mechanism, the system controls the distribution range of semantic information in the depth direction, ensuring that semantic updates only affect physical structures that actually contribute to visual imaging.

[0076] Specifically, the gradient gating module generates a dynamic gradient mask, including:

[0077] S340, Initialize the mask sequence. The system acquires pixel rays. The corresponding particle contribution list contains Gaussian particles sorted by depth and their mixed weights The system simultaneously acquires the physical mode identifiers determined by the preceding steps. For each particle on that ray. (in The system initializes a gradient mask value. The initial value is set to 0.

[0078] S350 generates a low-entropy narrow-window mask. (When the physical mode identifier...) When the mode is a hard surface, the system employs a single-peak locking strategy. The system iterates through the particle contribution list, searching for the particle index with the largest mixing weight. That is, satisfying Centered on the dominant particle, the system defines a narrow window encompassing the adjacent particles before and after it. ,in This is a preset window radius parameter (usually 1 or 2). Within this range, the system will apply the mask value... Set to 1; outside the range, the mask value remains 0. The mask calculation logic for this process is as follows: ;

[0079] This narrow-window mechanism forces the semantic gradient to propagate only to the dominant particles constituting the visible surface and their immediate surface layer particles, blocking the gradient from propagating to occluded internal particles or background particles. This prevents semantic leakage caused by depth blurring in hard surface regions, ensuring the clarity of the semantic boundaries of the object's surface.

[0080] S360 generates a medium-entropy multi-peak window mask. (When the physical mode identifier...) When the transition layer mode is active, the system executes a multi-peak detection strategy; the system first identifies all local maxima (peaks) in the mixed weight sequence, i.e., those that satisfy... and The system uses particle indexing to filter out peak values ​​with weights greater than a preset effective threshold, and then selects the top peak values ​​based on their weights. The dominant peaks constitute the peak index set. For each peak in the set The system generates a local window. The final gradient mask is the union of all local windows.

[0081] The mask calculation logic for this process is as follows:

[0082] ;

[0083] in For indicator functions, when the condition within the parentheses is true (i.e., particle index) With peak index The distance is less than or equal to the window radius. The value is 1 when the occlusion condition is met, and 0 otherwise. This multi-peak window mechanism allows the semantic gradient to act on multiple discontinuous surfaces on the ray simultaneously (such as background objects seen through a fence, or the front and back layers of leaves), solving the semantic assignment problem under complex occlusion relationships.

[0084] S370 generates a high-entropy full-window mask. When the physical mode identifier... When the mode is a volumetric medium, the system executes an all-pass strategy. Since imaging of a volumetric medium (such as smoke or fluid) is determined by the integral effect of all particles along the path, any depth truncation will lead to distortion of the physical representation; therefore, the system masks the values ​​of all effective particles along the ray. The uniform setting is 1, that is This all-through mechanism allows semantic gradients to penetrate the entire depth of the medium without hindrance, driving all particles involved in the formation of the medium to jointly optimize their semantic parameters.

[0085] S380 performs gradient modulation. This generates the final mask sequence. Then, the system applies it to the original semantic gradient during the backpropagation phase. Let the loss function be applied to the ... Semantic parameters of each particle The original gradient is The system calculates the modulated gradient. : ;

[0086] The system utilizes the modulated gradient The semantic properties of the Gaussian particle are updated; through the differentiated masking strategy under the above different modes, the system achieves precise physical constraints on the gradient flow direction.

[0087] In the semantic feedback control subsystem, to prevent potentially effective semantic information from being permanently masked due to overly strict physical mode division, the system executes step S390, which implements transient energy injection and restricted bypass control based on multi-condition constraints (the specific logic flow can be found in the appendix). Figure 3 This step allows Gaussian particles to instantaneously break through the current gradient mask constraint and perform a controlled semantic parameter update when certain physical and statistical conditions are met.

[0088] S390, the transient energy injection and restricted bypass control are performed by the energy injection module, including:

[0089] S391 monitors the semantic gradient strength signal. The system monitors in real time the action of each Gaussian particle during backpropagation. The original semantic gradient vector The system calculates the magnitude (L2 norm) of the gradient vector. and compare it with the preset activation threshold. Comparison. When When the gradient intensity exceeds this threshold, it indicates that the current particle has an error in the semantic understanding task, and this error has generated a strong correction request. This usually occurs in regions where the semantic category changes abruptly or on newly emerging, under-optimized object surfaces. The system only initiates the subsequent injection decision process when the gradient intensity exceeds this threshold in order to filter out false triggers caused by noise signals.

[0090] S392, Verifying geometric orthogonality constraints. To ensure that semantic updates do not disrupt the geometric stability of the scene, the system detects the orthogonality between the semantic gradient direction and the geometric / photometric gradient direction; the system acquires the orthogonality of the gradients acting on the same particle. geometric gradient vector (Usually composed of positional gradients or spherical harmonic coefficient gradients). The system calculates the absolute value of the cosine similarity between two vectors. :

[0091] ;

[0092] In the formula, Defined as a particle The geometric parameter gradient vector, which is derived from the position gradient Rotation gradient and scaling gradient It is pieced together; This represents the vector dot product operation. To prevent division by zero and extremely small quantities, the system will calculate the... With the preset orthogonal tolerance threshold (Usually, a small value close to 0 is used for comparison.) Only when... When this occurs, it indicates that the update direction of the semantic gradient is mainly orthogonal to the direction of geometric change, meaning that changes in semantic attributes will not cause particle position drift or texture collapse. At this point, the system determines that the update request conforms to the physical consistency constraint.

[0093] S393, Verification and Deduction of Exploration Budget. After passing the strength and orthogonality verification, the system checks the particle's current exploration budget value. This budget value is maintained and dynamically restored by the physical state sensing subsystem in step S230; the system determines... Is it greater than the energy cost required for a single injection? .like If the budget is sufficient, the bypass operation is allowed. Simultaneously, the system immediately updates the particle's budget status and performs the deduction operation. ;

[0094] This deduction mechanism has a forced cooling effect, preventing the same particle from frequently triggering energy injection in a short period of time, thus preventing the system from falling into high-frequency oscillations.

[0095] S394, Perform transient mask bypass; when all conditions in S391 to S393 are simultaneously met, the system generates a transient bypass signal. This signal acts on the gradient mask generated in the previous step. This forces the mask value of the particle in the current frame to be reset to the all-pass state (i.e., set to 1.0), ignoring the limitations of the physical mode to which the particle currently belongs (such as hard surfaces or transition layers).

[0096] This bypass control has strict transient and budget constraints; the system only temporarily modifies the mask value during the backpropagation phase of the current rendering frame; once the parameters of the current frame are updated, the bypass state immediately expires, and the particle mask generation logic automatically reverts to the conventional gating mechanism based on ray entropy in the next frame. This design ensures that energy injection is only used as a sparse "pulse" correction method to break local minimum deadlocks without changing the long-term gradient convergence trajectory of the particles; by limiting the frequency and conditions of injection, the system maintains the stability of global optimization while providing optimization flexibility, avoiding scene geometry degradation caused by unrestricted semantic gradient leakage.

[0097] To further illustrate the dynamic management mechanism of the exploration budget, please refer to the appendix. Figure 4 , Figure 4 This graph shows the temporal variation of the exploration budget value of a single Gaussian particle over consecutive time steps. The horizontal axis represents the number of rendered frames, and the vertical axis represents the particle's budget value. As can be seen, the budget value exhibits a "sawtooth" pattern of change: during the stable phase, the budget value recovers at a fixed rate. Linear growth; when a steep falling edge appears in the figure, it corresponds to the particle triggering the transient energy injection condition (i.e., satisfying the gradient strength and orthogonality requirements in step S390), and the system performs a budget deduction operation; this curve proves that the present invention can achieve targeted semantic mutations while ensuring system stability through dynamic resource scheduling.

[0098] During the operation of the geometric evolution subsystem, adaptive orthogonal gradient updates are performed through the orthogonal update module, photometric locking module, and structural splitting module, aiming to resolve gradient conflict issues in multi-task optimization. This process dynamically adjusts the degree of orthogonality of the semantic gradient relative to the geometric gradient based on the current photometric convergence state of the scene, thereby achieving effective embedding of semantic features while ensuring the stability of the 3D structure.

[0099] Specifically, the orthogonal update module performs adaptive orthogonalized gradient updates, including:

[0100] S411, the orthogonal update module calculates the relative stability threshold and evaluates the convergence state. The system obtains the current Gaussian particle... Photometric gradient vector and semantic gradient vector Photometric gradient Derived from the RGB color differences and structural similarity loss between the rendered and real images, this reflects the physical accuracy of particles in terms of geometric position and appearance attributes. The system calculates the modulus of the photometric gradient. This is used as an indicator to measure the stability of the particle's geometric state. Simultaneously, the system maintains a global or local relative stability threshold. This threshold is not a fixed constant, but is dynamically set based on the average photometric gradient modulus of all particles in the current training phase; the system compares... and The relationship is used to determine the current convergence state.

[0101] S412, Determine the orthogonalization control coefficients. Based on the evaluation results of step S411, the system switches between two strategies: "strict orthogonality" and "relaxed orthogonality," and determines the orthogonalization control coefficients accordingly. .

[0102] when This indicates that the region where the particle resides has not yet reached geometric and aesthetic stability, with significant photometric errors and a rapidly evolving geometric structure. At this point, the system employs a strictly orthogonal strategy, orthogonalizing the control coefficients... Set to 1 (or a value close to 1). The purpose of this strategy is to forcibly remove all components in the semantic gradient that are parallel to the photometric gradient, preventing the direction of semantic optimization from interfering with the normal convergence of the geometric structure and avoiding geometric collapse or artifacts.

[0103] when When the time is right, it indicates that the geometric structure of the particle region has stabilized and the photometric error is at a low level; at this time, the system adopts a relaxed orthogonal strategy to orthogonalize the control coefficients. Set to a decay function value between 0 and 1 (e.g.) This strategy allows semantic gradients to retain components that are in the same direction as the photometric gradients, using semantic information to fine-tune the geometry and thus optimize the alignment accuracy of object edges.

[0104] S413, Perform gradient projection correction; the system utilizes the determined control coefficients. For the original semantic gradient Perform projection correction to generate the final corrected gradient used for parameter updates. The modified calculation follows a variant of the Gram-Schmidt orthogonalization principle, as shown in the following formula:

[0105] ;

[0106] In the formula, This represents the corrected semantic gradient vector; This represents the semantic gradient vector obtained from the original backpropagation; Represents the photometric gradient vector; Represents the dot product operation of vectors; The magnitude squared represents the luminance gradient; To prevent extremely small constants with a denominator of zero; The orthogonalization control coefficients determined in step S412 have a value range of [0,1].

[0107] S414, Apply gradient update. The system uses the corrected gradient. The properties of the Gaussian particle (including position, rotation, scaling, and spherical harmonic coefficients) are updated. Through the above mechanism, the system clarifies the dominant role of photometric consistency in the optimization process, and establishes a controlled feedback channel for semantic information to geometric evolution, achieving a dynamic balance between geometric accuracy and semantic accuracy at different convergence stages.

[0108] In the geometry evolution subsystem, to prevent high-intensity semantic gradients from causing unnecessary over-segmentation in regions where the geometry is already accurate, a photometric residual locking mechanism is introduced. This mechanism acts as a conditional constraint, verifying the photometric fitting quality of the particles before performing the Gaussian particle encryption operation, ensuring that changes in geometric topology only occur in regions where visual reconstruction errors are significant.

[0109] In specific implementation, the photometric residual locking is performed by the photometric locking module, including:

[0110] S421, the photometric locking module calculates particle-level photometric residuals. The system first calculates the color reconstruction error (i.e., the L1 distance between the rendered image and the real image) for each pixel in screen space during rasterization. Then, the system back-projects these pixel-level errors onto the Gaussian particles involved in the rendering to evaluate the contribution of individual particles to the local photometric error. For the first... A Gaussian particle, with its average luminous residual It is calculated by weighted accumulation of the errors of all pixels within its coverage area:

[0111] ;

[0112] In the formula, Indicates the particle The set of pixels affected; Represents particles In pixels Mixed weights (opacity contribution) at the location; and Representing pixels The rendered color vector and the true base color vector at that location; Represents L1 norm operations; To prevent extremely small constants with a denominator of zero, this residual value... This intuitively reflects the accuracy of the fit between the spatial region represented by the particle and the texture and lighting.

[0113] S422, Set the dynamic residual threshold. The system sets the photometric residual threshold based on the global statistical characteristics of the current training iteration. This threshold is typically set as a multiple of the mean photometric residual of all active particles, or as a function that decays exponentially with the number of training steps; the purpose of setting this threshold is to define what constitutes “visually acceptable” reconstruction quality.

[0114] S423, Execute conditional split decision. In the decision-making phase for geometric evolution (i.e., splitting or cloning), the system employs a "double verification" logic; traditional methods rely solely on positional gradients. Does it exceed the location threshold? This determines whether to encrypt the particles. In this embodiment, the system adds a check for photometric residuals. The check only allows splitting or cloning operations when the particle simultaneously satisfies both the position gradient condition and the photometric residual condition.

[0115] This decision logic can be expressed as: In the formula, Represents the logical AND operation.

[0116] when Even at that time If a threshold is exceeded (which could be caused by strong gradients at semantic boundaries rather than geometric defects), the system will also forcibly "lock" the particle, preventing it from splitting. This locking operation indicates that the geometry of the region is sufficient to support the current visual appearance, without the need to increase geometric complexity. Through this mechanism, the system effectively suppresses unwanted fragmentation artifacts in textured but geometrically flat regions (such as patterned wallpaper or carpet), ensuring that computational resources are focused on correcting genuine geometric defects.

[0117] In the geometric evolution subsystem, to ensure that the encryption operation of Gaussian particles can accurately fit the geometric details of the object's surface without introducing non-physical volume expansion or surface thickening, the system executes a tangential structured splitting strategy. The core of this strategy lies in constructing a local tangential plane coordinate system by analyzing the local covariance properties of the particles, and strictly constraining the new particles generated by the splitting within this two-dimensional tangential plane, thereby maintaining the thin-shell characteristics of the geometric structure.

[0118] In specific implementation, the structural splitting module performs tangential structural splitting, including:

[0119] S431, the structural splitting module extracts local geometric feature bases. The system acquires the parent Gaussian particle to be split. covariance matrix Since the covariance matrix of a three-dimensional Gaussian particle is a symmetric positive definite matrix, it characterizes the ellipsoidal shape of the particle in three-dimensional space. To determine the principal axis directions of the particle, the system... Perform eigenvalue decomposition or directly utilize the rotation matrix used to construct the covariance matrix. and scaling vector .

[0120] System identification scaling vector The minimum component in is denoted as In most scenarios that exhibit surface structures, the direction vector corresponding to this minimum scaling component is the normal direction of the surface. This is because it represents the direction where the particle thickness is thinnest. Correspondingly, the direction vectors corresponding to the two larger components of the scaling vector. and This forms a local basis tangent to the surface; this set of basis vectors The local geometric tangent plane where the particle is located is defined.

[0121] S432, Constructing the tangential displacement sampling space. After determining the local tangential plane, the system achieves structured splitting by restricting the generation space of the displacement vector; the system rejects isotropic random sampling in the entire three-dimensional space, and instead constructs a displacement sampling function that contains only the tangential component.

[0122] Specifically, for each sub-particle to be generated, the system generates a pair of scalar sampling coefficients. and These two coefficients follow a distribution with a mean of 0 (e.g., a truncated normal distribution or a uniform distribution), and their amplitudes are proportional to the tangential scale of the parent particle. The system explicitly forces the sampling coefficients of the normal component to be 0, i.e., no sampling coefficients are generated along the vector. Displacement in direction.

[0123] S433, perform restricted position update. The system uses the aforementioned tangential basis vectors and sampling coefficients to calculate the sub-particles. Relative to the parent particle central position The position update formula is as follows:

[0124] ;

[0125] In the formula, and These represent the three-dimensional center coordinate vectors of the child particle and the parent particle, respectively; and Represents the two orthogonal unit tangential vectors determined in step S431; and The tangential displacement coefficient is obtained from sampling. Using this formula, the system ensures that the center position of the child particle falls precisely on the tangential plane defined by the parent particle, and the resulting geometric change is only manifested as a lateral extension along the surface, without introducing additional normal thickness.

[0126] S434, Adjust sub-particle shape parameters. While updating the position, the system anisotropically adjusts the scaling properties of the sub-particles; the system reduces the sub-particle's scaling in the tangential direction. and The scaling factor is adjusted to reflect the reduced coverage area after the surface is subdivided; meanwhile, the system maintains or only fine-tunes the normal direction. The scaling ratio is adjusted to maintain the physical thickness constraint of the surface. This anisotropic parameter inheritance mechanism ensures that the split particle set still closely fits the original geometric surface on a macroscopic scale, avoiding the surface "bloating" or blurring caused by particle stacking, which is common in traditional splitting algorithms. This achieves a refined and sharpened expression of the geometric structure.

[0127] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1.A scene semantic understanding system based on a 3D Gaussian splash model, configured to synchronously analyze scene semantic information in a three-dimensional scene reconstruction process, characterized in that, Comprise: a spatial topology construction subsystem configured to establish a spatial index structure of a set of Gaussian particles and obtain a set of physical neighborhoods of a target particle based on the spatial index structure; a physical state perception subsystem configured to calculate a ray opacity entropy value according to a particle contribution list in a rasterization rendering pipeline and track a historical visibility state of a Gaussian particle; wherein the physical state perception subsystem comprises an entropy field calculation module; the entropy field calculation module is configured to obtain the particle contribution list generated in the rasterization process, normalize the mixing weights in the particle contribution list, and calculate a Shannon entropy based on the normalized weights to obtain the ray opacity entropy value; the ray opacity entropy value is used to quantify the mixing uncertainty of the medium distribution in the line-of-sight direction; a semantic feedback control subsystem configured to determine a physical modality of a pixel ray according to the ray opacity entropy value and generate a gradient reception mask according to the physical modality to adjust a semantic gradient propagation path; wherein the semantic feedback control subsystem comprises a modality analysis module; the modality analysis module is configured to calculate the mean and standard deviation of all ray opacity entropy values under the current viewing angle, set a low entropy threshold and a high entropy threshold according to the statistical result, and classify the pixel ray into a hard surface modality, a transition layer modality or a volumetric medium modality according to the comparison result of the ray opacity entropy value with the low entropy threshold and the high entropy threshold; the semantic feedback control subsystem further comprises a gradient gating module, and the gradient gating module is configured to generate the gradient reception mask according to the physical modality: for the hard surface modality, a unimodal narrow window mask centered on the particle with the maximum weight is generated, and the semantic gradient is only allowed to propagate to the dominant surface particle; for the transition layer modality, local peaks in the mixing weight sequence are identified, and a union mask containing multiple local windows is generated; for the volumetric medium modality, a full-pass mask is generated, allowing the semantic gradient to penetrate all effective particles on the pixel ray. The geometric evolution subsystem is configured to correct the semantic gradient based on the photometric convergence state to update the Gaussian particle parameters, and perform structured splitting based on tangent planes; wherein the geometric evolution subsystem comprises an orthogonal update module; the orthogonal update module is configured to calculate the modulus of the photometric gradient and compare it with a relative stability threshold; when the modulus of the photometric gradient is greater than the relative stability threshold, a strict orthogonal strategy is adopted to remove the component parallel to the photometric gradient in the semantic gradient; when the modulus of the photometric gradient is less than or equal to the relative stability threshold, a relaxed orthogonal strategy is adopted to retain part of the parallel component; the orthogonal update module updates the geometric parameters of the Gaussian particle using the corrected semantic gradient; the geometric evolution subsystem further comprises a structure splitting module; the structure splitting module is configured to perform eigenvalue decomposition on the covariance matrix of the parent Gaussian particle that meets the splitting condition, identify the minimum scaling component representing the normal direction and the tangent vector representing the tangent plane direction; when generating a child particle, a displacement sampling function containing only the tangent component is constructed, the center position of the child particle relative to the parent Gaussian particle is calculated, so that the child particle is located in the local geometric tangent plane defined by the parent Gaussian particle, and the anisotropic adjustment of the scaling attribute of the child particle is performed. 2.The 3D Gaussian splash model based scene semantic understanding system of claim 1, wherein, The spatial topology construction subsystem comprises a Morton coding module and a grid index module; The Morton coding module is configured to obtain Gaussian particle center coordinates, quantize the Gaussian particle center coordinates, and then perform binary bit interleaving to generate a Morton code, and use radix sorting to generate an ordered particle index list; The grid index module is configured to traverse the ordered particle index list, detect the grid identification change of adjacent particles, establish a mapping relationship from the spatial grid cell identification to the particle list index range, and construct a grid range lookup table. 3.The 3D Gaussian splash model based scene semantic understanding system of claim 2, wherein, The spatial topology construction subsystem further comprises a neighborhood filtering module; The neighborhood filtering module is configured to set the grid cell size according to the average scale statistics of the Gaussian particles in the scene, query the grid range lookup table according to the target particle position to obtain the candidate particles in the target grid cell and the neighborhood grid cells, calculate the Euclidean distance between the target particle and the candidate particles, and retain the particles with a Euclidean distance less than the physical search radius as the physical neighborhood set. 4.The 3D Gaussian splash model based scene semantic understanding system of claim 1, wherein, The semantic feedback control subsystem further comprises an energy injection module; The energy injection module is configured to monitor the modulus of the original semantic gradient, and calculate the cosine similarity between the direction of the original semantic gradient and the gradient direction of the geometric parameters; When the modulus of the original semantic gradient exceeds the activation threshold, the cosine similarity satisfies the orthogonality constraint, and the exploration budget value of the Gaussian particle is sufficient, the energy injection module generates a bypass control signal, deducts the corresponding exploration budget value, and forcibly resets the gradient reception mask of the current frame to a full-pass state. 5.The 3D Gaussian splash model based scene semantic understanding system of claim 1, wherein, The geometric evolution subsystem further comprises a photometric locking module; The photometric locking module is configured to back-project the color reconstruction error in the screen space to the Gaussian particle, and calculate the particle-level photometric residual error; In determining a particle splitting condition, a splitting enable signal is generated only when a position gradient of the Gaussian particle exceeds a position threshold and a photometric residual of the particle level exceeds a photometric residual threshold; otherwise a photometric lock state is maintained, prohibiting the Gaussian particle from splitting.

Images