A spatial perception-oriented gaussian spatter level autonomous switching method and system

Abstract

The application provides a Gaussian splashing level autonomous switching method and system facing space perception, and the application constructs a level division result containing a basic level, a fine level and a super-fine level for a three-dimensional scene dense Gaussian cell; renders the level division of the Gaussian cell according to a real-time level comprehensive score; constructs a corresponding rendering pipeline for the Gaussian cell of different rendering levels, screens invalid and redundant Gaussian cells, and performs smooth transition processing on the rendering parameters of the target Gaussian cell that occurs level switching. The application performs sparse division on the dense Gaussian cell through an octree, divides the Gaussian cell into different levels based on semantic clustering fusion, realizes high-precision reservation of key areas, improves the detail rendering precision through the rendering of the Gaussian cell of different levels, and realizes inter-frame level switching through smooth transition processing, effectively solving the rendering artifacts such as picture flicker, edge jump and detail fault caused by the level mutation of the prior art.

Description

Technical Field

[0001] This invention relates to the fields of 3D scene rendering, 3D reconstruction, computer vision, and virtual reality, and in particular to a method and system for autonomous switching of Gaussian splashing levels for spatial perception. Background Technology

[0002] Real-time rendering and novel viewpoint synthesis of 3D scenes are core foundational technologies in fields such as virtual reality, augmented reality, 3D reconstruction, digital twins, and autonomous driving simulation. Compared to traditional NeRF-based implicit rendering methods, which suffer from low training and inference efficiency and difficulty in meeting real-time rendering requirements, 3D Gaussian Splatting (3DGS), as a novel explicit 3D scene representation technology, has become a mainstream research and application direction in the field of 3D visual rendering due to its efficient rasterization rendering process, high-precision detail restoration capabilities, and rapid scene reconstruction advantages. This technology achieves high-quality novel viewpoint rendering by fitting scene geometry and texture colors with a massive number of 3D Gaussian ellipsoidal primitives.

[0003] However, existing native 3D Gaussian splashing technology has inherent technical flaws that severely restrict its large-scale application in large-scale scenes, dynamic viewpoints, and multi-hardware platform scenarios. During large-scale 3D scene rendering, a massive number of redundant Gaussian primitives with low pixel contribution, long distances, and complete occlusion are generated, continuously consuming high amounts of video memory and GPU computing power. This easily leads to problems such as video memory overflow and sudden drops in rendering frame rates, making it impossible to achieve a dynamic balance between rendering performance and image quality. Static, fixed rendering modes are difficult to adapt to different working conditions such as stationary camera observation, slow-speed roaming, and high-speed scanning. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a method and system for autonomous switching of Gaussian splashing levels based on spatial awareness, aiming to overcome the technical deficiencies of existing 3D Gaussian splashing.

[0005] In a first aspect, the present invention provides a method for autonomous switching of Gaussian splashing levels for spatial awareness, comprising: Spatial octree sparse partitioning and semantic clustering are performed on dense Gaussian elements in a 3D scene to construct a hierarchical partitioning result containing basic, fine, and ultra-fine levels, and spatial semantic masks are generated for Gaussian elements at each level. Real-time acquisition of intrinsic and extrinsic parameters and motion parameters of the rendering camera is used to extract the spatial static features and temporal dynamic features of the current Gaussian elements, and the spatial static features and temporal dynamic features are concatenated into a perception feature matrix; The real-time hierarchical comprehensive score of the Gaussian primitive is calculated based on the perceptual feature matrix, and the rendering hierarchy of the Gaussian primitive is divided based on the real-time hierarchical comprehensive score. Construct corresponding rendering pipelines for Goski primitives at different rendering levels, filter invalid and redundant Goski primitives based on the spatial static features, temporarily mask the invalid and redundant Goski primitives, and update the visual state of the masked invalid and redundant Goski primitives in real time. The rendering parameters of the target Gaussian pixel undergoing the level switch are smoothly transitioned based on the original rendering parameters of the Gaussian pixels before and after the switch.

[0006] Secondly, the present invention provides a spatially aware Gaussian splashing hierarchy autonomous switching system, comprising: The Gaussian hierarchy partitioning module is used to perform spatial octree sparse partitioning and semantic clustering fusion on dense Gaussian elements in a 3D scene, constructing a hierarchy partitioning result that includes basic, fine, and ultra-fine levels, and generating spatial semantic masks for each level of Gaussian elements. The perceptual feature extraction module is used to collect the intrinsic and extrinsic parameters and motion parameters of the rendering camera in real time, so as to extract the spatial static features and temporal dynamic features of the current Gaussian unit, and to concatenate the spatial static features and temporal dynamic features into a perceptual feature matrix. The rendering level division module is used to calculate the real-time level comprehensive score of Gaussian elements based on the perceptual feature matrix, and to divide the rendering level of Gaussian elements based on the real-time level comprehensive score. The rendering execution module is used to construct corresponding rendering pipelines for Goski elements at different rendering levels, filter invalid and redundant Goski elements based on the spatial static features, temporarily mask the invalid and redundant Goski elements, and update the visual status of the masked invalid and redundant Goski elements in real time. The rendering smooth transition module is used to perform smooth transition processing on the rendering parameters of the target Gaussian pixel undergoing a level switch, based on the original rendering parameters of the Gaussian pixels before and after the switch.

[0007] Thirdly, the present invention provides an electronic device, comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, it implements the space-aware Gaussian splashing level autonomous switching method.

[0008] The beneficial technical effects of this invention are as follows: 1. This invention uses an octree to sparsely partition dense Gaussian units and uses semantic clustering fusion to divide Gaussian units into different levels, achieving high-precision preservation of key areas; by rendering different levels of Gaussian units, the accuracy of detail rendering is improved, and at the same time, smooth transition processing is used to achieve inter-frame level switching, effectively solving rendering artifacts such as screen flickering, edge jumps, and detail breaks caused by abrupt changes in level in existing technologies, and greatly improving the 3D roaming visual experience.

[0009] 2. This invention integrates spatial static features and temporal dynamic features, adapting to conditions such as camera stationary, low-speed roaming, and high-speed scanning.

[0010] 3. This invention only temporarily masks redundant primitives in video memory, and can quickly resume rendering after the viewpoint changes. While reducing computing power consumption, it completely avoids the problem of permanent loss of scene details.

[0011] 4. This invention features smooth transition processing, resulting in higher precision in layer switching, while maintaining both static detail accuracy and dynamic rendering smoothness. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the method of an embodiment of the present invention; Figure 2 This is a schematic diagram of the hierarchical division process according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the Gaussian primitive rendering process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the system framework according to an embodiment of the present invention; In the diagram, 10 is the Gaussian hierarchy partitioning module; 20 is the perceptual feature extraction module; 30 is the rendering hierarchy partitioning module; 40 is the rendering execution module; and 50 is the rendering smooth transition module. Detailed Implementation

[0013] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings: like Figure 1 As shown, this embodiment provides a spatially aware Gaussian splashing hierarchy autonomous switching method, including: S1: Perform spatial octree sparse partitioning and semantic clustering fusion on dense Gaussian elements in a 3D scene to construct a hierarchical partitioning result containing basic, fine, and ultra-fine levels, and generate spatial semantic masks for Gaussian elements at each level.

[0014] Compared to traditional single-layer hierarchical schemes that rely solely on primitive scales, this embodiment uses an octree to sparsely partition dense Gaussian primitives and then uses semantic clustering fusion to divide the Gaussian primitives into different levels, thereby achieving high-precision preservation of key regions.

[0015] S2: Real-time acquisition of camera intrinsic and extrinsic parameters and motion parameters to extract the spatial static and temporal dynamic features of the current Gaussian elements, and concatenation of the spatial static and temporal dynamic features into a perceptual feature matrix.

[0016] This embodiment integrates static spatial features with dynamic camera motion features, adapting to scenarios involving a stationary camera, low-speed roaming, and high-speed scanning. It addresses the technical problems of existing technologies that cannot dynamically render and ignore inter-frame scene changes.

[0017] S3: Calculate the real-time hierarchical comprehensive score of Gaussian primitives based on the perceptual feature matrix, and divide the Gaussian primitives into rendering levels based on the real-time hierarchical comprehensive score.

[0018] S4: Construct corresponding rendering pipelines for Goski elements at different rendering levels, and filter invalid redundant Goski elements based on the spatial static features. Perform temporary masking on the invalid redundant Goski elements, and update the visual status of the masked invalid redundant Goski elements in real time.

[0019] This embodiment builds an independent and parallel differentiated GPU rendering pipeline for the three-level rendering hierarchy to realize on-demand allocation of computing resources. At the same time, by temporarily shielding invalid and redundant high-level primitives, it not only significantly reduces the real-time GPU computing power and memory overhead, solving the problems of memory overflow and low frame rate in large-scene rendering, but also completely eliminates the defects of permanent loss of scene details and incomplete image after perspective change caused by traditional pruning.

[0020] S5: Perform smooth transition processing on the rendering parameters of the target Gaussian primitive that has undergone a level switch based on the original rendering parameters of the Gaussian primitive before and after the switch.

[0021] This embodiment effectively solves the rendering artifacts commonly found in Gaussian layered rendering, such as screen flickering, edge jumps, detail breaks, and inter-frame jitter, through smooth transition processing.

[0022] In this embodiment, in step S1, spatial octree sparse partitioning and semantic clustering are performed on the dense Gaussian elements of the 3D scene to construct a hierarchical partitioning result containing basic, fine, and ultra-fine levels, and spatial semantic masks are generated for each level of Gaussian elements, such as... Figure 2 As shown, it includes: S11: Perform spatial octree sparse partitioning on the dense Gaussian units of the 3D scene to obtain several sub-voxel units, including: By iterating through the center coordinates of all Gaussian elements within the 3D scene, the boundary of the scene space bounding box is determined; that is: ; In the formula, , These are the X-axis boundaries of the bounding box; , These are the Y-axis boundaries of the bounding box; , These are the Z-axis boundaries of the bounding box; For the first The center coordinates of each Gaussian element; This represents the total number of Gaussian elements.

[0023] Within the bounding box, the maximum recursion depth of the octree is preset. Threshold for the maximum number of primitives that a single voxel can hold If the number of Gaussian units in the current voxel to be subdivided is greater than the threshold And the current recursion depth is less than the maximum recursion depth If the current voxel is divided into three equal parts along the axis, it will generate 8 sub-voxel units; otherwise, the division will stop and the current local space grouping will be completed.

[0024] For each Gaussian cell, its sub-voxel is determined based on its center coordinates. After the subdivision is completed, empty voxels with zero Gaussian cells are removed.

[0025] S12: Construct the covariance matrix and perform eigenvalue decomposition on the Gaussian ellipsoidal coordinates of the sub-voxel units, obtain the decomposed eigenvalues, solve for the scene geometric curvature of the sub-voxel units based on the decomposed eigenvalues, solve for the equivalent spatial scale of the Gaussian ellipsoidal units based on the Gaussian ellipsoidal covariance matrix, and normalize the scene geometric curvature and the equivalent spatial scale of the primitives; that is: ; ; In the formula, The scene geometry curvature of the current sub-voxel unit; This is the equivalent spatial scale of the current Gaussian unit; , , These are the three eigenvalues ​​of the covariance matrix of the primitive spatial coordinates within the sub-voxel unit, and ; , , These are the three principal axis scale parameters of the Gaussian ellipsoid.

[0026] If the scene geometry curvature of the current sub-voxel unit If the curvature is greater than or equal to the preset curvature threshold, the corresponding sub-voxel unit is identified as a high curvature detail region; otherwise, it is identified as a low curvature flat region.

[0027] S13: Based on the scene semantic segmentation results, assign corresponding semantic weights to Gaussian primitives of different semantic categories, and construct hierarchical judgment features by combining the normalized scene geometric curvature and primitive equivalent spatial scale. Based on the hierarchical judgment features, divide the primitives in the sub-voxel unit into basic level, fine level and ultra-fine level.

[0028] The hierarchical determination feature is represented as follows: ; In the formula, Features for stratification; , These are the scene geometric curvature and the equivalent spatial scale of the primitives, respectively, after normalization. For semantic weights; , , These are the corresponding weighting coefficients.

[0029] Based on the hierarchical determination characteristics, the Gaussian units within the sub-voxel units are divided into basic, fine, and super-fine levels, specifically: If the hierarchical determination features satisfy Then the current Gaussian unit will be classified into the basic level.

[0030] If the hierarchical determination features satisfy Then the current Gaussian unit will be divided into fine-grained levels.

[0031] If the hierarchical determination features satisfy Then the current Gaussian unit will be divided into a super-fine level.

[0032] in, and These are the threshold values ​​for the first level and the second level, respectively.

[0033] S14: Combining semantic weights and hierarchical partitioning results, a binary semantic mask matrix is ​​generated for each sub-voxel unit. The effective mask area corresponds to the key rendering area of ​​the scene, and the invalid mask area corresponds to the lightweight rendering area; that is: ; In this embodiment, if the semantic weight of the current Gaussian unit... Greater than the semantic weight determination threshold And hierarchical determination features Greater than or equal to the first-level threshold If the condition is met, the mask of the current Gaussian element is set to 1; otherwise, it is set to 0. This embodiment distinguishes key areas and background areas of a scene using semantic masks, solving the technical problems of coarse layering and loss of key details in existing technologies.

[0034] In the formula, The threshold for semantic weight determination. For binary semantic masking, Indicates the key rendering areas of the scene; Corresponding to lightweight rendering areas.

[0035] In this embodiment, step S2 involves real-time acquisition of the rendering camera's intrinsic and extrinsic parameters and motion parameters to extract the spatial static and temporal dynamic features of the current Gaussian elements. These spatial static and temporal dynamic features are then concatenated into a perceptual feature matrix, including: S21: Acquire and render the camera's intrinsic and extrinsic parameters and motion parameters frame by frame, and calculate the distance between the camera and the current Gaussian pixel based on the current frame's optical center 3D coordinates and the current Gaussian pixel center coordinates. .

[0036] The motion parameters include the change in camera pose between consecutive frames and the frame interval time.

[0037] S22: Based on the distance between the camera and the Gaussian unit The X and Y principal axis scale parameters of the Gaussian primitive map the three-dimensional spatial scale of the Gaussian primitive to the two-dimensional screen pixel space to obtain the screen projection pixel diameter of the current Gaussian primitive, thereby quantifying the visual contribution of the Gaussian primitive to the final rendered image; that is: ; In the formula, The screen projection pixel diameter of the current Gaussian pixel; To fix the scale constant for the camera's intrinsic parameters, These are the X and Y principal axis scale parameters of the Gaussian element.

[0038] S23: By calculating the camera's real-time rotational angular velocity using the changes in camera attitude between consecutive frames and the frame interval, the speed of camera scanning and rotation can be quantified. ; In the formula, For the camera's real-time angular velocity; The magnitude of the change in camera pose between consecutive frames; The frame interval between two adjacent frames.

[0039] S24: Calculate the motion compensation coefficient based on the camera's real-time rotational angular velocity, i.e.: ; In the formula, The motion compensation coefficient for the camera; This is the preset maximum effective angular velocity of the camera.

[0040] S25: Based on scene depth caching and rasterization occlusion detection, the proportion of visible pixels of a single Gaussian pixel to the total number of pixels is calculated to obtain the local occlusion rate, i.e.: ; In the formula, The local occlusion rate of the current Gaussian element; The number of pixels in the Gaussian pixel projection area that are occluded by the foreground. This represents the total number of pixels projected onto the Goshawk screen.

[0041] S26: Distance between the camera and the current Gaussian unit The current screen projection pixel diameter P of the Gaussian pixel, and the current local occlusion rate of the Gaussian pixel. The semantic weight of the current Gaussian unit The splicing is used to obtain the current spatial static features of the Gaussian elements; the motion compensation coefficients are then used. As a time-dynamic feature, the spatial static feature and the time-dynamic feature are concatenated into a perceptual feature matrix.

[0042] In this embodiment, step S3 involves calculating the real-time hierarchical comprehensive score of Gaussian elements based on the perceptual feature matrix, and then dividing the Gaussian elements into rendering hierarchical levels based on the real-time hierarchical comprehensive score, including: S31: The spatial static features are weighted and fused, and camera motion compensation coefficients are introduced to complete dynamic correction. Real-time hierarchical comprehensive scores are calculated for each individual Gaussian unit, i.e.: ; In the formula, This is a real-time, hierarchical comprehensive score; , , , These are the weight coefficients for the corresponding terms, and =1.

[0043] In this embodiment, the motion compensation coefficient is dynamically adjusted according to the real-time motion state of the camera. When the camera is stationary and observing statically, the motion compensation coefficient mentioned above... When the motion compensation coefficient approaches 1, it is reduced when the camera is roaming at low speed. When the camera is scanning at high speed, increase the motion compensation coefficient. .

[0044] S32: Real-time hierarchical comprehensive score based on Gaussian elements The rendering hierarchy is divided into Gaussian elements; that is: If Gao Siyuan's real-time hierarchical comprehensive score If the value is greater than or equal to 0 and less than the first rendering threshold, then the current high-level primitive is rendered using the basic level.

[0045] If Gao Siyuan's real-time hierarchical comprehensive score If the value is greater than or equal to the first rendering threshold and less than the second rendering threshold, then the current high-level primitive is rendered using a fine-level rendering method.

[0046] If Gao Siyuan's real-time hierarchical comprehensive score If the value is greater than or equal to the second rendering threshold and less than or equal to 1, then the current high-level primitive is rendered using an ultra-fine rendering layer.

[0047] In this embodiment, in step S4, corresponding rendering pipelines are constructed for Gaussian primitives at different rendering levels, and invalid redundant Gaussian primitives are filtered based on the spatial static features. These invalid redundant Gaussian primitives are temporarily masked, and the visual state of the masked invalid redundant Gaussian primitives is updated in real time. Figure 3 As shown, it includes: S41: Construct independent rendering pipelines for the Gaussian primitives of the basic rendering level, fine rendering level, and ultra-fine rendering level.

[0048] In this embodiment, for the Gaussian primitives of the basic rendering layer, distant background rendering is used, and subpixel subdivision sampling is turned off.

[0049] For the high-level rendering of the aforementioned high-level primitives, conventional scene rendering is adopted, and standard subpixel sampling and full covariance calculation are enabled.

[0050] For the high-resolution primitives of the ultra-fine layer rendering, close-up key detail rendering is adopted, and high-precision subpixel subdivision sampling, edge anti-aliasing optimization, and complete texture and color iterative calculation are enabled to maximize the preservation of scene object outlines, textures, and bump details.

[0051] S42: Filter invalid redundant Gaussian cells frame-by-frame based on the distance between the camera and the current Gaussian cell, the Gaussian cell's screen projection pixel diameter, and the Gaussian cell's local occlusion rate, including: In this embodiment, if the distance between the camera and the current high-level primitive is greater than a preset distant distance threshold, the current high-level primitive is determined to be an invalid redundant high-level primitive.

[0052] If the screen projection pixel diameter of the current high-order element is less than the minimum pixel projection threshold, then the current high-order element is determined to be an invalid redundant high-order element.

[0053] If the local occlusion rate of the current Gaussian element is greater than or equal to the maximum occlusion threshold If so, the current Gaussian element is determined to be an invalid redundant Gaussian element.

[0054] In some embodiments, the maximum occlusion threshold Set it to 0.95.

[0055] S43: Temporarily mask the invalid redundant Gaussian elements, removing them only from the rasterization calculation and pixel fusion process in the current rendering frame, while retaining the geometry, color, transparency, and covariance parameters.

[0056] S44: After completing the masking operation for redundant and invalid Gaussian primitives in each frame, a redundant primitive index cache table is established. For all masked invalid Gaussian primitives in the cache table, the distance between the camera and the current Gaussian primitive, the screen projection pixel diameter of the Gaussian primitive, and the local occlusion rate of the Gaussian primitive are recalculated frame by frame to update the real-time visibility status of the masked invalid Gaussian primitives; that is: ; In the formula, Render valid markers for Gaussian meta-rendering. =1 indicates normal participation in rasterization and pixel fusion calculations; Temporarily remove rendering calculations for the current frame; Preset a distant view distance threshold; The minimum pixel projection threshold; This represents the maximum occlusion threshold.

[0057] In this embodiment, for Gaussian primitives determined to be recoverable, they are reconnected to the GPU parallel rendering pipeline and rendered according to their offline pre-divided basic, fine, and ultra-fine layer attributes.

[0058] S45: Executes three-level rendering and redundant high-level primitive scheduling through the GPU multi-stream parallel rendering pipeline, prioritizing the rendering of ultra-fine-level key areas, and then performing fine-level regular rendering and basic-level background rendering.

[0059] In this embodiment, step S5 involves performing a smooth transition processing on the rendering parameters of the target Gaussian pixel undergoing the level switch based on the original rendering parameters of the Gaussian pixels before and after the switch, including: S51: Iterate through all Gaussian elements that have completed the level determination frame by frame, compare the current frame level result with the previous frame rendering level, filter out the target Gaussian elements that have undergone level switching, and add inter-frame smooth transition markers to the target Gaussian elements.

[0060] S52: Set the appropriate total number of smooth frames based on the camera's real-time motion. For Gaussian elements entering the transition process, accumulate the number of transition frames frame by frame, starting with the level switching trigger frame. And calculate the gradient fusion weights based on the total number of smoothed frames and the number of transition frames, that is: ; In the formula, Let be the gradient fusion weights for frame t; To smooth out the total number of frames.

[0061] In this embodiment, when the camera is in a stationary observation and low-speed roaming state, the smoothed total frame rate is set to 10-15 frames; when the camera is in a high-speed scanning state, the smoothed total frame rate is set to 3-5 frames.

[0062] S53: Gradient-based weight fusion The rendering parameters of the Gaussian primitives entering the transition process are smoothed to obtain the final rendering parameters, namely: ; In the formula, The final rendering parameters for the Gaussian primitives in the transition process; To switch the original rendering parameters of the previous high-level primitive; These are the original rendering parameters of the Gaussian primitive after the switch.

[0063] The smooth transition ends when the number of transition frames equals the total number of smooth frames.

[0064] like Figure 4 As shown, another embodiment of this application provides a space-aware Gaussian splashing hierarchical autonomous switching system, wherein: The Gaussian hierarchy partitioning module 10 is used to perform spatial octree sparse partitioning and semantic clustering fusion on dense Gaussian elements in a 3D scene, constructing a hierarchy partitioning result that includes basic, fine, and ultra-fine levels, and generating spatial semantic masks for each level of Gaussian elements.

[0065] The perceptual feature extraction module 20 is used to collect the intrinsic and extrinsic parameters and motion parameters of the rendering camera in real time, so as to extract the spatial static features and temporal dynamic features of the current Gaussian elements, and to concatenate the spatial static features and temporal dynamic features into a perceptual feature matrix.

[0066] The rendering level division module 30 is used to calculate the real-time level comprehensive score of Gaussian elements based on the perceptual feature matrix, and to divide the Gaussian elements into rendering levels based on the real-time level comprehensive score.

[0067] The rendering execution module 40 is used to construct corresponding rendering pipelines for Goski elements at different rendering levels, filter invalid redundant Goski elements based on the spatial static features, temporarily mask the invalid redundant Goski elements, and update the visual status of the masked invalid redundant Goski elements in real time.

[0068] The rendering smooth transition module 50 is used to perform smooth transition processing on the rendering parameters of the target Gaussian pixel that has undergone a level switch, based on the original rendering parameters of the Gaussian pixel before and after the switch.

[0069] In this embodiment, the Gaussian pixel hierarchical partitioning module 10 performs spatial octree sparse partitioning on the dense Gaussian pixels of the 3D scene to obtain several sub-voxel units, including: By iterating through the center coordinates of all Gaussian elements within the 3D scene, the boundary of the scene space bounding box is determined; that is: ; In the formula, , These are the X-axis boundaries of the bounding box; , These are the Y-axis boundaries of the bounding box; , These are the Z-axis boundaries of the bounding box; For the first The center coordinates of each Gaussian element; This represents the total number of Gaussian elements.

[0070] Preset maximum recursion depth of octree Threshold for the maximum number of primitives that a single voxel can hold If the number of Gaussian units in the current voxel to be subdivided is greater than the threshold And the current recursion depth is less than the maximum recursion depth If the current voxel is divided into three equal parts along the axis, it will generate 8 sub-voxel units; otherwise, the division will stop and the current local space grouping will be completed.

[0071] For each Gaussian cell, its sub-voxel is determined based on its center coordinates. After the subdivision is completed, empty voxels with zero Gaussian cells are removed.

[0072] In this implementation, the Gaussian ellipsoid hierarchical partitioning module 10 constructs a covariance matrix and performs eigenvalue decomposition on the Gaussian ellipsoid spatial coordinates of the sub-voxel units, obtains the decomposed eigenvalues, solves for the scene geometric curvature of the sub-voxel units based on the decomposed eigenvalues, solves for the equivalent spatial scale of the Gaussian ellipsoids based on the Gaussian ellipsoid covariance matrix, and normalizes the scene geometric curvature and the equivalent spatial scale of the primitives; that is: ; ; In the formula, The scene geometry curvature of the current sub-voxel unit; This is the equivalent spatial scale of the current Gaussian unit; , , These are the three eigenvalues ​​of the covariance matrix of the primitive spatial coordinates within the sub-voxel unit, and ; , , These are the three principal axis scale parameters of the Gaussian ellipsoid.

[0073] If the scene geometry curvature of the current sub-voxel unit If the curvature is greater than or equal to the preset curvature threshold, the corresponding sub-voxel unit is identified as a high curvature detail region; otherwise, it is identified as a low curvature flat region.

[0074] In this implementation, the Gaussian unit hierarchical division module 10 assigns corresponding semantic weights to Gaussian units of different semantic categories based on the scene semantic segmentation results, and constructs hierarchical judgment features by combining the normalized scene geometric curvature and the equivalent spatial scale of the primitives, and divides the primitives in the sub-voxel unit into basic level, fine level and ultra-fine level according to the hierarchical judgment features. The hierarchical determination feature is represented as follows: ; In the formula, Features for stratification; , These are the scene geometric curvature and the equivalent spatial scale of the primitives, respectively, after normalization. For semantic weights; , , These are the corresponding weighting coefficients.

[0075] Based on the hierarchical determination characteristics, the Gaussian units within the sub-voxel units are divided into basic, fine, and super-fine levels, specifically: If the hierarchical determination features satisfy Then the current Gaussian unit will be classified into the basic level.

[0076] If the hierarchical determination features satisfy Then the current Gaussian unit will be divided into fine-grained levels.

[0077] If the hierarchical determination features satisfy Then the current Gaussian unit will be divided into a super-fine level.

[0078] in, and These are the threshold values ​​for the first level and the second level, respectively.

[0079] In this implementation, the Gaussian voxel hierarchical partitioning module 10 combines semantic weights and hierarchical partitioning results to generate a binary semantic mask matrix for each sub-voxel unit. The effective mask region corresponds to the key rendering region of the scene, and the invalid mask region corresponds to the lightweight rendering region; that is: ; In this embodiment, if the semantic weight of the current Gaussian unit... Greater than the semantic weight determination threshold And hierarchical determination features Greater than or equal to the first-level threshold If the result is positive, the mask of the current Gaussian element is set to 1; otherwise, it is set to 0.

[0080] In the formula, The threshold for semantic weight determination. For binary semantic masking, Indicates the key rendering areas of the scene; Corresponding to lightweight rendering areas.

[0081] In this embodiment, the perceptual feature extraction module 20 acquires the intrinsic and extrinsic parameters and motion parameters of the rendering camera frame by frame, and calculates the distance between the camera and the current Gaussian unit based on the three-dimensional coordinates of the camera's optical center and the coordinates of the current Gaussian unit's center. .

[0082] The motion parameters include the change in camera pose between consecutive frames and the frame interval time.

[0083] In this embodiment, the perceptual feature extraction module 20 is based on the distance between the camera and the Gaussian unit. The X and Y principal axis scale parameters of the Gaussian primitive map the three-dimensional spatial scale of the Gaussian primitive to the two-dimensional screen pixel space to obtain the screen projection pixel diameter of the current Gaussian primitive, thereby quantifying the visual contribution of the Gaussian primitive to the final rendered image; that is: ; In the formula, The screen projection pixel diameter of the current Gaussian pixel; To fix the scale constant for the camera's intrinsic parameters, These are the X and Y principal axis scale parameters of the Gaussian element.

[0084] In this embodiment, the perception feature extraction module 20 calculates the real-time rotational angular velocity of the camera by using the changes in camera pose between consecutive frames and the frame interval time, in order to quantify the speed of camera scanning and rotation. ; In the formula, For the camera's real-time angular velocity; The magnitude of the change in camera pose between consecutive frames; The frame interval between two adjacent frames.

[0085] In this embodiment, the perception feature extraction module 20 calculates the motion compensation coefficient based on the real-time rotational angular velocity of the camera, that is: ; In the formula, The motion compensation coefficient for the camera; This is the preset maximum effective angular velocity of the camera.

[0086] In this embodiment, the perceptual feature extraction module 20, based on scene depth caching and rasterization occlusion detection, calculates the ratio of visible pixels of a single Gaussian pixel to the total number of pixels to obtain the local occlusion rate, i.e.: ; In the formula, The local occlusion rate of the current Gaussian element; The number of pixels in the Gaussian pixel projection area that are occluded by the foreground. This represents the total number of pixels projected onto the Goshawk screen.

[0087] In this embodiment, the perceptual feature extraction module 20 measures the distance between the camera and the current Gaussian unit. The current screen projection pixel diameter P of the Gaussian pixel, and the current local occlusion rate of the Gaussian pixel. The semantic weight of the current Gaussian unit The splicing is used to obtain the current spatial static features of the Gaussian elements; the motion compensation coefficients are then used. As a time-dynamic feature, the spatial static feature and the time-dynamic feature are concatenated into a perceptual feature matrix.

[0088] In this embodiment, the rendering hierarchy division module 30 performs weighted fusion of the spatial static features, introduces camera motion compensation coefficients to complete dynamic correction, and calculates the real-time hierarchy comprehensive score for each individual Gaussian unit, that is: ; In the formula, This is a real-time, hierarchical comprehensive score; , , , These are the weight coefficients for the corresponding terms, and =1.

[0089] In this embodiment, the motion compensation coefficient is dynamically adjusted according to the real-time motion state of the camera. When the camera is stationary and observing statically, the motion compensation coefficient mentioned above... When the motion compensation coefficient approaches 1, it is reduced when the camera is roaming at low speed. When the camera is scanning at high speed, increase the motion compensation coefficient. .

[0090] In this embodiment, the rendering level division module 30 is based on the real-time level comprehensive score of Gaussian elements. The rendering hierarchy is divided into Gaussian elements; that is: If Gao Siyuan's real-time hierarchical comprehensive score If the value is greater than or equal to 0 and less than the first rendering threshold, then the current high-level primitive is rendered using the basic level.

[0091] If Gao Siyuan's real-time hierarchical comprehensive score If the value is greater than or equal to the first rendering threshold and less than the second rendering threshold, then the current high-level primitive is rendered using a fine-level rendering method.

[0092] If Gao Siyuan's real-time hierarchical comprehensive score If the value is greater than or equal to the second rendering threshold and less than or equal to 1, then the current high-level primitive is rendered using an ultra-fine rendering layer.

[0093] In this embodiment, the rendering execution module 40 constructs independent rendering pipelines for the basic rendering level, fine rendering level, and ultra-fine rendering level Gaussian primitives.

[0094] In this embodiment, for the Gaussian primitives of the basic rendering layer, distant background rendering is used, and subpixel subdivision sampling is turned off.

[0095] For the high-level rendering of the aforementioned high-level primitives, conventional scene rendering is adopted, and standard subpixel sampling and full covariance calculation are enabled.

[0096] For the high-resolution primitives of the ultra-fine layer rendering, close-up key detail rendering is adopted, and high-precision subpixel subdivision sampling, edge anti-aliasing optimization, and complete texture and color iterative calculation are enabled to maximize the preservation of scene object outlines, textures, and bump details.

[0097] In this embodiment, the rendering execution module 40 filters invalid redundant Gaussian primitives frame by frame, combining the distance between the camera and the current Gaussian primitive, the screen projection pixel diameter of the Gaussian primitive, and the local occlusion rate of the Gaussian primitive, including: In this embodiment, if the distance between the camera and the current high-level primitive is greater than a preset distant distance threshold, the current high-level primitive is determined to be an invalid redundant high-level primitive.

[0098] If the screen projection pixel diameter of the current high-order element is less than the minimum pixel projection threshold, then the current high-order element is determined to be an invalid redundant high-order element.

[0099] If the local occlusion rate of the current Gaussian element is greater than or equal to the maximum occlusion threshold If so, the current Gaussian element is determined to be an invalid redundant Gaussian element.

[0100] In some embodiments, the maximum occlusion threshold Set it to 0.95.

[0101] In this embodiment, the rendering execution module 40 temporarily masks the invalid redundant Gaussian pixels, removing them only from the rasterization calculation and pixel fusion process in the current rendering frame, while retaining the geometry, color, transparency, and covariance parameters.

[0102] In this embodiment, after completing the masking operation of redundant and invalid Gaussian primitives in each frame, the rendering execution module 40 establishes a redundant primitive index cache table. For all masked invalid and redundant Gaussian primitives in the cache table, it recalculates the distance between the camera and the current Gaussian primitive, the screen projection pixel diameter of the Gaussian primitive, and the local occlusion rate of the Gaussian primitive frame by frame to update the real-time visibility status of the masked invalid and redundant Gaussian primitives; that is: ; In the formula, Render valid markers for Gaussian meta-rendering. =1 indicates normal participation in rasterization and pixel fusion calculations; Temporarily remove rendering calculations for the current frame; Preset a distant view distance threshold; The minimum pixel projection threshold; This represents the maximum occlusion threshold.

[0103] In this embodiment, for Gaussian primitives determined to be recoverable, they are reconnected to the GPU parallel rendering pipeline and rendered according to their offline pre-divided basic, fine, and ultra-fine layer attributes.

[0104] In this embodiment, the rendering execution module 40 performs three-level rendering and redundant Gaussian scheduling through the GPU multi-stream parallel rendering pipeline, prioritizing the completion of ultra-fine-level key area rendering, and then performing fine-level regular rendering and basic-level background rendering.

[0105] In this embodiment, the rendering smooth transition module 50 traverses all high-level primitives that have completed the level determination frame by frame, compares the current frame level result with the previous frame rendering level, filters out the target high-level primitives that have undergone level switching, and adds inter-frame smooth transition markers to the target high-level primitives.

[0106] In this embodiment, the rendering smooth transition module 50 sets the corresponding total number of smooth frames according to the real-time motion of the camera. For Gaussian elements entering the transition process, the number of transition frames is accumulated frame by frame, starting from the level switching trigger frame. And calculate the gradient fusion weights based on the total number of smoothed frames and the number of transition frames, that is: ; In the formula, Let be the gradient fusion weights for frame t; To smooth out the total number of frames.

[0107] In this embodiment, when the camera is in a stationary observation and low-speed roaming state, the smoothed total frame rate is set to 10-15 frames; when the camera is in a high-speed scanning state, the smoothed total frame rate is set to 3-5 frames.

[0108] In this embodiment, the rendering smooth transition module 50 is based on gradient fusion weights. The rendering parameters of the Gaussian primitives entering the transition process are smoothed to obtain the final rendering parameters, namely: ; In the formula, The final rendering parameters for the Gaussian primitives in the transition process; To switch the original rendering parameters of the high-resolution primitive; These are the original rendering parameters of the Gaussian primitive after the switch.

[0109] The smooth transition ends when the number of transition frames equals the total number of smooth frames.

[0110] In a third embodiment of this application, an electronic device is provided, comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, it implements the space-aware Gaussian splashing level autonomous switching method described in any of the above embodiments.

[0111] In the fourth embodiment of this application, a computer storage medium is provided, which stores a computer program. When the computer program is executed by a processor, it implements the space-aware Gaussian splashing level autonomous switching method described in any of the above embodiments.

[0112] In this application, the memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. A processor, coupled to the memory, is used to execute computer programs stored in the memory.

[0113] Computer programs include computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. Computer-readable media can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, recording media, computer memory, and read-only memory (ROM). Only memory), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical discs.

[0114] The embodiments and descriptions above are merely illustrative of the principles and preferred embodiments of the present invention. Various changes and modifications may be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed.

Claims

1. A method for autonomous switching of Gaussian splashing levels based on spatial awareness, characterized in that, include: Spatial octree sparse partitioning and semantic clustering are performed on dense Gaussian elements in a 3D scene to construct a hierarchical partitioning result containing basic, fine, and ultra-fine levels, and spatial semantic masks are generated for Gaussian elements at each level. Real-time acquisition of intrinsic and extrinsic parameters and motion parameters of the rendering camera is used to extract the spatial static features and temporal dynamic features of the current Gaussian elements, and the spatial static features and temporal dynamic features are concatenated into a perception feature matrix; The real-time hierarchical comprehensive score of the Gaussian primitive is calculated based on the perceptual feature matrix, and the rendering hierarchy of the Gaussian primitive is divided based on the real-time hierarchical comprehensive score. Construct corresponding rendering pipelines for Goski primitives at different rendering levels, filter invalid and redundant Goski primitives based on the spatial static features, temporarily mask the invalid and redundant Goski primitives, and update the visual state of the masked invalid and redundant Goski primitives in real time. The rendering parameters of the target Gaussian pixel undergoing the level switch are smoothly transitioned based on the original rendering parameters of the Gaussian pixels before and after the switch.

2. The switching method according to claim 1, characterized in that, A spatial octree sparse partitioning and semantic clustering fusion is performed on dense Gaussian elements in a 3D scene to construct a hierarchical partitioning result containing basic, fine, and ultra-fine levels. Spatial semantic masks are generated for each level of Gaussian elements, including: The dense Gaussian elements of the 3D scene are sparsely partitioned into spatial octrees to obtain several sub-voxel units. Based on the normalized scene geometric curvature of the sub-voxel unit, the normalized primitive equivalent spatial scale of the current Gaussian unit, and the semantic weight of the current Gaussian unit, a hierarchical determination feature is constructed. According to the hierarchical determination feature, the Gaussian units in the sub-voxel unit are divided into basic level, fine level, and ultra-fine level. By combining semantic weights with the hierarchical division results of basic, fine, and ultra-fine levels, a binary semantic mask matrix is ​​generated for each sub-voxel unit.

3. The switching method according to claim 2, characterized in that, A covariance matrix is ​​constructed and eigenvalues ​​are decomposed for the Gaussian ellipsoidal coordinates of the sub-voxel units. The scene geometric curvature of the sub-voxel units is solved based on the decomposed eigenvalues, and the equivalent spatial scale of the Gaussian ellipsoidal units is solved based on the Gaussian ellipsoidal covariance matrix. The scene geometric curvature and the equivalent spatial scale of the primitives are then normalized to obtain the normalized scene geometric curvature and the equivalent spatial scale of the primitives.

4. The switching method according to claim 1, characterized in that, Real-time acquisition of camera intrinsic and extrinsic parameters and motion parameters is used to extract the spatial static and temporal dynamic features of the current Gaussian elements. These features are then concatenated into a perceptual feature matrix, including: The intrinsic and extrinsic parameters and motion parameters of the rendering camera are acquired frame by frame, and the distance between the camera and the current Gaussian element is calculated based on the three-dimensional coordinates of the camera optical center and the coordinates of the current Gaussian element center in the current frame. Based on the distance between the camera and the Gaussian element, and the Gaussian element's X and Y principal axis scale parameters, the three-dimensional spatial scale of the Gaussian element is mapped to the two-dimensional screen pixel space to obtain the screen projection pixel diameter of the current Gaussian element. The real-time rotational angular velocity of the camera is calculated by taking into account the changes in camera attitude between consecutive frames and the frame interval. Motion compensation coefficients are calculated based on the real-time rotational angular velocity of the camera. Based on scene depth caching and rasterized occlusion detection, the proportion of visible pixels of a single Gaussian pixel to the total number of pixels is calculated to obtain the local occlusion rate; The distance between the camera and the current Gaussian unit, the screen projection pixel diameter of the current Gaussian unit, the local occlusion rate of the current Gaussian unit, and the semantic weight of the current Gaussian unit are concatenated to form the spatial static feature of the current Gaussian unit; the motion compensation coefficient is used as the temporal dynamic feature, and the spatial static feature and the temporal dynamic feature are concatenated to form the perceptual feature matrix.

5. The switching method according to claim 4, characterized in that, The real-time hierarchical comprehensive score of Gaussian elements is calculated based on the perceptual feature matrix, and the rendering hierarchy of Gaussian elements is divided based on the real-time hierarchical comprehensive score, including: The spatial static features are weighted and fused, and a camera motion compensation coefficient is introduced to complete dynamic correction. The real-time hierarchical comprehensive score is calculated for each individual Gaussian element. Real-time hierarchical comprehensive score based on Gaussian elements The Gaussian primitives are divided into rendering levels, resulting in basic rendering, fine-level rendering, and ultra-fine-level rendering.

6. The switching method according to claim 5, characterized in that, A corresponding rendering pipeline is constructed for Gaussian primitives at different rendering levels. Invalid and redundant Gaussian primitives are filtered based on the spatial static features. These invalid and redundant Gaussian primitives are temporarily masked, and the visual state of the masked invalid and redundant Gaussian primitives is updated in real time, including: Independent rendering pipelines are constructed for the Gaussian primitives of the basic rendering level, fine rendering level, and ultra-fine rendering level; Invalid redundant Gaussian cells are filtered frame by frame by combining the distance between the camera and the current Gaussian cell, the screen projection pixel diameter of the Gaussian cell, and the local occlusion rate of the Gaussian cell. The invalid redundant Gaussian elements are temporarily masked. After completing the masking operation of redundant and invalid redundant Gaussian elements in each frame, the distance between the camera and the current Gaussian element, the screen projection pixel diameter of the Gaussian element, and the local occlusion rate of the Gaussian element are recalculated frame by frame for all masked invalid redundant Gaussian elements in order to update the real-time visibility status of the masked invalid redundant Gaussian elements. Three-level rendering and redundant Gaussian scheduling are performed through the GPU multi-stream parallel rendering pipeline.

7. The switching method according to claim 6, characterized in that, Invalid and redundant Gaussian cells are filtered frame-by-frame based on the distance between the camera and the current Gaussian cell, the screen projection pixel diameter of the Gaussian cell, and the local occlusion rate of the Gaussian cell. This includes: If the distance between the camera and the current high-order Gaussian element is greater than the preset distant view distance threshold, the current high-order Gaussian element is determined to be an invalid redundant high-order Gaussian element. If the screen projection pixel diameter of the current high-level element is less than the minimum pixel projection threshold, then the current high-level element is determined to be an invalid redundant high-level element. If the local occlusion rate of the current Gaussian cell is greater than or equal to the maximum occlusion threshold, then the current Gaussian cell is determined to be an invalid redundant Gaussian cell.

8. The switching method according to claim 6, characterized in that, The rendering parameters of the target Gaussian primitive undergoing the level switch are smoothly transitioned based on the original rendering parameters of the Gaussian primitives before and after the switch, including: Iterate through all the Gaussian elements that have completed the level determination frame by frame, compare the current frame level result with the previous frame rendering level, filter out the target Gaussian elements that have undergone level switching, and add inter-frame smooth transition markers to the target Gaussian elements. Based on the camera's real-time motion, a corresponding total number of smooth frames is set. For Gaussian elements entering the transition process, the number of transition frames is accumulated frame by frame, starting from the level switching trigger frame. And calculate gradient fusion weights based on the total number of smooth frames and the number of transition frames; The rendering parameters of the Gaussian primitives entering the transition process are smoothly processed based on gradient fusion weights to obtain the final rendering parameters.

9. A Gaussian splashing hierarchy autonomous switching system for spatial awareness, characterized in that, include: The Gaussian hierarchy partitioning module is used to perform spatial octree sparse partitioning and semantic clustering fusion on dense Gaussian elements in a 3D scene, constructing a hierarchy partitioning result that includes basic, fine, and ultra-fine levels, and generating spatial semantic masks for each level of Gaussian elements. The perceptual feature extraction module is used to collect the intrinsic and extrinsic parameters and motion parameters of the rendering camera in real time, so as to extract the spatial static features and temporal dynamic features of the current Gaussian unit, and to concatenate the spatial static features and temporal dynamic features into a perceptual feature matrix. The rendering level division module is used to calculate the real-time level comprehensive score of Gaussian elements based on the perceptual feature matrix, and to divide the rendering level of Gaussian elements based on the real-time level comprehensive score. The rendering execution module is used to construct corresponding rendering pipelines for Goski elements at different rendering levels, filter invalid and redundant Goski elements based on the spatial static features, temporarily mask the invalid and redundant Goski elements, and update the visual status of the masked invalid and redundant Goski elements in real time. The rendering smooth transition module is used to perform smooth transition processing on the rendering parameters of the target Gaussian pixel undergoing a level switch, based on the original rendering parameters of the Gaussian pixels before and after the switch.

10. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, and when the computer program is executed by the at least one processor, it implements the space-aware Gaussian splashing level autonomous switching method as described in any one of claims 1-8.

Images