A space intelligent panoramic gauss block streaming rendering method and system

Abstract

This invention relates to the field of spatial intelligence and real-time graphics rendering, providing a spatial intelligent panoramic Gaussian block-based streaming rendering method and system. On one hand, by structurally dividing the spatial scene into blocks and explicitly expressing it using Gaussian parameters, each rendering unit possesses clear spatial location, orientation information, and energy distribution attributes, improving the interpretability and traceability of the rendering process. On the other hand, through cross-block energy conservation constraints and geometric continuity correction mechanisms, it effectively eliminates problems such as inter-block brightness drift, boundary discontinuities, and topological discontinuities that are prone to occur in traditional parallel rendering. Simultaneously, this invention integrates rendering, stitching, and performance feedback mechanisms into a dynamic closed-loop system. During each frame rendering process, it collects the performance status of the execution entity and rendering quality indicators in real time, and adaptively adjusts parameters such as block granularity, sampling density, and illumination weight, thereby achieving millisecond-level streaming output and stable, continuous panoramic rendering effects.

Description

Technical Field

[0001] This invention belongs to the field of spatial intelligence and real-time graphics rendering, specifically relating to a spatial intelligent panoramic Gaussian block streaming rendering method and system. Background Technology

[0002] In recent years, with the development of computer vision and graphics technologies, implicit 3D scene representation and rendering methods, represented by Neural Radiance Fields (NeRF) and Gaussian Splatting, have gradually become important research directions in the fields of 3D reconstruction and real-time visualization. These methods jointly model scene geometry and appearance using implicit functions or parameterized Gaussian distributions, demonstrating significant advantages in high-quality reconstruction and viewpoint synthesis of complex scenes, and showing promising application prospects in virtual reality, digital twins, and immersive visualization.

[0003] However, existing 3D scene rendering methods based on Gaussian splashing still have significant limitations in real-time rendering of large-scale panoramic scenes. Current technologies typically employ a globally unified rendering architecture, requiring projection calculations and color mixing for all Gaussian elements in the scene during each frame rendering or update process. As the scene scale increases, the number of Gaussian elements often reaches millions or even higher. This global rendering approach leads to a sharp increase in computational complexity, placing high demands on GPU computing power and video memory resources, making it difficult to meet the frame rate and response latency requirements of real-time interactive scenes.

[0004] Meanwhile, existing Gaussian rendering systems generally lack intelligent spatial partitioning and scheduling mechanisms. Most methods rely on fixed resolution or regular spatial division methods for processing, failing to adaptively adjust based on changes in the user's viewpoint, spatial semantic importance, or scene dynamics, resulting in low rendering resource allocation efficiency. In practical applications, it is common to see redundant computational resources in some areas while rendering delays occur in critical viewpoint areas, affecting overall visual continuity and immersive experience.

[0005] Furthermore, existing technologies mostly employ a non-streaming rendering mode of "overall loading and whole-frame compositing," lacking a continuous streaming loading and stitching mechanism for large-scale scenes. In scenarios with large scene scale or multi-terminal collaborative rendering, the system struggles to achieve on-demand loading and progressive stitching of Gaussian data, easily leading to noticeable rendering delays, screen stuttering, or stitching gaps, thus limiting its application in panoramic and dynamic scenes.

[0006] When performing spatial segmentation of a scene, existing Gaussian rendering methods still face the problem of maintaining Gaussian parameter consistency. Since each sub-block is usually modeled and optimized independently, the Gaussian distribution parameters (including spatial variance, color distribution, and directional features) between blocks are difficult to maintain in a continuous and consistent manner in the boundary area. This can easily lead to visual artifacts such as color abrupt changes, depth discontinuities, or lighting inconsistencies at the splicing positions, further affecting the overall consistency of the rendering results.

[0007] On the other hand, most current Gaussian splash rendering systems rely solely on geometric information or point distribution features for calculations, without effectively incorporating spatial semantic information or task-driven mechanisms. They are unable to distinguish the importance of different regions in visual perception or application tasks, thus lacking intelligence and specificity in rendering accuracy control and computational resource scheduling. Summary of the Invention

[0008] This invention provides a spatial intelligent panoramic Gaussian block streaming rendering method and system to solve the problem that existing technologies cannot guarantee visual continuity and parameter consistency while taking into account real-time performance and system efficiency under complex and large-scale scene conditions. To address the aforementioned technical problems, the present invention discloses the following technical solutions: One aspect of the present invention provides a spatial intelligent panoramic Gaussian block streaming rendering method, comprising: Acquire multimodal spatial data of the 3D scene to be rendered, extract the spatial features and semantic information of the multimodal spatial data, and divide the 3D scene into multiple spatial blocks; The spatial features and semantic information of each spatial block are converted into Gaussian parameters to form the corresponding Gaussian parameter set. Multiple executors are used to perform parallel rendering on the Gaussian parameter set of all spatial blocks, generating local rendering results for each spatial block; After performing inter-block consistency constraint processing on the local rendering results, the results are stitched and blended to generate a continuous panoramic rendering result. Dynamic feedback analysis is performed based on the performance status of each executor during the rendering process and the rendering results, and the current rendering parameters are adaptively adjusted according to the feedback results.

[0009] Optionally, acquiring the multimodal spatial data of the 3D scene to be rendered includes: At least the geometric information, lighting information, and semantic data of the three-dimensional scene shall be collected as spatial data; The spatial data is preprocessed, and the preprocessing includes at least noise removal, geometric smoothing, and illumination normalization. After preprocessing, spatial data from different sources are time-synchronized and spatially aligned to generate multimodal spatial data that is consistent in time and space.

[0010] Optionally, the step of extracting the spatial features and semantic information of the multimodal spatial data and dividing the three-dimensional scene into multiple spatial blocks includes: Spatial feature analysis is performed on spatial data to obtain the geometric complexity of each spatial point; Based on the geometric complexity, the 3D scene is divided into multiple regions representing different structural complexities; A pre-trained spatial semantic network is used to determine the semantic category of each region, and the rendering priority corresponding to each region is calculated according to a preset algorithm. For each region, an adaptive partitioning strategy with different granularities is used to divide it into multiple spatial blocks based on the corresponding structural complexity and rendering priority.

[0011] Optionally, the step of converting the spatial features and semantic information of each sampling point in each spatial block into Gaussian parameters to form a corresponding Gaussian parameter set includes: For each spatial block, the Gaussian parameter set is constructed using the following method: Within the spatial blocks, adaptive sampling with different densities is performed according to preset rules to obtain the spatial features and semantic information of each sampling point; Based on the corresponding spatial features and semantic information, Gaussian parameters are calculated for each sampling point using a preset statistical regression model. The Gaussian parameters include at least spatial location, scale, direction, illumination intensity, and opacity. The Gaussian parameters of all sampling points are hierarchically encoded according to geometric information, illumination information and semantic information to form the Gaussian parameter set corresponding to the spatial block.

[0012] Optionally, the step of employing multiple executors to perform parallel rendering on the Gaussian parameter sets of all spatial blocks to generate local rendering results for each spatial block includes: Based on the corresponding structural complexity and rendering priority, calculate the task weight for each spatial block; Based on task weights, all spatial block rendering tasks are mapped to multiple executors for parallel execution, generating local rendering results for each spatial block.

[0013] Optionally, the step of performing inter-block consistency constraint processing on the local rendering results and then stitching and blending them to generate a continuous panoramic rendering result includes: Boundary difference detection is performed on the local rendering results of adjacent spatial blocks, and consistency correction is performed using a preset global illumination distribution model; The system performs streaming stitching on the local rendering results after consistency correction, weighted fusion processing on overlapping areas, and temporal continuity fusion between consecutive frames to output continuous panoramic rendering results.

[0014] Optionally, the step of performing inter-block consistency constraint processing on the local rendering results and then stitching and blending them to generate a continuous panoramic rendering result further includes: When the rendering time delay of any spatial block is detected to exceed a preset threshold, the local rendering result of the previous frame of the spatial block is obtained and used as the local rendering result of the current output frame to participate in the splicing and fusion.

[0015] Optionally, the dynamic feedback analysis based on the performance status of each executor during the rendering process and the rendering results, and the adaptive adjustment of the current rendering parameters based on the feedback results, includes: After each frame is rendered, the performance metrics and rendering quality metrics of the execution unit are collected as feedback data. Deviation detection is performed on the feedback data to identify abnormal performance degradation or rendering consistency issues in the execution unit. The current rendering parameters are adjusted based on the detection results. The rendering parameters include at least spatial block granularity, sampling density, or illumination weight allocation parameters. Verification rendering is performed using the adjusted rendering parameters, and the rendering parameters are saved when the rendering result meets a preset stability threshold for use in subsequent rendering processes.

[0016] Optionally, after performing the step of stitching and blending the local rendering results after performing inter-block consistency constraint processing to generate a continuous panoramic rendering result, the method further includes: Visual corrections are performed on the panoramic rendering results. These visual corrections include at least global brightness normalization, color space conversion, and local dynamic range compression.

[0017] Another aspect of the present invention provides a spatial intelligent panoramic Gaussian block streaming rendering system, comprising: The feature extraction and spatial segmentation module is configured to acquire multimodal spatial data of the 3D scene to be rendered, extract the spatial features and semantic information of the multimodal spatial data, and divide the 3D scene into multiple spatial segments. The Gaussian parameter set construction module is configured to convert the spatial features and semantic information of each spatial block into Gaussian parameters to form the corresponding Gaussian parameter set. The local rendering result generation module is configured to use multiple executors to perform parallel rendering on the Gaussian parameter set of all spatial blocks, generating local rendering results for each spatial block; The panoramic rendering result generation module is configured to perform inter-block consistency constraint processing on the local rendering results and then stitch and merge them to generate a continuous panoramic rendering result. The adaptive adjustment module is configured to perform dynamic feedback analysis based on the performance status of each executor and the rendering results during the rendering process, and to adaptively adjust the current rendering parameters according to the feedback results.

[0018] This invention discloses a spatial intelligent panoramic Gaussian block-based streaming rendering method and system. By constructing a structured execution system of "spatial block division, parallel rendering, consistency constraints, streaming stitching, and dynamic feedback," it achieves explicit expression and closed-loop control of the rendering process. On the one hand, by structurally dividing the spatial scene into blocks and using Gaussian parameters for explicit expression, each rendering unit has clear spatial location, orientation information, and energy distribution attributes, thereby improving the interpretability and traceability of the rendering process. On the other hand, through cross-block energy conservation constraints and geometric continuity correction mechanisms, it effectively eliminates the problems of inter-block brightness drift, boundary discontinuities, and topological discontinuities that are prone to occur in traditional parallel rendering, significantly enhancing semantic and geometric consistency, and ensuring the global coordination and physical rationality of the panoramic output at the lighting, depth, and structural levels.

[0019] Furthermore, this invention integrates rendering, stitching, and performance feedback mechanisms into a dynamic closed-loop system. During each frame's rendering process, it collects the performance status of the execution entity and rendering quality indicators in real time, and adaptively adjusts parameters such as block granularity, sampling density, and illumination weights, thereby achieving millisecond-level streaming output and stable, continuous panoramic rendering effects. In complex or dynamic scenes, the system can automatically improve robustness through task rescheduling and parameter optimization mechanisms, and supports the fusion processing of multimodal input data, achieving high-precision rendering and generalization adaptability across scenes.

[0020] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify essential or necessary features of this disclosure, nor is it intended to limit the scope of this disclosure. Attached Figure Description

[0021] The above and other objects, features and advantages of this disclosure will become more apparent from the accompanying drawings, in which like reference numerals generally denote like parts.

[0022] Figure 1 A schematic flowchart of a spatial intelligent panoramic Gaussian block streaming rendering method provided in an embodiment of the present invention; Figure 2 An implementation provided by an embodiment of the present invention Figure 1A flowchart illustrating step S100; Figure 3 Another implementation provided by the embodiments of the present invention Figure 1 A flowchart illustrating step S100; Figure 4 An implementation provided by an embodiment of the present invention Figure 1 A flowchart illustrating step S200; Figure 5 An implementation provided by an embodiment of the present invention Figure 1 A flowchart illustrating step S300; Figure 6 An implementation provided by an embodiment of the present invention Figure 1 A flowchart illustrating step S400; Figure 7 An implementation provided by an embodiment of the present invention Figure 1 A flowchart of step S500; Figure 8 This is a schematic diagram of the structure of a spatial intelligent panoramic Gaussian block streaming rendering system provided in an embodiment of the present invention. Detailed Implementation

[0023] Embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0024] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0025] Figure 1 This is a flowchart illustrating a spatial intelligent panoramic Gaussian block streaming rendering method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, it includes the following steps: Step S100: Obtain multimodal spatial data of the 3D scene to be rendered, extract the spatial features and semantic information of the multimodal spatial data, and divide the 3D scene into multiple spatial blocks.

[0026] In one embodiment of the present invention, such as Figure 2 As shown, obtaining multimodal spatial data of the 3D scene to be rendered includes the following steps: Step S111: Collect at least the geometric information, lighting information and semantic data of the 3D scene as spatial data.

[0027] Multimodal spatial data of a 3D scene is acquired. This multimodal spatial data includes at least geometric information, lighting information, and semantic data. Geometric information describes the spatial structure and morphological features of objects in the scene and may include point cloud data, mesh model data, or depth map data. Lighting information characterizes the distribution of incident light energy, reflection intensity, and direction features in the scene and may include illumination intensity maps, ambient light maps, or direction vector fields. Semantic data identifies the categories, attributes, and dynamic states of different objects in the scene and can be obtained through semantic segmentation models or labeling systems. Through these multi-source data acquisition methods, a multidimensional dataset containing geometric, lighting, and semantic fields is formed.

[0028] Step S112: Preprocess the spatial data.

[0029] The preprocessing process includes at least noise removal, geometric smoothing, and illumination normalization. Specifically, outliers or measurement errors in the geometric data are removed using statistical filtering or neighborhood density analysis. For areas with discontinuous local geometric surfaces or sampling fluctuations, surface fitting or normal smoothing algorithms are used for geometric smoothing to eliminate high-frequency jitter. For illumination data, global brightness statistical analysis and energy normalization algorithms are used to standardize the illumination intensity, ensuring that illumination data obtained from different acquisition devices or time periods maintain consistent energy scales.

[0030] Step S113: After preprocessing, perform time synchronization and spatial alignment processing on spatial data from different sources to generate multimodal spatial data that is consistent in time and space.

[0031] In the temporal dimension, frame-level synchronization of different modal data is performed based on the acquisition timestamps. Interpolation or frame alignment mechanisms are used to eliminate temporal deviations, ensuring that geometric changes, illumination changes, and semantic updates correspond consistently under the same time reference. In the spatial dimension, data generated by different sensors or data sources are uniformly mapped to the same world coordinate system through a coordinate transformation matrix, and scale correction and rotation alignment are performed to eliminate perspective differences and spatial offsets. After the above dual temporal and spatial alignment processing, a unified multimodal spatial data set is generated, which simultaneously satisfies the requirements of temporal and spatial consistency.

[0032] In one embodiment of this invention, spatial features and semantic information of multimodal spatial data are extracted, and the three-dimensional scene is divided into multiple spatial blocks, such as... Figure 3 As shown, it includes the following steps: Step S121: Perform spatial feature analysis on the spatial data to obtain the geometric complexity of each spatial point.

[0033] A comprehensive complexity evaluation model is constructed based on multi-dimensional geometric features such as point cloud density distribution, rate of curvature change, normal vector gradient, depth difference, and neighborhood structural entropy. Regions with drastic curvature changes, significant normal vector fluctuations, or highly uneven point density are classified as having high geometric complexity; regions with smooth structures, continuous shapes, and small gradient changes are classified as having low complexity. These geometric complexities can be mapped to standardized complexity coefficients using a weighted function to characterize the computational demand on the structural representation of spatial points.

[0034] Step S122: Divide the 3D scene into multiple regions representing different structural complexities based on geometric complexity.

[0035] The 3D scene is initially divided into regions based on geometric complexity to form several spatial regions representing different levels of structural complexity. Specific implementation methods can include threshold-based hierarchical clustering, density-adaptive region growing algorithms, or hierarchical segmentation strategies based on spatial tree structures (such as octrees). According to the distribution of complexity coefficients, the scene is divided into high-complexity, medium-complexity, and low-complexity regions, ensuring a relatively consistent structural change trend within each region. This partitioning process not only improves the targeting of subsequent block divisions but also provides a structural reference for calculating task weights during parallel rendering, thus avoiding the uneven load problem caused by simple uniform partitioning.

[0036] Step S123: Use a pre-trained spatial semantic network to determine the semantic category of each region, and calculate the rendering priority of each region according to a preset algorithm.

[0037] Spatial semantic networks can be models trained based on 3D convolutional neural networks or Transformer architectures. Their input includes geometric features, lighting distribution characteristics, and historical semantic labels within a region. The output is the semantic category of the corresponding region, such as a static background, dynamic object, highly reflective material region, or highly light-sensitive region. After obtaining the semantic category, a preset rendering strategy model is used to calculate the rendering priority for each region. The rendering priority comprehensively considers factors such as semantic importance, visibility probability, dynamic change frequency, and user perspective relevance, quantifying the processing order and resource allocation level of regions during the rendering process in numerical form.

[0038] Step S124: For each region, based on the corresponding structural complexity and rendering priority, an adaptive partitioning strategy with different granularities is used to divide it into multiple spatial blocks.

[0039] For high-complexity and high-priority regions, a fine-grained partitioning strategy is adopted to reduce the computational load of a single block and improve the accuracy of local detail representation. For low-complexity or low-priority regions, a coarser-grained partitioning strategy is adopted to reduce the number of blocks and lower scheduling overhead. The adaptive partitioning process can be implemented based on a dynamic threshold control mechanism or a complexity-priority mapping function to ensure that the partitioning scale is consistent with the weights of subsequent parallel rendering tasks. Through this step, several sets of spatial blocks that are relatively balanced in terms of structural complexity and rendering importance can be generated.

[0040] Step S200: Convert the spatial features and semantic information of each spatial block into Gaussian parameters to form the corresponding Gaussian parameter set.

[0041] In one embodiment of the present invention, such as Figure 4 As shown, for each spatial block, the Gaussian parameter set is constructed in the following way: Step S201: Perform adaptive sampling at different densities within the spatial blocks according to preset rules to obtain the spatial features and semantic information of each sampling point.

[0042] The sampling density is determined based on the geometric complexity distribution, normal rate of change, depth gradient, and illumination intensity variations within each block. In edge regions, areas with abrupt curvature changes, high reflectivity regions, or shadow transition regions, the sampling density is increased to enhance the depiction of detailed structure and energy variations; in flat or gently changing regions, the sampling density is decreased to reduce redundant data. Each sampling point records its corresponding spatial coordinates, local normal vector, color distribution, brightness value, and semantic category, forming a high-dimensional feature vector containing both geometric features and semantic labels.

[0043] Step S202: Based on the corresponding spatial features and semantic information, calculate the Gaussian parameters for each sampling point using a preset statistical regression model.

[0044] Based on the spatial features and semantic information corresponding to each sampling point, a pre-defined statistical regression model is used to calculate its Gaussian parameters. The statistical regression model can be a parameter mapping model based on least squares estimation, probability density fitting, or neural network regression. Its input is the feature vector of the sampling point, and its output is the corresponding Gaussian kernel parameters. The Gaussian parameters include at least the spatial location (as the center of the Gaussian), scale parameters (characterizing the spatial extent), orientation vector (describing the anisotropic distribution characteristics), illumination (characterizing energy intensity), and opacity (used for subsequent depth synthesis and occlusion calculations). The scale and orientation parameters together form a covariance matrix to characterize the morphological features of the Gaussian distribution in 3D space; illumination and opacity are used to describe the contribution intensity of the Gaussian point to the local light field. For sampling points with similar parameters and spatial proximity, parameter clustering and merging operations can be further performed to reduce redundant Gaussian representations and improve storage efficiency and rendering performance.

[0045] Step S203: Encode the Gaussian parameters of all sampling points in layers according to geometric information, illumination information and semantic information to form a set of Gaussian parameters corresponding to the spatial blocks.

[0046] The Gaussian parameters are divided into at least three information layers: a geometric information layer, a lighting information layer, and a semantic information layer. The geometric information layer stores the Gaussian center location, scale, and orientation parameters; the lighting information layer stores brightness, energy weights, and lighting-related directional parameters; and the semantic information layer identifies the object category, dynamic state, or block number of each sampling point. Through this hierarchical encoding structure, different types of data can be accessed and stored in a layered manner with differential updates, supporting cross-block referencing, local reconstruction, and fast rendering scheduling. Finally, each spatial block forms a structured set of Gaussian parameters, stored in matrix or tensor form, providing a unified input for Gaussian projection and lighting integral calculations in the subsequent parallel rendering stage.

[0047] Step S300: Use multiple executors to perform parallel rendering on the Gaussian parameter set of all spatial blocks to generate local rendering results for each spatial block.

[0048] Multiple executors perform parallel rendering on the Gaussian parameter sets of all spatial blocks to generate local rendering results for each spatial block. Executors can be multiple graphics processing units (GPUs), multiple central processing cores (CPUs), heterogeneous computing units, or other processing modules that support parallel computing. Through block-level task decomposition and parallel execution mechanisms, parallel computing in the spatial dimension and pipelined processing in the temporal dimension are achieved, thereby significantly improving overall computational efficiency while ensuring rendering accuracy.

[0049] In one embodiment of the present invention, such as Figure 5As shown, this step can be achieved in the following ways: Step S301: Calculate the task weight of each spatial block based on the corresponding structural complexity and rendering priority.

[0050] Structural complexity can be determined by a combination of factors, including the number of Gaussian parameters within a block, the distribution of geometric curvature, the degree of change in lighting gradients, and the proportion of dynamic objects. Rendering priority can be calculated based on semantic category, visibility weight, user view range, or real-time requirements. These factors are normalized using a preset weighting function to generate a comprehensive task weight value for each block. This weight value characterizes the computational resource intensity and scheduling priority required by that block within the current rendering cycle, thus providing a quantitative basis for subsequent task mapping and resource allocation.

[0051] Step S302: Map all spatial block rendering tasks to multiple executors for parallel execution according to task weights, and generate local rendering results for each spatial block.

[0052] First, a task queue is established, and the blocks are sorted or grouped according to their weights. Then, the scheduler dynamically allocates the block tasks to different executors based on their real-time load status, computing power, and historical processing time. Each executor builds an independent rendering pipeline locally, performing operations such as projection calculation, light integration, depth filtering, and pixel aggregation on the Gaussian parameter set of its corresponding block, generating the corresponding local rendered image, depth map, or related intermediate results. During parallel execution, the task allocation ratio can be dynamically adjusted based on real-time performance feedback to achieve load balancing and optimized resource utilization. Finally, each executor outputs the local rendering results for its assigned block, providing input data for subsequent inter-block consistency correction and streaming stitching / fusion steps.

[0053] Step S400: After performing inter-block consistency constraint processing on the local rendering results, the results are stitched and blended to generate a continuous panoramic rendering result.

[0054] The local rendering results output by each execution unit are subjected to inter-block consistency constraints, and then stitched and blended after consistency correction to generate a continuous panoramic rendering result. Since each spatial block is computed in parallel on an independent execution unit in the preceding steps, its illumination integration, depth synthesis, and color output are all based on local parameters. Therefore, problems such as brightness deviation, color jumps, or depth discontinuities may exist at block boundaries. To ensure that the final panoramic image has no obvious stitching artifacts visually and satisfies the requirements of energy conservation and illumination consistency in a physical sense, this step uses mechanisms such as boundary detection, global illumination correction, spatial stitching, and temporal fusion to achieve unified constraints and continuous output of block-level results.

[0055] In one embodiment of the present invention, such as Figure 6 As shown, this step can be achieved in the following ways: Step S401: Perform boundary difference detection on the local rendering results of adjacent spatial blocks, and perform consistency correction using a preset global illumination distribution model.

[0056] The system extracts overlapping boundary regions between adjacent blocks and compares and analyzes their color distribution, brightness gradient, illumination direction vector, and depth values. By calculating the pixel difference, energy offset, and normal difference of the boundary regions, it determines whether there are abrupt changes or discontinuities. When a difference exceeds a preset threshold, the corresponding boundary region is marked as a region to be corrected. Subsequently, a preset global illumination distribution model is invoked to perform consistency correction on this region. The global illumination distribution model can be established based on panoramic ambient light parameters or global energy statistics to unify the illumination benchmark of each block. During the correction process, the brightness values, energy weights, and illumination directions at the boundaries are regressively adjusted to make adjacent blocks more consistent in energy distribution and color representation. Simultaneously, for regions with depth or normal discontinuities, the system can perform local interpolation and smoothing to eliminate geometric abrupt changes.

[0057] Step S402: Stream the local rendering results after consistency correction, perform weighted fusion processing on overlapping areas, and perform temporal continuity fusion between consecutive frames to output continuous panoramic rendering results.

[0058] A streaming stitching cache structure is established in the main control unit, and the rendering results of each block are entered into the corresponding cache area according to spatial coordinates and timestamps. The stitching module aligns and combines each block image according to its spatial position based on the preset block index relationship and spatial positioning information. When there are overlapping areas, a fusion coefficient is calculated based on distance weight, brightness difference weight, or energy distribution weight, and weighted mixing processing is performed on the pixels in the overlapping area to make the transition area present a gradual change effect, thereby avoiding obvious seams. At the same time, a temporal continuity fusion mechanism is established between consecutive frames to monitor the brightness change rate and energy change rate of adjacent frames. When abrupt changes or inter-frame fluctuations are detected and exceed the threshold, static areas are stabilized by multi-frame averaging or exponential smoothing, and dynamic areas are stabilized by adaptive weighted fusion to maintain details and clarity. Through the synergistic effect of spatial stitching and temporal fusion, a panoramic rendering result that is continuous in the spatial dimension, smooth in the temporal dimension, and meets the lighting consistency constraint is finally generated.

[0059] In one embodiment of the present invention, when the rendering time delay of any spatial block is detected to exceed a preset threshold, the local rendering result of the previous frame of the spatial block is obtained and used as the local rendering result of the current output frame to participate in splicing and fusion.

[0060] When the rendering time delay of any spatial block exceeds a preset threshold, the local rendering result of the previous frame of the spatial block is obtained and used as the local rendering result of the current output frame for stitching and fusion. This embodiment is mainly used to solve the problem of individual block rendering lag caused by fluctuations in computing load, resource contention, or sudden increases in dynamic scene complexity during multi-executor parallel rendering, thereby avoiding overall stuttering or frame-level incompleteness of the panoramic frame due to local delay.

[0061] Specifically, the execution status of all spatial blocks is monitored in real time during each rendering cycle. Monitoring metrics include at least the block task start time, current execution progress, estimated completion time, and current frame timestamp. If the rendering time of a spatial block exceeds a preset latency threshold (e.g., exceeding a preset proportion of the target frame cycle or exceeding the maximum allowed frame latency), it is determined that the block cannot complete rendering output on time within the current frame. In this case, the system triggers a latency backfill mechanism.

[0062] In the delayed backfill mechanism, the system reads the completed local rendering result of the previous frame for the spatial block from the cache module. The local rendering result of the previous frame has usually passed the consistency correction and stitching preparation process, so it maintains overall consistency with the current frame in terms of spatial structure and lighting reference. The system temporarily uses the result of the previous frame as the output data of the corresponding block of the current frame and marks it as "backfill data". It is then entered into the stitching and blending module along with the results of other blocks that have been rendered.

[0063] To avoid noticeable visual lag or flickering in the backfilled data, a time-weighted adjustment mechanism can be introduced during the backfilling process. For example, during the stitching and blending stage, lightweight brightness smoothing or boundary softening can be applied to the backfilled area to ensure a natural transition with the output results of other blocks in the current frame. Simultaneously, when the delayed block completes new rendering calculations in subsequent frames, the system automatically replaces the previous backfilled data with the latest results and resumes the normal parallel rendering process.

[0064] Through the above mechanism, the integrity and continuity of the panoramic output frames can be ensured without blocking the overall rendering process. Even if short-term performance fluctuations occur in individual blocks, a stable frame rate output can still be maintained, avoiding screen tearing, flickering, or overall stuttering, thereby significantly improving the robustness and real-time performance of the large-scale panoramic Gaussian block rendering system.

[0065] In one embodiment of the present invention, visual correction is also required for the panoramic rendering result. The visual correction includes at least global brightness normalization, color space conversion, and local dynamic range compression.

[0066] After completing inter-block consistency processing and streaming stitching, the resulting panoramic rendering still requires further visual correction to improve the visual consistency and viewing quality of the output image on the actual display. Because the block-based parallel rendering and cross-frame fusion processes may result in overall brightness fluctuations, color shifts, or excessively large local dynamic ranges, it is necessary to perform unified visual hierarchy optimization on the final panoramic frames. Visual correction includes at least global brightness normalization, color space conversion, and local dynamic range compression.

[0067] First, global brightness normalization is performed on the panoramic rendering results. By statistically analyzing the brightness distribution characteristics of the entire image, such as average brightness, brightness histogram, and brightness deviation, the difference between the current frame and the target brightness benchmark is calculated, and a corresponding brightness correction factor is generated. Subsequently, a unified brightness mapping is performed on the panoramic image to make the overall brightness distribution tend to a stable range, thereby eliminating possible residual overall brightness differences between different blocks and reducing cross-frame brightness drift.

[0068] After brightness normalization, the image undergoes color space conversion. Since front-end rendering is typically performed in a physical linear color space or high dynamic range space, while terminal display devices have specific display color gamuts and gamma characteristics, the rendering results need to be mapped to the target display color space. This process includes gamma correction, color gamut mapping, and necessary white balance adjustments to ensure the output image maintains natural colors and clear detail across different lighting environments and display devices.

[0069] Furthermore, to avoid highlight clipping or loss of detail in shadows, local dynamic range compression is performed on the panoramic rendering results. By analyzing the brightness gradient and contrast distribution in local areas of the image, highlight areas are compressed and mapped, while shadow areas are moderately enhanced, improving detail retention while maintaining overall brightness stability. This dynamic range compression process also incorporates temporal continuity constraints to ensure smooth transitions in brightness between consecutive frames, preventing screen flickering or abrupt changes.

[0070] Through the above visual correction processing, the final output panoramic image frame sequence achieves a unified and balanced state in terms of brightness, color and contrast, and can be directly used in immersive display, virtual reality or visual simulation systems, thereby further improving the visual stability and engineering practicality of the present invention in real-time rendering applications of large-scale spatial scenes.

[0071] Step S500: Perform dynamic feedback analysis based on the performance status of each executor and the rendering results during the rendering process, and adaptively adjust the current rendering parameters according to the feedback results.

[0072] Because this embodiment of the invention employs a multi-executor parallel rendering architecture, system performance and visual consistency may dynamically fluctuate under different scene complexities, viewpoint changes, and hardware load conditions. Therefore, this step continuously optimizes the current rendering strategy through real-time performance monitoring and rendering quality analysis, thereby ensuring the stability and robustness of the system during long-term operation.

[0073] In one embodiment of the present invention, such as Figure 7 As shown, this step can be achieved in the following ways: Step S501: After each frame is rendered, collect the performance metrics and rendering quality metrics of the executor as feedback data.

[0074] After each frame is rendered, the performance metrics of each executor and the rendering quality metrics of the current frame are automatically collected as feedback data input. Performance metrics include at least the executor's computational utilization, memory usage, average rendering time per block, frame-level latency, and task queue length; rendering quality metrics include at least inter-block brightness deviation, energy conservation error, boundary color difference, and cross-frame brightness change rate. This data is recorded in real-time through a unified performance monitoring interface and stored in a feedback cache table, providing a structured data foundation for subsequent analysis.

[0075] Step S502: Perform deviation detection on the feedback data to identify abnormal performance degradation or rendering consistency of the executor.

[0076] Establish statistical benchmark models for performance and quality metrics, such as calculating historical mean and variance based on a sliding time window. An abnormal state is identified when the current frame's metrics exceed a preset fluctuation range or stability threshold. An abnormal increase in execution utilization or a continuous rise in frame latency is identified as an abnormal performance degradation; excessive inter-block brightness deviation or excessive cross-frame illumination variation rate is identified as an abnormal rendering consistency. Simultaneously, multi-metric correlation analysis is used to pinpoint the source of the anomaly, such as excessively fine block granularity, insufficient sampling density, or unbalanced illumination weight allocation.

[0077] Step S503: Adjust the current rendering parameters based on the detection results. The rendering parameters include at least spatial block granularity, sampling density, or illumination weight allocation parameters.

[0078] For anomaly detection results, the current rendering parameters are adaptively adjusted. If the issue is determined to be a performance bottleneck, the computational load can be reduced by increasing the spatial block granularity or decreasing the local sampling density. If the issue is determined to be insufficient local detail or brightness shift, the sampling density in the corresponding area is increased or the lighting weight parameters are reallocated. If the issue is determined to be an inter-block energy imbalance, the lighting weight allocation strategy or boundary blending coefficient is adjusted. These parameter adjustments can be performed based on a preset rule model, or they can be combined with reinforcement learning or regression models to perform self-learning optimization based on historical feedback data, thereby gradually approaching the optimal rendering configuration.

[0079] Step S504: Perform verification rendering using the adjusted rendering parameters, and save the rendering parameters when the rendering result meets the preset stability threshold for subsequent rendering processes.

[0080] Verification rendering is performed using the adjusted rendering parameters. This involves regenerating local or panoramic rendering results in the next frame or a specified test frame using the new parameter configuration, and comparing the performance curves and quality metrics before and after optimization. When the verification results meet preset stability thresholds—for example, frame rate fluctuations are less than a set percentage, energy errors are below a preset upper limit, and boundary color differences are within acceptable limits—the system records the current rendering parameter configuration as a valid parameter template and stores it in the parameter database. This template can be preferentially invoked in subsequent similar scenes or under similar load conditions, thereby shortening the system convergence time and improving overall adaptive efficiency.

[0081] Figure 8 This is a schematic diagram of the structure of a spatial intelligent panoramic Gaussian block streaming rendering system disclosed in this invention, as shown below. Figure 8 As shown, it includes the following modules: The feature extraction and spatial segmentation module 1 is configured to acquire multimodal spatial data of the 3D scene to be rendered, extract spatial features and semantic information of the multimodal spatial data, and divide the 3D scene into multiple spatial segments.

[0082] Module 2, which consists of Gaussian parameter sets, is configured to convert the spatial features and semantic information of each spatial block into Gaussian parameters to form the corresponding Gaussian parameter sets.

[0083] The local rendering result generation module 3 is configured to use multiple executors to perform parallel rendering on the Gaussian parameter set of all spatial blocks, generating local rendering results for each spatial block.

[0084] The panoramic rendering result generation module 4 is configured to perform inter-block consistency constraint processing on the local rendering results and then stitch and merge them to generate a continuous panoramic rendering result.

[0085] The adaptive adjustment module 5 is configured to perform dynamic feedback analysis based on the performance status of each executor and the rendering results during the rendering process, and to adaptively adjust the current rendering parameters according to the feedback results.

[0086] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A spatial intelligent panoramic Gaussian block streaming rendering method, characterized in that, include: Acquire multimodal spatial data of the 3D scene to be rendered, extract the spatial features and semantic information of the multimodal spatial data, and divide the 3D scene into multiple spatial blocks; The spatial features and semantic information of each spatial block are converted into Gaussian parameters to form the corresponding Gaussian parameter set. Multiple executors are used to perform parallel rendering on the Gaussian parameter set of all spatial blocks, generating local rendering results for each spatial block; After performing inter-block consistency constraint processing on the local rendering results, the results are stitched and blended to generate a continuous panoramic rendering result. Dynamic feedback analysis is performed based on the performance status of each executor during the rendering process and the rendering results, and the current rendering parameters are adaptively adjusted according to the feedback results.

2. The method according to claim 1, characterized in that, The acquisition of multimodal spatial data of the 3D scene to be rendered includes: At least the geometric information, lighting information, and semantic data of the three-dimensional scene shall be collected as spatial data; The spatial data is preprocessed, and the preprocessing includes at least noise removal, geometric smoothing, and illumination normalization. After preprocessing, spatial data from different sources are time-synchronized and spatially aligned to generate multimodal spatial data that is consistent in time and space.

3. The method according to claim 1, characterized in that, The step involves extracting spatial features and semantic information from the multimodal spatial data and dividing the 3D scene into multiple spatial blocks, including: Spatial feature analysis is performed on spatial data to obtain the geometric complexity of each spatial point; Based on the geometric complexity, the 3D scene is divided into multiple regions representing different structural complexities; A pre-trained spatial semantic network is used to determine the semantic category of each region, and the rendering priority corresponding to each region is calculated according to a preset algorithm. For each region, an adaptive partitioning strategy with different granularities is used to divide it into multiple spatial blocks based on the corresponding structural complexity and rendering priority.

4. The method according to claim 1, characterized in that, The process of converting the spatial features and semantic information of each sampling point in each spatial block into Gaussian parameters to form a corresponding Gaussian parameter set includes: For each spatial block, the Gaussian parameter set is constructed using the following method: Within the spatial blocks, adaptive sampling with different densities is performed according to preset rules to obtain the spatial features and semantic information of each sampling point; Based on the corresponding spatial features and semantic information, Gaussian parameters are calculated for each sampling point using a preset statistical regression model. The Gaussian parameters include at least spatial location, scale, direction, illumination intensity, and opacity. The Gaussian parameters of all sampling points are hierarchically encoded according to geometric information, illumination information and semantic information to form the Gaussian parameter set corresponding to the spatial block.

5. The method according to claim 3, characterized in that, The method employs multiple executors to perform parallel rendering on the Gaussian parameter sets of all spatial blocks, generating local rendering results for each spatial block, including: Based on the corresponding structural complexity and rendering priority, calculate the task weight for each spatial block; Based on task weights, all spatial block rendering tasks are mapped to multiple executors for parallel execution, generating local rendering results for each spatial block.

6. The method according to claim 1, characterized in that, The step of performing inter-block consistency constraint processing on the local rendering results and then stitching and blending them to generate a continuous panoramic rendering result includes: Boundary difference detection is performed on the local rendering results of adjacent spatial blocks, and consistency correction is performed using a preset global illumination distribution model; The system performs streaming stitching on the local rendering results after consistency correction, weighted fusion processing on overlapping areas, and temporal continuity fusion between consecutive frames to output continuous panoramic rendering results.

7. The method according to claim 6, characterized in that, The step of performing inter-block consistency constraint processing on the local rendering results and then stitching and blending them to generate a continuous panoramic rendering result further includes: When the rendering time delay of any spatial block is detected to exceed a preset threshold, the local rendering result of the previous frame of the spatial block is obtained and used as the local rendering result of the current output frame to participate in the splicing and fusion.

8. The method according to claim 1, characterized in that, The dynamic feedback analysis based on the performance status of each executor during the rendering process and the rendering results, and the adaptive adjustment of the current rendering parameters based on the feedback results, includes: After each frame is rendered, the performance metrics and rendering quality metrics of the execution unit are collected as feedback data. Deviation detection is performed on the feedback data to identify abnormal performance degradation or rendering consistency issues in the execution unit. The current rendering parameters are adjusted based on the detection results. The rendering parameters include at least spatial block granularity, sampling density, or illumination weight allocation parameters. Verification rendering is performed using the adjusted rendering parameters, and the rendering parameters are saved when the rendering result meets a preset stability threshold for use in subsequent rendering processes.

9. The method according to claim 1, characterized in that, After performing the step of stitching and blending the local rendering results after applying inter-block consistency constraints to generate a continuous panoramic rendering result, the method further includes: Visual corrections are performed on the panoramic rendering results. These visual corrections include at least global brightness normalization, color space conversion, and local dynamic range compression.

10. A spatial intelligent panoramic Gaussian block streaming rendering system, characterized in that, include: The feature extraction and spatial segmentation module is configured to acquire multimodal spatial data of the 3D scene to be rendered, extract the spatial features and semantic information of the multimodal spatial data, and divide the 3D scene into multiple spatial segments. The Gaussian parameter set construction module is configured to convert the spatial features and semantic information of each spatial block into Gaussian parameters to form the corresponding Gaussian parameter set. The local rendering result generation module is configured to use multiple executors to perform parallel rendering on the Gaussian parameter set of all spatial blocks, generating local rendering results for each spatial block; The panoramic rendering result generation module is configured to perform inter-block consistency constraint processing on the local rendering results and then stitch and merge them to generate a continuous panoramic rendering result. The adaptive adjustment module is configured to perform dynamic feedback analysis based on the performance status of each executor and the rendering results during the rendering process, and to adaptively adjust the current rendering parameters according to the feedback results.

Images