Screen projection error and node time closed loop scheduling method and device and storage device

Abstract

The present application relates to the field of real-time rendering, especially to a screen projection error and node time closed-loop scheduling method and device and a storage device, a hierarchical Gaussian pyramid of a target scene is constructed to obtain a plurality of hierarchical nodes, a screen projection error estimation value of a current frame is calculated, and a current rendering work set of the current frame is obtained, a budget deviation based on a rendering time consumption measured value and a target time budget is used to feedback adjust a screen projection error threshold value of an adjacent frame to obtain an adjacent rendering work set of the adjacent frame, adjacent rendering nodes are determined based on rendering differences, the current rendering work set is processed in blocks, a processing strategy of a to-be-processed block is generated, and a viewpoint adaptive continuous detail level management is realized, so that distant and near scenes are smoothly transitioned without hierarchical jumps, rendering tasks are balanced, and the frame rate and memory utilization efficiency of large-scale scene real-time rendering are significantly improved.

Description

Technical Field

[0001] This invention relates to the field of real-time rendering, and more particularly to a method, apparatus, and storage device for screen projection error and node time closed-loop scheduling. Background Technology

[0002] Existing real-time rendering methods for large-scale scenes based on 3D Gaussian splashing typically treat all Gaussian primitives as equally important rendering units, directly rasterizing and sorting all Gaussian points in each frame. However, for ultra-large-scale scenes such as city-level buildings or natural landscapes, the number of Gaussian points can reach billions, far exceeding the real-time processing capabilities of current graphics hardware, leading to a sharp drop in frame rate and memory overflow, making it difficult to support interactive roaming. Existing detail level selection strategies mostly rely on spatial distance or fixed level switching, without adaptively determining based on the screen projection error of the current viewpoint. This easily results in blurring due to undersampling at close distances or wide-angle views, while redundant primitives affect rendering efficiency at long distances. At the same time, existing scene tree traversal mechanisms lack closed-loop judgment, easily leading to branch divergence and repeated visits, with scheduling overhead increasing significantly with scene size. Furthermore, when rendering screen blocks in parallel, the load differences between blocks are large, and static task allocation makes it difficult to balance computing resources, resulting in insufficient hardware utilization.

[0003] Therefore, it is necessary to propose a closed-loop scheduling method, apparatus, and storage device for screen projection error and node time to solve at least one of the above-mentioned technical problems. Summary of the Invention

[0004] To address these issues, this invention provides a screen projection error and node time closed-loop scheduling method, apparatus, and storage device to overcome problems in existing technologies, such as frame rate drops and memory overflows caused by full primitive processing during large-scale Gaussian scene rendering, undersampling blur and long-distance redundancy caused by lack of screen projection error adaptive judgment in detail level selection, branch divergence and repeated access overhead caused by lack of closed-loop scheduling in scene tree traversal, and uneven load and insufficient hardware utilization caused by static allocation in screen block parallel rendering.

[0005] In a first aspect, this disclosure provides a closed-loop scheduling method for screen projection errors and node time, including: Offline clustering and merging of all Gaussian parameters of the target scene is performed to construct a hierarchical Gaussian pyramid, resulting in multiple hierarchical nodes. Each hierarchical node carries an upper bound of approximate error and a spatial range. The upper bound of approximate error is used to characterize the maximum error introduced when rendering the hierarchical node. Obtain the current frame and the screen projection error threshold of the current frame, traverse the hierarchical Gaussian pyramid according to the upper bound of the approximate error and the spatial range to obtain the screen projection error estimate of the current frame, and calculate the current rendering working set of the current frame. Obtain the measured rendering time of the current rendering work set, and calculate the budget deviation between the measured rendering time and the target time budget; Based on the budget deviation, the screen projection error threshold of the adjacent frames of the current frame is adjusted to obtain the adjacent rendering working set of the adjacent frames; Calculate the rendering difference between the current rendering work set and the adjacent rendering work sets, and determine the adjacent rendering nodes based on the rendering difference; The current rendering work set is divided into blocks to generate at least one screen block. The screen block containing the adjacent rendering node is determined as the block to be processed, and a processing strategy for the block to be processed is generated.

[0006] Further, the step of obtaining the current frame and the screen projection error threshold of the current frame, traversing the hierarchical Gaussian pyramid based on the approximate error upper bound and the spatial range to obtain the estimated screen projection error value of the current frame, and calculating the current rendering working set of the current frame includes: Starting from the bottom level node of the hierarchical Gaussian pyramid, for each level node traversed, in response to the intersection of the spatial range of the level node with the view frustum range of the current frame, the screen projection error estimate corresponding to the level node is calculated based on the upper bound of the approximation error of the level node and the screen projection scale of the level node in the current frame. For the level node, the following comparison operation is performed: the screen projection error estimate of the level node is compared with the screen projection error threshold of the current frame. If the screen projection error estimate is not greater than the screen projection error threshold of the current frame, the level node is added to the current rendering working set, and traversal of each child node of the level node is stopped. If the screen projection error estimate is greater than the screen projection error threshold of the current frame, the threshold comparison is performed on each child node of the level node until the screen projection error estimate of the child node is not greater than the screen projection error threshold of the current frame or the traversal has reached the top level node of the level Gaussian pyramid.

[0007] Further, obtaining the measured rendering time of the current rendering working set and calculating the budget deviation between the measured rendering time and the target time budget includes: The start timestamp and end timestamp of the current frame are obtained respectively, and the difference between the start timestamp and the end timestamp is determined as the measured value of the rendering time. Calculate the difference between the measured rendering time and the target time budget, and use the difference as the budget deviation.

[0008] Further, the step of adjusting the screen projection error threshold of the adjacent frames of the current frame based on the budget deviation to obtain the adjacent rendering working set of the adjacent frames includes: The screen projection error threshold of the adjacent frames is calculated based on the budget deviation, and the threshold adjustment amount is proportional to the absolute value of the budget deviation; Based on the adjacent frames and their screen projection error thresholds, the traversal of the hierarchical Gaussian pyramid is performed to bring the frame time of the adjacent frames to the target time budget, thereby generating the adjacent rendering working set of the adjacent frames.

[0009] Further, the step of calculating the rendering difference between the current rendering working set and the adjacent rendering working sets, and determining adjacent rendering nodes based on the rendering difference, includes: The hierarchical node set of the current rendering work set and the hierarchical node set of the adjacent rendering work set are compared by hierarchical switching to determine the adjacent rendering nodes; The adjacent rendering nodes are subjected to a rendering gradient transition process.

[0010] Further, the step of performing block processing on the current rendering working set to generate at least one screen block, determining the screen block containing the adjacent rendering node as the block to be processed, and generating a processing strategy for the block to be processed includes: The screen space of the current frame is divided into at least one screen block of a preset size; For each adjacent rendering node, the projection coverage area of ​​the adjacent rendering node in the screen space is determined based on the spatial range of the Gaussian pyramid in the hierarchy. Screen blocks that overlap with the projection coverage area are identified as the blocks to be processed; Generate a processing strategy for each of the blocks to be processed, the processing strategy including: performing pixel rendering on the blocks to be processed, and performing smooth transition processing on the adjacent rendering nodes involved in the pixel rendering.

[0011] Secondly, this disclosure provides a screen projection error and node time closed-loop scheduling device, including: The construction module is configured to perform offline clustering and merging of all Gaussian parameters of the target scene to construct a hierarchical Gaussian pyramid, resulting in multiple hierarchical nodes. Each hierarchical node carries an upper bound of the approximate error and a spatial range. The upper bound of the approximate error is used to characterize the maximum error introduced when rendering the hierarchical node. The traversal module is configured to obtain the current frame and the screen projection error threshold of the current frame, traverse the hierarchical Gaussian pyramid according to the upper bound of the approximate error and the spatial range, obtain the screen projection error estimate of the current frame, and calculate the current rendering work set of the current frame. The deviation calculation module is configured to obtain the measured rendering time of the current rendering work set and calculate the budget deviation between the measured rendering time and the target time budget. The adjustment module is configured to adjust the screen projection error threshold of the adjacent frames of the current frame based on the budget deviation, so as to obtain the adjacent rendering working set of the adjacent frames; The node determination module is configured to calculate the rendering difference between the current rendering work set and the adjacent rendering work sets, and determine the adjacent rendering nodes based on the rendering difference; The processing module is configured to perform block processing on the current rendering work set, generate at least one screen block, determine the screen block containing the adjacent rendering node as the block to be processed, and generate the processing strategy for the block to be processed.

[0012] Thirdly, this disclosure provides a storage device, including: One or more processors; Storage device, on which one or more programs are stored, When the above-described one or more programs are executed by the above-described one or more processors, the above-described one or more processors implement the method as described in any embodiment of the first aspect of this disclosure.

[0013] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any embodiment of the first aspect of this disclosure.

[0014] Fifthly, this disclosure provides a computer program product including a computer program / instructions that, when executed by a processor, implement the method described in any embodiment of the first aspect of this disclosure.

[0015] Compared with existing technologies, the beneficial effects of this invention are as follows: by constructing a hierarchical Gaussian pyramid of the target scene to obtain multiple hierarchical nodes, calculating the screen projection error estimate of the current frame, and obtaining the current rendering work set of the current frame, the screen projection error threshold of adjacent frames is adjusted based on the budget deviation between the actual rendering time and the target time budget to obtain adjacent rendering work sets of adjacent frames, adjacent rendering nodes are determined based on rendering differences, the current rendering work set is divided into blocks, and a processing strategy for the blocks to be processed is generated. This achieves continuous detail hierarchy management with viewpoint adaptation, resulting in smooth transitions between near and far objects without hierarchical jumps. At the same time, the rendering tasks are balanced, significantly improving the frame rate and memory utilization efficiency of real-time rendering of large-scale scenes. Attached Figure Description

[0016] Figure 1 This is a flowchart of an embodiment of the screen projection error and node time closed-loop scheduling method disclosed herein; Figure 2 This is an exploded flowchart of one embodiment of step 102 of this disclosure; Figure 3 This is an exploded flowchart of one embodiment of step 104 of this disclosure; Figure 4 This is an exploded flowchart of one embodiment of step 106 of this disclosure; Figure 5 This is a schematic diagram of an embodiment of the screen projection error and node time closed-loop scheduling device according to the present disclosure. Detailed Implementation

[0017] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0018] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0019] It should be noted that in the description of this invention, the terms "upper", "lower", "left", "right", "inner", "outer", etc., which indicate directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings. This is only for the convenience of description and is not intended to indicate or imply that the device or element must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation of this invention.

[0020] Furthermore, it should be noted that, in the description of this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0021] Please refer to Figure 1 , Figure 1 A flowchart 100 is shown as an embodiment of the screen projection error and node time closed-loop scheduling method according to this disclosure. Figure 1 The screen projection error and node time closed-loop scheduling method shown can be applied to Figure 1 The terminal device or server shown includes the following steps: Step 101: Perform offline clustering and merging of all Gaussian parameters of the target scene to construct a hierarchical Gaussian pyramid and obtain multiple hierarchical nodes.

[0022] In this embodiment, the execution entity of the screen projection error and node time closed-loop scheduling method can first obtain the full Gaussian parameters of the target scene. The full Gaussian parameters refer to the set of all Gaussian primitive parameters in the complete Gaussian sputtering model generated to represent the 3D scene, including the position, covariance matrix, opacity, and spherical harmonic coefficients of each Gaussian primitive, representing the original highest-precision representation of the scene. Offline clustering and merging refers to the preprocessing procedure performed before the real-time rendering pipeline starts. As an example, offline clustering and merging recursively clusters multiple similar Gaussian primitives with minimal visual impact into a single surrogate node by performing spatial nearest neighbor search and cross-scale error measurement (such as merging cost evaluation based on mutual information or geometric distance) on the full Gaussian parameters.

[0023] A hierarchical Gaussian pyramid is a multi-resolution hierarchical structure composed of multiple hierarchical nodes arranged in a tree-like manner. Each hierarchical node carries an upper bound on the approximate error and a spatial range. The upper bound on the approximate error characterizes the maximum numerical error introduced when rendering by replacing its original subset with that hierarchical node. The spatial range refers to the bounding box of the primitive represented by the hierarchical node in three-dimensional space, providing a scene proxy with different levels of detail that can be quickly queried during rendering. A hierarchical node refers to the scene proxy data structure at a certain level in the hierarchical Gaussian pyramid. It is not the original discrete Gaussian primitive, but a logical rendering proxy unit generated by offline clustering and merging of all Gaussian parameters.

[0024] By pre-compiling and merging massive discrete Gaussian primitives into hierarchical proxy nodes carrying upper bounds of error during the offline stage, the transformation of 3D scenes from a flat primitive set to a multi-resolution hierarchical structure is realized. This provides a basic data structure for real-time rendering that can adaptively select the level of detail based on viewpoint and performance constraints, and solves the problem of excessive traversal overhead caused by directly manipulating all primitives.

[0025] Step 102: Obtain the current frame and the screen projection error threshold of the current frame. Based on the upper bound of the approximate error and the spatial range, traverse the hierarchical Gaussian pyramid to obtain the estimated value of the screen projection error of the current frame, and calculate the current rendering working set of the current frame.

[0026] The screen projection error threshold is a numerical constraint set by the user to balance rendering quality and performance. It specifies the maximum tolerable projection error of all rendered nodes in the current frame in screen space. The size of the screen projection error threshold is dynamically given by the upper-level scheduling strategy.

[0027] The screen projection error estimate is an evaluation value calculated in real time for each traversed level node in a hierarchical Gaussian pyramid.

[0028] As an example, by projecting the upper bound of the approximate error of the hierarchical node onto the screen space of the current frame and scaling it according to its distance from the camera and the screen resolution, a screen pixel-related error prediction value is obtained, which is used to quantify the visibility of geometric or appearance defects that may occur when the hierarchical node is rendered at the current viewpoint.

[0029] The current rendering working set refers to the set of hierarchical nodes that are finally determined for the actual drawing of the current frame after traversal and error filtering. Each of these hierarchical nodes satisfies the condition that its estimated screen projection error value is not greater than the screen projection error threshold of the current frame.

[0030] By establishing a mapping relationship from 3D node error to 2D screen error, and performing depth-first threshold pruning traversal on the hierarchical structure accordingly, dynamic and automated selection of rendering precision for different areas of the scene is achieved. This ensures that the overall error level of each frame is controllable while eliminating a large number of high-order details that contribute little to the visual experience, thus greatly reducing the number of computational primitives to be rendered per frame.

[0031] In some alternative implementations, step 102 may include, for example: Figure 2 The following steps 1021 to 1022 are shown: Step 1021: Starting from the bottom level node of the hierarchical Gaussian pyramid, for each traversed level node, in response to the intersection of the spatial range of the level node and the view frustum range of the current frame, calculate the estimated screen projection error value corresponding to the level node based on the upper bound of the approximate error of the level node and the screen projection scale of the level node in the current frame.

[0032] Here, the view frustum of the current frame refers to the visible space region determined by the virtual camera pose and projection parameters corresponding to the current rendering frame.

[0033] Screen projection scale is a pixel-based quantification used to estimate the visual significance of geometric or appearance errors introduced in the final screen image when rendering uses the current-level node in place of its child nodes.

[0034] By forcing a visibility check at the beginning of the traversal and combining the screen projection scale of the hierarchical nodes to calculate the screen error directly related to human visual perception, the importance of hierarchical nodes is accurately measured, avoiding invalid computational overhead in invisible areas and ensuring that all hierarchical nodes entering the subsequent comparison stage are visible and comparable in importance.

[0035] Step 1022: Perform a comparison operation for the hierarchical nodes.

[0036] Here, the comparison operation includes the following steps one and two: Step 1: Compare the estimated screen projection error of the hierarchical node with the screen projection error threshold of the current frame. If the estimated screen projection error is not greater than the screen projection error threshold of the current frame, add the hierarchical node to the current rendering work set and stop traversing the child nodes of the hierarchical node.

[0037] Step 2: If the estimated screen projection error is greater than the screen projection error threshold of the current frame, perform threshold comparison on each child node of the level node until the estimated screen projection error of the child node is not greater than the screen projection error threshold of the current frame or the top level node of the level Gaussian pyramid has been reached.

[0038] Here, adding a level node to the current rendering workset means storing the identifier of the level node and the Gaussian proxy parameters it contains into the rendering object queue. The current rendering workset will then serve as the direct data source for subsequent drawing calls.

[0039] By utilizing the monotonicity of hierarchical error to design a single-pass threshold pruning strategy, adaptive termination of rendering granularity is achieved. Under the premise of meeting screen error requirements, the coarsest available node is automatically selected, thereby minimizing the rendering load.

[0040] Step 103: Obtain the actual rendering time of the current rendering workset and calculate the budget deviation between the actual rendering time and the target time budget.

[0041] Here, the actual rendering time refers to the real time consumed to execute one complete set of drawing instructions based on the current rendering work set determined in step 102, and is used as a real-time feedback of system load.

[0042] The target time budget refers to the maximum time limit that all processing tasks (rendering, calculation, etc.) allocated to a single frame can occupy in order to maintain a set target frame rate (such as 60 FPS).

[0043] Budget deviation refers to the difference between the target time budget and the actual rendering time. The sign and absolute value of this difference indicate the degree of performance surplus or performance bottleneck in the current frame, respectively.

[0044] By introducing a negative feedback loop based on actual time consumption measurements, the abstract rendering quality target is quantitatively correlated with the actual execution capability of the hardware, realizing the transformation from open-loop quality selection to closed-loop performance constraints, and providing a precise and measurable basis for subsequent dynamic quality adjustments.

[0045] In some alternative implementations, step 103 may include steps A and B: Step A: Obtain the start timestamp and end timestamp of the current frame, and determine the difference between the start timestamp and end timestamp as the measured value of rendering time.

[0046] Here, a timestamp refers to a synchronization marker point obtained through a high-precision timer, used to define the start and end times of a frame rendering task.

[0047] By employing a timestamp-based difference measurement method, the rendering time of a single frame is accurately quantified, avoiding measurement errors caused by asynchronous execution or drive overhead, thus providing accurate system state observations for closed-loop control.

[0048] Step B: Calculate the difference between the actual rendering time and the target time budget, and use the difference as the budget deviation.

[0049] Step 104: Adjust the screen projection error threshold of the adjacent frames of the current frame based on the budget deviation to obtain the adjacent rendering working set of the adjacent frames.

[0050] Here, feedback adjustment refers to a closed-loop control mechanism that uses inter-frame performance deviation signals to correct the quality parameters of the next frame. As an example, the budget deviation can be input into a preset controller (such as a PID controller) to adjust the screen projection error threshold for the next frame.

[0051] The screen projection error threshold for adjacent frames refers to the new error threshold that is prepared to be applied to adjacent frames after the budget deviation of the current frame has been corrected.

[0052] By obtaining adjacent rendering working sets of adjacent frames, a dynamic self-balancing mechanism is achieved among scene complexity, rendering strategy, and hardware performance, thereby ensuring stable frame rate and completely solving the problem of sudden changes in frame latency and stuttering experience caused by changes in scene or viewpoint.

[0053] In some alternative implementations, step 104 may include, for example: Figure 3 The following steps 1041 and 1042 are shown: Step 1041: Calculate the screen projection error threshold of adjacent frames based on the budget deviation. The threshold adjustment amount is proportional to the absolute value of the budget deviation.

[0054] Here, the threshold adjustment amount refers to the magnitude of the change in the error threshold.

[0055] By employing linear proportional adjustment, a direct match is achieved between the adjustment range and the degree of performance deviation. When frame latency deviates significantly from the budget, it can be quickly pulled back, and when the deviation is minor, it can be finely adjusted, thus achieving a balance between response speed and system stability.

[0056] Step 1042: Based on adjacent frames and their screen projection error thresholds, perform a traversal of the hierarchical Gaussian pyramid to bring the frame time of adjacent frames to the target time budget, and generate adjacent rendering working sets of adjacent frames.

[0057] Frame time convergence refers to the dynamic process in which the actual measured rendering time of each frame hovers around the target time budget and tends to stabilize after closed-loop feedback adjustment for multiple consecutive frame cycles.

[0058] By traversing the hierarchical Gaussian pyramid based on adjacent frames and their screen projection error thresholds, the frame time of adjacent frames converges to the target time budget, generating adjacent rendering work sets of adjacent frames. This enables the immediate effect of control commands and the real-time iteration of rendering load, thus forming a continuous, self-correcting runtime scheduling loop that drives the system to continuously approach and maintain the preset performance target.

[0059] Step 105: Calculate the rendering difference between the current rendering work set and the adjacent rendering work sets, and determine the adjacent rendering nodes based on the rendering difference.

[0060] Here, rendering difference refers to the set of differences between the two rendering worksets of the current frame and the adjacent frame on the constituent nodes, specifically represented by newly added nodes, removed nodes, and nodes that have undergone level switching.

[0061] Adjacent rendering nodes refer to those specific nodes that undergo hierarchical changes or replacements between two frames. Their state transitions are the fundamental reason for the sudden changes in detail in the current frame.

[0062] By accurately identifying incremental changes in the rendered content between frames, precise positioning of elements that may cause visual jumps is achieved, thus paving the way for subsequent smooth transition processing only on locally changed areas and avoiding the additional overhead of global operations.

[0063] In some alternative implementations, step 105 may include steps a and b: Step a: Perform a level switching comparison between the set of hierarchical nodes of the current rendering workset and the set of hierarchical nodes of the adjacent rendering worksets to determine the adjacent rendering nodes.

[0064] Here, level switching comparison refers to a set operation. As an example, unique identifiers can be established for nodes in two working sets, and the pyramid level of a node at the same spatial location can be compared to see if it changes in different frames. If the levels are different, the node is marked as an adjacent rendering node.

[0065] By directly comparing sparse hierarchical node sets, a fast and low-complexity difference detection process is achieved. Its computational overhead is far less than that of pixel-by-pixel comparison frame buffers, thus providing accurate input for temporal anti-mutation algorithms efficiently.

[0066] Step b: Perform a rendering gradient transition on adjacent rendering nodes.

[0067] Here, rendering gradient transition processing refers to a temporal artifact elimination technique. As an example, interpolation operations can be performed on the Gaussian parameters (such as opacity or covariance scale) of adjacent rendering nodes between adjacent frames, so that the parameter values ​​of the old state smoothly transition to the parameter values ​​of the new state within a few frames.

[0068] By applying short-term interpolation of opacity or geometry to nodes where level switching occurs, continuous changes in the level of detail in the temporal domain are achieved, greatly enhancing the interactive viewing experience at the streaming media level.

[0069] Step 106: Perform block processing on the current rendering work set to generate at least one screen block, determine the screen block containing adjacent rendering nodes as the block to be processed, and generate the processing strategy for the block to be processed.

[0070] Here, screen partitioning refers to dividing the entire screen space evenly into several non-overlapping rectangular regions, with each partition independently organizing its covered list of rendering primitives. This is an optimization for modern GPU architectures.

[0071] Plots to be processed refer to all screen blocks in screen space that overlap with the projected areas of adjacent rendering nodes. These plots to be processed are marked individually and require customized processing strategies to achieve a smooth transition.

[0072] By transforming the full-screen processing problem into targeted processing for a few screen segments, a fine-grained on-demand scheduling mechanism is implemented, which greatly restricts the execution scope of the high-overhead operation of smooth transition, thereby eliminating artifacts while minimizing the impact on overall rendering performance.

[0073] In some alternative implementations, step 106 may include, for example: Figure 4 The following steps 1061 to 1064 are shown: Step 1061: Divide the screen space of the current frame into at least one screen block of a preset size.

[0074] Here, the preset size refers to the screen block size preset based on GPU hardware characteristics (such as shared memory size and rasterization throughput), which is usually 16×16 or 32×32 pixels.

[0075] By using conventional uniform block operations, basic parallel units that are compatible with GPU hardware processing are provided for subsequent rendering operations, which helps to increase computational locality and cache utilization.

[0076] Step 1062: For each adjacent rendering node, determine the projection coverage area of ​​the adjacent rendering node in the screen space based on the spatial range of the Gaussian pyramid of the hierarchy.

[0077] Here, the projection coverage area refers to the smallest bounding rectangle area formed after the eight corner points of the 3D node bounding box are projected from perspective onto the 2D screen.

[0078] By determining the projection coverage area of ​​the adjacent rendering node in screen space, the screen influence of the node is quickly and safely defined, ensuring that no screen area affected by the smooth transition is missed, thus guaranteeing the integrity of the visual effect.

[0079] Step 1063: Screen blocks that overlap with the projection coverage area are identified as blocks to be processed.

[0080] By using simple two-dimensional geometric intersection determination, it is possible to accurately assign fine-grained smooth transition tasks to a small number of uniquely related computing units, thereby avoiding full-screen traversal and achieving precise allocation of computing resources.

[0081] Step 1064: Generate a processing strategy for each block to be processed. The processing strategy includes: performing pixel rendering on the block to be processed and performing smooth transition processing on adjacent rendering nodes involved in the pixel rendering.

[0082] Here, the processing strategy refers to a set of rendering instruction flows customized for the chunk to be processed. As an example, in the pixel shader, for fragments within the chunk to be processed that are contributed by adjacent rendering nodes, when calculating the final color, the current transition interpolation coefficient of the node is dynamically queried, and this coefficient is applied to the node's Gaussian parameters (such as affecting opacity) before lighting calculation.

[0083] By generating processing strategies for each block to be processed, a pixel-level deep fusion of smooth transition effects and standard rendering processes is achieved, thereby efficiently and consistently achieving smooth image processing within blocks without increasing the number of additional rendering pipelines.

[0084] As an embodiment of this disclosure, let e(v) be the upper bound of the approximation error of node v in the hierarchical Gaussian pyramid, and let B(v) be the bounding box representing its spatial range. The current viewpoint is o. The screen projection error estimate Ψ(v,o) of node v is jointly determined by the projection scale of e(v) and B(v) relative to viewpoint o. The projection scale decreases as the viewpoint distance increases. Therefore, the screen projection error estimate of the same node when viewed from a distance is lower than that when viewed up close, which is consistent with intuition: the approximation error at a distance is not easily perceived in screen space. The hierarchical detail node selection rule is as follows: starting from the root node, if Ψ(v,o) is lower than the current error threshold τ, then v is selected; otherwise, its child nodes are recursively examined. The time complexity of this traversal process is related to the size of the final selected working set and the depth of the pyramid, which is much smaller than the overhead of performing a step-by-step judgment on the entire Gaussian pyramid. In the frame-time closed-loop control section, let the actual rendering time of frame t be T_t and the target frame budget be T_target. The error threshold is updated according to the proportional feedback rule: τ_{t+1} equals τ_t multiplied by (1 plus η multiplied by (T_t minus T_target) divided by T_target), where η is a preset adjustment gain coefficient. This update rule increases τ_{t+1} when T_t exceeds T_target (selecting a coarser working set to reduce the time of the next frame) and decreases τ_{t+1} when T_t is lower than T_target. Upper and lower limits are imposed on the single adjustment amplitude to prevent oscillation. The transparency gradient of the layer transition can be described as follows: Let the switching start time be t_0 and the transition duration be Δt. The transparency weight of the node before the switch at time t is max(0, 1 minus (t minus t_0) divided by Δt), and the transparency weight of the node after the switch is min(1, (t minus t_0) divided by Δt). Both participate in alpha synthesis according to their respective weights until the transition is completed. In the chunk sorting overhead reduction section, let γ be the change ratio of the Gaussian set within a chunk compared to the previous frame. When γ is lower than the preset threshold τ_reuse, a reuse plus incremental rearrangement strategy is adopted. Its computational complexity is approximately proportional to the size of the changed part rather than the total number of Gaussians. When γ exceeds the threshold, a complete sort is performed. For parts determined to have low visual importance, a bucket approximate sort is adopted. The number of buckets is determined based on the proportion of the contribution of this part to the transparency of the final composite result. The lower the contribution ratio, the coarser the buckets can be.The pseudocode is described as follows: In the offline stage, bottom-up clustering and merging are performed on the full Gaussian data to construct a hierarchical Gaussian pyramid; during runtime, each frame is traversed from top to bottom on the pyramid based on the current error threshold τ to obtain the rendering working set; after rendering, the actual frame time is measured, and τ is updated according to the proportional feedback rules; a hierarchical transition is performed on the difference between the current frame and the previous frame working set, and the viewpoint of the next frame is predicted based on camera motion, and the expected working set data is pre-fetched; the change ratio of the current frame working set is judged by blocks, small change blocks are reused or incrementally rearranged, important parts in large change blocks are completely sorted, and minor parts are approximately sorted; after multiple rendering sessions, the frame time and switching frequency distribution are statistically analyzed, and the control gain, pyramid hierarchical granularity, and device parameter configuration are optimized accordingly.

[0085] Specifically, taking a Gaussian splash scene of a city block-level building complex loaded on a city information modeling platform as the object, with a total of approximately 45 million Gaussian elements in the scene, and performing interactive browsing in a web-based rendering environment on a mid-range integrated graphics laptop, with the target frame budget set to approximately 16.7 milliseconds per second corresponding to 60 frames, the specific implementation process of the method of this invention is explained.

[0086] The first step: In the offline stage, bottom-up clustering and merging are performed on 45 million Gaussians to construct a six-level Gaussian pyramid. The root node level contains about 12,000 Gaussians, covering the entire scene range. The leaf node level is the original full-precision Gaussians. The upper bound of the approximation error of each intermediate level decreases gradually as the level decreases.

[0087] The second step: When the user browses to an aerial view above the city blocks, the rendering workset selected by traversing the hierarchy pyramid is about 1.8 million Gauss, mainly composed of merged nodes of the intermediate level; when the user's viewpoint descends to observe the main entrance of a building up close, the workset of the corresponding area of ​​the building switches to a fine level of leaf nodes, while other distant buildings in the field of view still maintain a coarser level, and the total size of the frame workset is about 6.4 million Gauss.

[0088] The third step: The initial error threshold τ is set to an empirical value corresponding to medium precision, and the gain coefficient η is adjusted to 0.15. When the user performs a rapid, large-range view rotation, causing a sudden increase in the instantaneous working set size and a single-frame rendering time of 24 milliseconds (exceeding the target budget by about 40%), the closed-loop control increases τ in the following two frames, and the working set size decreases accordingly. The rendering time of the third frame drops back to 17.2 milliseconds, and the fifth frame converges to around 16.5 milliseconds, restoring the frame rate to the target level.

[0089] Step 4: When the user moves the viewpoint forward along the main road of the building complex, a 0.2-second transparency transition rendering is performed on the building facade area along the way when switching levels. After the transition processing, no obvious flickering of the level jump was observed. At the same time, the viewpoint is predicted by extrapolating the pose trajectory of the last eight frames, and data prefetching is initiated for the expected newly added fine nodes one frame in advance. The actual test showed that this mechanism reduced the additional time consumption at the moment of level switching from about 18% when prefetching was not used to about 6%.

[0090] Step 5: For the rendering workset, the percentage change of adjacent frames is calculated by dividing the screen into blocks. When the viewpoint moves slowly, about 76% of the blocks have a percentage change of less than the reuse threshold τ_reuse (set to 0.1). The sorting result of the previous frame is directly reused and incremental reordering is performed. In actual tests, the time taken for the sorting stage in this scenario is reduced by about 44% compared to full sorting frame by frame. For blocks where the viewpoint changes quickly or the level jumps, a full sorting is performed. Among them, the low importance parts of the distant view are sorted by bucketing approximation. The number of buckets is set to eight to sixteen levels according to the proportion of transparency contribution of this part.

[0091] Step 6: Analyze the frame latency distribution of a continuous 30-minute interactive browsing session on the device. The standard deviation of frame latency is lower than the control scheme before smoothing, the level switching frequency is within the normal range, and no control gain adjustment is triggered. Archive the initial error threshold and control gain after this convergence as the scheduling parameter configuration corresponding to the device's performance level. This will be used as initial parameters when subsequent devices of the same performance level load the same scene, shortening the time required for closed-loop control convergence. The specific thresholds, gain coefficients, and number of hierarchical levels involved in the embodiment are preferred reference ranges set based on internal test data. In actual implementation, they can be adjusted within the range given in this specification according to the target device performance and scene scale.

[0092] The screen projection error and node time closed-loop scheduling method provided in the above embodiments of this disclosure constructs a hierarchical Gaussian pyramid of the target scene to obtain multiple hierarchical nodes, calculates the estimated value of the screen projection error of the current frame, and obtains the current rendering work set of the current frame. Based on the budget deviation between the actual value of rendering time and the target time budget, the screen projection error threshold of adjacent frames is adjusted to obtain adjacent rendering work sets of adjacent frames. Adjacent rendering nodes are determined based on rendering differences, and the current rendering work set is processed in blocks to generate a processing strategy for the blocks to be processed. This achieves continuous detail level management with viewpoint adaptation, enabling smooth transitions between near and far objects without hierarchical jumps. At the same time, the rendering tasks are balanced, significantly improving the frame rate and memory utilization efficiency of real-time rendering of large-scale scenes.

[0093] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a screen projection error and node time closed-loop scheduling device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0094] like Figure 5 As shown, the screen projection error and node time closed-loop scheduling device 200 of this embodiment includes: a construction module 201, a traversal module 202, a deviation calculation module 203, an adjustment module 204, a node determination module 205, and a processing module 206. The construction module 201 is configured to perform offline clustering and merging of all Gaussian parameters of the target scene to construct a hierarchical Gaussian pyramid, obtaining multiple hierarchical nodes. Each hierarchical node carries an approximate error upper bound and a spatial range. The approximate error upper bound is used to characterize the maximum error introduced when rendering the hierarchical node. The traversal module 202 is configured to obtain the current frame and its screen projection error threshold, traverse the hierarchical Gaussian pyramid according to the approximate error upper bound and spatial range, obtain the estimated screen projection error value of the current frame, and calculate the current rendering working set of the current frame. The deviation calculation module 203 is configured to obtain the current rendering working set. The rendering time is measured, and the budget deviation between the measured rendering time and the target time budget is calculated. The adjustment module 204 is configured to adjust the screen projection error threshold of the adjacent frames of the current frame based on the budget deviation to obtain the adjacent rendering work set of the adjacent frames. The node determination module 205 is configured to calculate the rendering difference between the current rendering work set and the adjacent rendering work set, and determine the adjacent rendering nodes based on the rendering difference. The processing module 206 is configured to perform block processing on the current rendering work set, generate at least one screen block, determine the screen block containing the adjacent rendering nodes as the block to be processed, and generate the processing strategy for the block to be processed.

[0095] It should be noted that the implementation details and technical effects of each module and unit in the screen projection error and node time closed-loop scheduling device provided in the embodiments of this disclosure can be referred to the description of other embodiments in this disclosure, and will not be repeated here.

[0096] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A closed-loop scheduling method for screen projection error and node time, characterized in that, include: Offline clustering and merging of all Gaussian parameters of the target scene is performed to construct a hierarchical Gaussian pyramid, resulting in multiple hierarchical nodes. Each hierarchical node carries an upper bound of approximate error and a spatial range. The upper bound of approximate error is used to characterize the maximum error introduced when rendering the hierarchical node. Obtain the current frame and the screen projection error threshold of the current frame, traverse the hierarchical Gaussian pyramid according to the upper bound of the approximate error and the spatial range to obtain the screen projection error estimate of the current frame, and calculate the current rendering working set of the current frame. Obtain the measured rendering time of the current rendering work set, and calculate the budget deviation between the measured rendering time and the target time budget; Based on the budget deviation, the screen projection error threshold of the adjacent frames of the current frame is adjusted to obtain the adjacent rendering working set of the adjacent frames; Calculate the rendering difference between the current rendering work set and the adjacent rendering work sets, and determine the adjacent rendering nodes based on the rendering difference; The current rendering work set is divided into blocks to generate at least one screen block. The screen block containing the adjacent rendering node is determined as the block to be processed, and a processing strategy for the block to be processed is generated.

2. The method according to claim 1, characterized in that, The steps of obtaining the current frame and its screen projection error threshold, traversing the hierarchical Gaussian pyramid based on the approximate error upper bound and the spatial range to obtain the estimated screen projection error of the current frame, and calculating the current rendering working set of the current frame, include: Starting from the bottom level node of the hierarchical Gaussian pyramid, for each level node traversed, in response to the intersection of the spatial range of the level node with the view frustum range of the current frame, the screen projection error estimate corresponding to the level node is calculated based on the upper bound of the approximation error of the level node and the screen projection scale of the level node in the current frame. For the level node, the following comparison operation is performed: the screen projection error estimate of the level node is compared with the screen projection error threshold of the current frame. If the screen projection error estimate is not greater than the screen projection error threshold of the current frame, the level node is added to the current rendering working set, and traversal of each child node of the level node is stopped. If the screen projection error estimate is greater than the screen projection error threshold of the current frame, the threshold comparison is performed on each child node of the level node until the screen projection error estimate of the child node is not greater than the screen projection error threshold of the current frame or the traversal has reached the top level node of the level Gaussian pyramid.

3. The method according to claim 2, characterized in that, The step of obtaining the measured rendering time of the current rendering working set and calculating the budget deviation between the measured rendering time and the target time budget includes: The start timestamp and end timestamp of the current frame are obtained respectively, and the difference between the start timestamp and the end timestamp is determined as the measured value of the rendering time. Calculate the difference between the measured rendering time and the target time budget, and use the difference as the budget deviation.

4. The method according to claim 3, characterized in that, The step of adjusting the screen projection error threshold of adjacent frames of the current frame based on the budget deviation to obtain the adjacent rendering working set of the adjacent frames includes: The screen projection error threshold of the adjacent frames is calculated based on the budget deviation, and the threshold adjustment amount is proportional to the absolute value of the budget deviation; Based on the adjacent frames and their screen projection error thresholds, the traversal of the hierarchical Gaussian pyramid is performed to bring the frame time of the adjacent frames to the target time budget, thereby generating the adjacent rendering working set of the adjacent frames.

5. The method according to claim 4, characterized in that, The step of calculating the rendering difference between the current rendering working set and the adjacent rendering working sets, and determining adjacent rendering nodes based on the rendering difference, includes: The hierarchical node set of the current rendering work set and the hierarchical node set of the adjacent rendering work set are compared by hierarchical switching to determine the adjacent rendering nodes; The adjacent rendering nodes are subjected to a rendering gradient transition process.

6. The method according to claim 5, characterized in that, The step of segmenting the current rendering workset to generate at least one screen block, identifying the screen block containing the adjacent rendering node as the block to be processed, and generating a processing strategy for the block to be processed includes: The screen space of the current frame is divided into at least one screen block of a preset size; For each adjacent rendering node, the projection coverage area of ​​the adjacent rendering node in the screen space is determined based on the spatial range of the Gaussian pyramid in the hierarchy. Screen blocks that overlap with the projection coverage area are identified as the blocks to be processed; Generate a processing strategy for each of the blocks to be processed, the processing strategy including: performing pixel rendering on the blocks to be processed, and performing smooth transition processing on the adjacent rendering nodes involved in the pixel rendering.

7. A screen projection error and node time closed-loop scheduling device, characterized in that, include: The construction module is configured to perform offline clustering and merging of all Gaussian parameters of the target scene to construct a hierarchical Gaussian pyramid, resulting in multiple hierarchical nodes. Each hierarchical node carries an upper bound of the approximate error and a spatial range. The upper bound of the approximate error is used to characterize the maximum error introduced when rendering the hierarchical node. The traversal module is configured to obtain the current frame and the screen projection error threshold of the current frame, traverse the hierarchical Gaussian pyramid according to the upper bound of the approximate error and the spatial range, obtain the screen projection error estimate of the current frame, and calculate the current rendering work set of the current frame. The deviation calculation module is configured to obtain the measured rendering time of the current rendering work set and calculate the budget deviation between the measured rendering time and the target time budget. The adjustment module is configured to adjust the screen projection error threshold of the adjacent frames of the current frame based on the budget deviation, so as to obtain the adjacent rendering working set of the adjacent frames; The node determination module is configured to calculate the rendering difference between the current rendering work set and the adjacent rendering work sets, and determine the adjacent rendering nodes based on the rendering difference; The processing module is configured to perform block processing on the current rendering work set, generate at least one screen block, determine the screen block containing the adjacent rendering node as the block to be processed, and generate the processing strategy for the block to be processed.

8. A storage device, comprising: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, wherein, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

10. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the method as described in any one of claims 1-6.

Images