Graphic processing unit cooperative processing method and system for accelerating three-dimensional generative model

Abstract

The invention discloses a graphic processing unit cooperative processing method and system for accelerating a three-dimensional generative model. The model comprises a diffusion processing stage and a Gaussian splash rendering stage. In the diffusion processing stage, the time step number and the data precision calculated by the neural network are dynamically adjusted according to the visual angle sensitivity, and a specific graphic processing unit core is scheduled for execution. In the Gaussian splash rendering stage, pixel block tasks are predicted and intelligently scheduled to a stream processor based on historical data; and in the stream processor, the processing sequence of the Gaussian ball by the rendering unit is optimized according to the geometric distance, and the close-range unit is preferentially processed. During back propagation, the built-in logic of the stream processor preaggregates the gradient contributions of the same Gaussian ball, and then updates the gradient contributions through a single write operation. Through collaborative optimization of software and hardware, the efficiency bottleneck of the 3D generation model on a graphic processing unit is effectively solved, the calculation speed, the throughput and the hardware utilization rate are remarkably improved, and the model generation time is shortened.

Description

Technical Field

[0001] This invention relates to the field of computer graphics processing technology, and in particular to a GPU co-processing method and system for accelerating 3D generative models. Background Art

[0002] Existing 3D generative models, especially those combining 3D Gaussian splashing (3DGS) technology with diffusion processes, such as DreamGaussian, excel in generating high-quality 3D content. However, such models face significant challenges in terms of performance on graphics processing units (GPUs).

[0003] First, the 3DGS rendering pipeline suffers from insufficient hardware utilization on the GPU. During rendering, the CUDA cores within the GPU's Streaming Processors (SMs) are underutilized. Furthermore, the scheduling and execution of thread bundles within the SMs may also exhibit redundancy and inefficiency.

[0004] Secondly, the workload among SMs is severely unbalanced during the 3DGS rendering stage. The number of Gaussian spheres required to render varies greatly depending on the pixel block, causing the SM handling the heavily loaded pixel blocks to become the bottleneck of the entire rendering process, while other SMs may be idle or lightly loaded. Existing static or simple scheduling strategies are insufficient to effectively solve this problem.

[0005] To address the aforementioned issues, there is an urgent need for a collaborative processing method that can optimize GPU resource utilization, balance the load between SMs, and improve overall rendering efficiency, thereby accelerating the GPU execution speed of 3D generative models.

[0006] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0007] In view of this, the present invention provides a method and system for collaborative processing of graphics processing units to accelerate the generation of 3D models, thereby overcoming the problem of unsatisfactory running efficiency of 3D generated models on graphics processing units in the prior art. By dynamically adjusting the calculation parameters based on viewpoint sensitivity and collaboratively utilizing different types of computing cores in the diffusion processing stage, scheduling the pixel blocks to be processed using the rendering workload data of the already rendered pixel blocks in the Gaussian splash rendering stage, determining the execution order of the rendering units of the Gaussian sphere to be rendered within the selected stream processor, and merging multiple gradient components in the backpropagation sub-stage, the computational efficiency of the graphics processing unit and the utilization rate of hardware resources are improved, ultimately shortening the 3D model generation time.

[0008] This invention provides a method for accelerating the collaborative processing of graphics processing units in 3D generated models. The 3D generated model includes a diffusion processing stage and a Gaussian splash rendering stage. The method includes the following steps:

[0009] During the diffusion processing stage, the following operations are performed using the graphics processing unit:

[0010] The sensitivity level of the current processing perspective is determined based on at least one predefined sensitivity assessment parameter associated with the current processing perspective.

[0011] Based on the determined sensitivity level, adjust one or more computation configuration parameters for the neural network computation used in the diffusion processing phase. These computation configuration parameters include the number of computation time steps and the data processing precision.

[0012] The specific type of computing core within the graphics processing unit performs neural network computation based on one or more adjusted computation configuration parameters. When the data processing precision is adjusted to a first preset low precision, the specific type of computing core is a tensor core.

[0013] During the Gaussian splash rendering stage, the following operations are performed using the graphics processing unit:

[0014] A method for accelerating 3D generative models using graphics processing units based on at least one rendered pixel block, wherein the 3D generative model includes a diffusion processing stage and a Gaussian splash rendering stage, and the method includes the following steps:

[0015] During the diffusion processing stage, the following operations are performed using the graphics processing unit:

[0016] The sensitivity level of the current processing perspective is determined based on at least one predefined sensitivity assessment parameter associated with the current processing perspective.

[0017] Based on the determined sensitivity level, adjust one or more computation configuration parameters for the neural network computation used in the diffusion processing phase. These computation configuration parameters include the number of computation time steps and the data processing precision.

[0018] The specific type of computing core within the graphics processing unit performs neural network computation based on one or more adjusted computation configuration parameters. When the data processing precision is adjusted to a first preset low precision, the specific type of computing core is a tensor core. The recorded rendering workload data generates the predicted rendering workload for the pixel block to be processed.

[0019] Based on the predicted rendering workload and the availability of multiple stream processors within the graphics processing unit, the pixel block to be processed is assigned to a selected stream processor for rendering.

[0020] Within the selected stream processor, for a Gaussian sphere to be rendered within the pixel block to be processed, the geometric information of the Gaussian sphere and the geometric information of a set of rendering units within the pixel block to be processed are obtained. Each rendering unit corresponds to a thread bundle of the graphics processing unit.

[0021] Based on the geometric information of the Gaussian sphere and the geometric information of a set of rendering units, the execution order of the rendering operations of the Gaussian sphere to be rendered is determined. The execution order makes the rendering units that are closer to the geometric center of the Gaussian sphere to be rendered in space prioritized.

[0022] According to a defined execution order, control a group of rendering units to perform rendering operations on the Gaussian sphere to be rendered;

[0023] In the backpropagation sub-stage of the Gaussian splash rendering phase, multiple gradient components pointing to the same target Gaussian sphere are generated in parallel by multiple threads within a single thread bundle:

[0024] Within the stream processor where the thread is currently executing, the built-in arithmetic logic resources of the stream processor are used to merge multiple gradient components into a single aggregated gradient data.

[0025] The gradient state of the target Gaussian sphere is updated using a single aggregated gradient data by making a write operation to the shared cache of the graphics processing unit for the target Gaussian sphere.

[0026] In some optional embodiments, predefined sensitivity evaluation parameters include: the number of Gaussian spheres accumulated in previous processing cycles and / or the accumulated rendering loss value associated with the spatial region to which the current processing viewpoint belongs.

[0027] In some alternative embodiments, the step of adjusting one or more computation configuration parameters for the neural network computation used in the diffusion processing phase includes: if the determined sensitivity level indicates low sensitivity, reducing the number of computation time steps to a first proportion of a baseline time step and setting the data processing precision to a first preset low precision.

[0028] In some optional embodiments, a specific type of computing core is a CUDA core within a graphics processing unit when the data processing precision is not adjusted to a first preset low precision.

[0029] In some optional embodiments, the step of assigning a pixel block to be processed to a selected stream processor for rendering processing based on the predicted rendering workload and the availability of multiple stream processors within the graphics processing unit includes: when the predicted rendering workload exceeds a predetermined threshold, assigning the pixel block to the stream processor that first becomes idle among all available stream processors.

[0030] In some optional embodiments, after controlling a group of rendering units to perform the rendering operation on the Gaussian sphere to be rendered according to a determined execution order, the method further includes:

[0031] If a scheduled rendering unit determines that its alpha contribution to the Gaussian sphere to be rendered is lower than a preset pruning threshold after completing the calculation, then for other rendering units in a group that have not yet been scheduled and whose spatial proximity to the geometric center of the Gaussian sphere to be rendered is lower than that of the scheduled rendering unit, the rendering operation of the Gaussian sphere to be rendered is terminated.

[0032] In some alternative embodiments, the arithmetic logic resources built into the stream processor include an addition tree consisting of one or more parallel adders.

[0033] In some optional embodiments, the shared cache of the graphics processing unit is an L2-level cache, and write operations are atomic addition operations.

[0034] This invention also provides a method for accelerating the collaborative processing of graphics processing units in 3D generated models. The 3D generated model includes a diffusion processing stage and a Gaussian splash rendering stage. The method includes the following steps:

[0035] For the diffusion processing stage, adaptive computing resource allocation is performed based on viewpoint characteristics. The allocation includes dynamically setting the number of iterations and data representation precision for neural network computation in this stage based on the evaluation results of the computational difficulty of the current processing viewpoint, and scheduling computing units within the graphics processing unit that match the set data representation precision to perform neural network computation.

[0036] For the Gaussian splash rendering stage, collaborative optimizations are performed with the goal of improving the parallel processing efficiency of the graphics processing unit. These collaborative optimizations include:

[0037] Implement a stream processor-level task load balancing scheduling strategy based on historical rendering data to even out the computational load of each SM;

[0038] Inside the stream processor, a thread bundle-level rendering task sorting mechanism based on geometric proximity is implemented to prioritize rendering interactions with high contribution probability and allow computations based on early evaluation results to be skipped.

[0039] Inside the stream processor, a hardware-assisted aggregation mechanism for gradient updates is implemented to reduce the frequency of atomic operations on shared memory resources.

[0040] In some optional embodiments, the assessment of the computational difficulty of the current processing viewpoint includes comparing the rendering complexity metric of the spatial region to which the current viewpoint belongs with the global or neighborhood average complexity metric.

[0041] In some optional embodiments, the stream processor-level task load balancing scheduling strategy includes: using a table structure that maintains the actual execution cycle of the most recently processed pixel blocks to predict the execution cycle of the pixel blocks to be scheduled, and assigning pixel blocks with longer predicted execution cycles to currently or soon-to-be-idle stream processors.

[0042] In some optional embodiments, the thread bundle-level rendering task sorting mechanism includes: for a Gaussian object to be rendered, calculating the distance between it and the center point of the rendering area corresponding to all thread bundles within the covered pixel block, and scheduling these thread bundles in ascending order of distance; if any thread bundle determines that the Gaussian object's contribution to the current area is negligible after processing, then subsequent thread bundles that are further away will no longer process the Gaussian object.

[0043] In some alternative embodiments, the hardware-assisted aggregation mechanism includes: summing the gradients generated by all threads within a single thread bundle for the same Gaussian object using a parallel adder array within the SM to form a summed gradient, and then updating the gradient of the Gaussian object through a single memory transaction.

[0044] This invention also provides a graphics processing unit collaborative processing system for accelerating 3D generated models. The 3D generated model includes a diffusion processing stage and a Gaussian splash rendering stage. The system includes at least one graphics processing unit, which is configured to:

[0045] Perform the following operations during the diffusion process:

[0046] The sensitivity level of the current processing perspective is determined based on at least one predefined sensitivity assessment parameter associated with the current processing perspective.

[0047] Based on the determined sensitivity level, adjust one or more computation configuration parameters for the neural network computation used in the diffusion processing phase. These computation configuration parameters include the number of computation time steps and the data processing precision.

[0048] Neural network computation is performed using a specific type of computational core within the graphics processing unit, based on one or more adjusted computational configuration parameters, wherein the specific type of computational core is a tensor core when the data processing precision is adjusted to a first preset low precision.

[0049] Perform the following operations during the Gaussian splash rendering stage:

[0050] Based on the rendering workload data recorded by at least one rendered pixel block, generate the predicted rendering workload for the pixel block to be processed.

[0051] Based on the predicted rendering workload and the availability of multiple stream processors within the graphics processing unit, the pixel block to be processed is assigned to a selected stream processor for rendering.

[0052] Within the selected stream processor, for a Gaussian sphere to be rendered within the pixel block to be processed, the geometric information of the Gaussian sphere and the geometric information of a group of rendering units within the pixel block to be processed are obtained, with each rendering unit corresponding to a thread bundle of the graphics processing unit.

[0053] Based on the geometric information of the Gaussian sphere and the geometric information of a set of rendering units, the execution order of the rendering operations of the Gaussian sphere to be rendered is determined. The execution order makes the rendering units that are closer to the geometric center of the Gaussian sphere to be rendered in space prioritized.

[0054] According to a defined execution order, control a group of rendering units to perform rendering operations on the Gaussian sphere to be rendered;

[0055] In the backpropagation sub-stage of the Gaussian splash rendering phase, multiple gradient components pointing to the same target Gaussian sphere are generated in parallel by multiple threads within a single thread bundle:

[0056] Within the stream processor where the thread is currently executing, the built-in arithmetic logic resources of the stream processor are used to merge multiple gradient components into a single aggregated gradient data.

[0057] The gradient state of the target Gaussian sphere is updated using a single aggregated gradient data by making a write operation to the shared cache of the graphics processing unit for the target Gaussian sphere.

[0058] In some optional embodiments, the graphics processing unit is further configured during the diffusion processing phase to update predefined sensitivity evaluation parameters associated with the spatial region to which the current processing viewpoint belongs after determining the sensitivity level of the current processing viewpoint.

[0059] In some alternative embodiments, when the graphics processing unit adjusts one or more computation configuration parameters for neural network computation in the diffusion processing phase, if the determined sensitivity level indicates low sensitivity, it reduces the number of computation time steps to a first proportion of a reference time step and sets the data processing precision to a first preset low precision.

[0060] In some optional embodiments, after the graphics processing unit controls a group of rendering units to perform rendering operations on the Gaussian sphere to be rendered according to a determined execution order, it is further configured to: if a scheduled rendering unit determines that the alpha contribution of its alpha contribution to the Gaussian sphere to be rendered is lower than a preset pruning threshold after completing the calculation of the alpha contribution of the Gaussian sphere to be rendered, then for other rendering units in the group of rendering units that have not yet been scheduled and whose spatial proximity to the geometric center of the Gaussian sphere to be rendered is lower than that of the scheduled rendering unit, the rendering operations on the Gaussian sphere to be rendered are terminated.

[0061] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention.

[0062] The graphics processing unit collaborative processing method and system for accelerating the generation of 3D models, as proposed in this invention, have the following beneficial effects:

[0063] This invention significantly reduces computational load under low-sensitivity perspectives and improves tensor core utilization by introducing viewpoint sensitivity assessment and dynamic modulation of computational parameters during the diffusion processing stage. In the Gaussian splash rendering stage, the pixel block prediction engine achieves load balancing among stream processors, the thread bundle rearrangement engine reduces redundant alpha calculations, and the gradient accumulation unit effectively reduces atomic operation latency. Experimental results show that this collaborative processing method significantly improves 3D generation efficiency and graphics processing unit hardware resource utilization while reducing power consumption, providing an effective solution for the rapid generation of high-quality 3D content, all while ensuring generation quality. Attached Figure Description

[0064] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0065] Figure 1 This is a flowchart of the diffusion processing stage of a graphics processing unit collaborative processing method for accelerating 3D generated models according to an embodiment of the present invention.

[0066] Figure 2 This is a flowchart of a Gaussian splash rendering stage of a graphics processing unit collaborative processing method for accelerating 3D generated models according to an embodiment of the present invention. Detailed Implementation

[0067] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0068] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0069] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined. Therefore, the actual execution order may change depending on the specific circumstances.

[0070] The technical solution involved in this invention relies on a deep understanding of the computational bottlenecks in 3D generative models, particularly those combining diffusion processes and Gaussian splash rendering. The diffusion process is an iterative denoising process, and its computational complexity is directly related to the number of iterations and the size of the neural network. Gaussian splash rendering, however, faces challenges such as uneven load distribution between pixel blocks, redundant computation within thread bundles, and atomic operation bottlenecks in backpropagation. By utilizing viewpoint sensitivity assessment to guide computational resource allocation, the computational load of the diffusion process can be adaptively adjusted, thereby reducing overall computational overhead while maintaining generation quality. Furthermore, through refined management of the Gaussian splash rendering pipeline, load balancing between streaming processors (SMs), computational optimization within SMs, and acceleration of backpropagation can be achieved. These optimization measures work synergistically to significantly improve computational efficiency and hardware resource utilization when running 3D generative models on GPUs (Graphics Processing Units), thereby shortening model generation time.

[0071] like Figure 1 and Figure 2 As shown, this embodiment of the invention provides a GPU co-processing method for accelerating 3D generative models. This method targets 3D generative models that include a diffusion processing stage and a splatter rendering stage, aiming to improve their running efficiency on a GPU.

[0072] During the diffusion processing phase, the following operations are performed using the GPU:

[0073] S110. Determine the sensitivity level of the current processing viewpoint based on at least one predefined sensitivity evaluation parameter associated with the current processing viewpoint. Sensitivity evaluation parameters are indicators used to quantify the importance of a viewpoint, reflecting the complexity of the visual information contained within that viewpoint or the degree of need for refined processing. These parameters can be the number of Gaussian spheres in the spatial region associated with the viewpoint, rendering loss values, etc., used to distinguish important viewpoints requiring high-precision processing from less important viewpoints that can be processed with lower precision. The process of determining the sensitivity level is essentially prioritizing viewpoints, providing a basis for subsequent allocation of computational resources.

[0074] S120. Based on the determined sensitivity level, adjust one or more computational configuration parameters for the neural network computation used in the diffusion processing stage. These parameters include the number of computation time steps and the data processing precision. The computational configuration parameters determine the complexity and precision of the neural network computation, directly affecting the computational load and the quality of the generated data. For example, the number of computation time steps determines the number of iterations in the diffusion process, and the data processing precision determines the precision of the numerical calculations. Adjusting these parameters aims to reduce computational complexity while ensuring the quality of the generated data, thereby improving computational efficiency. Specifically, a low-sensitivity perspective can reduce the number of computation time steps and decrease the data processing precision, thus saving computational resources.

[0075] S130. Utilize specific types of computing cores within the GPU to perform neural network computations based on one or more adjusted computational configuration parameters. When the data processing precision is adjusted to a first preset low precision, the specific type of computing core is a tensor core. When the data processing precision decreases to the first preset low precision, such as INT8, the tensor cores in the GPU are scheduled for computation, fully leveraging the advantages of tensor cores in low-precision matrix operations to further improve computational efficiency. Optionally, when the data processing precision is not adjusted to the first preset low precision, CUDA cores or tensor cores supporting the corresponding data precision can be used to perform neural network computations. This stage dynamically adjusts the computational configuration of the diffusion processing stage through perspective sensitivity evaluation and collaboratively utilizes different types of computing cores within the GPU. This reduces computational overhead while ensuring generation quality, alleviating the problem of huge computational overhead in the diffusion process, and optimizing the collaboration of different computing units within the GPU, thereby improving computational efficiency and GPU throughput.

[0076] During the Gaussian splash rendering stage, the following operations are performed using the GPU:

[0077] S210. Based on the rendering workload data recorded for at least one rendered pixel block, generate a predicted rendering workload for the pixel block to be processed. The rendering workload data for the rendered pixel block refers to information such as the computational amount or time cost of the previously rendered pixel block, reflecting the rendering complexity of the pixel block. Using this data to predict the workload of the pixel block to be processed can achieve more intelligent task scheduling and balance the load among SMs. Optionally, the rendering workload data may include the number of Gaussian spheres processed for the pixel block or the number of SM execution cycles.

[0078] S220: Based on the predicted rendering workload and the availability of multiple SMs (Streaming Processors) within the GPU, the pixel block to be processed is assigned to a selected SM for rendering. Assigning pixel blocks to appropriate SMs based on the predicted rendering workload aims to address the severe workload imbalance among SMs during the 3DGS rendering stage. If a pixel block is predicted to require a high amount of computation, it is preferentially assigned to an idle SM to avoid situations where some SMs are overloaded while others are idle.

[0079] S230. Within the selected SM (Streaming Manager), for a Gaussian sphere to be rendered within the pixel block to be processed, acquire the geometric information of the Gaussian sphere and the geometric information of a group of rendering units within the pixel block to be processed. Each rendering unit corresponds to a thread bundle of the GPU. The geometric information of the Gaussian sphere includes parameters such as its position and shape, while the geometric information of the rendering unit includes the position information of the pixel area covered by the rendering unit. This geometric information is used to calculate the distance between the Gaussian sphere and the rendering unit, thereby determining the rendering order.

[0080] S240. Based on the geometric information of the Gaussian sphere and a set of rendering units, the execution order of rendering operations on the Gaussian sphere to be rendered by the set of rendering units is determined. The execution order prioritizes rendering units that are spatially closer to the geometric center of the Gaussian sphere to be rendered. By prioritizing rendering units closer to the Gaussian sphere, it is possible to determine earlier in the alpha pruning process whether the Gaussian sphere contributes to the final color, thereby reducing redundant calculations and improving rendering efficiency. This process solves the problem of a large amount of redundant calculations in the 3DGS rendering stage.

[0081] S250. Following a predetermined execution order, control a group of rendering units to perform rendering operations on the Gaussian sphere to be rendered. Scheduling rendering units according to the ordered sequence ensures earlier alpha pruning, thereby reducing the computational overhead for invalid Gaussian spheres. Optionally, if a scheduled rendering unit, after completing its alpha contribution calculation for the Gaussian sphere to be rendered, determines that the alpha contribution is below a preset pruning threshold, then for other rendering units in the group that have not yet been scheduled and whose spatial proximity to the geometric center of the Gaussian sphere to be rendered is lower than that of the scheduled rendering unit, their rendering operations on the Gaussian sphere to be rendered are suspended.

[0082] Furthermore, in the backpropagation sub-stage of the Gaussian splash rendering phase, for multiple gradient components pointing to the same target Gaussian sphere, generated by multiple threads in parallel computation within a single thread bundle: First, within the SM currently executing the thread bundle, the multiple gradient components are merged using the SM's built-in arithmetic logic resources to form a single aggregated gradient data. This step aims to address the significant latency introduced by atomic operations in 3DGS backpropagation. By pre-aggregating gradients within the SM, the number of atomic operations on shared memory can be reduced, thereby reducing latency caused by memory access conflicts and serialization execution. The SM's built-in arithmetic logic resources can include an addition tree composed of one or more parallel adders for efficient accumulation of gradient components. Optionally, the arithmetic logic resources can be dedicated hardware adders or reused ALU resources within the SM.

[0083] S260. The gradient state of the target Gaussian sphere is updated using a single aggregated gradient data through a single write operation to the GPU's shared cache. By combining multiple atomic operations into one, contention on the shared memory bus is significantly reduced, further accelerating the backpropagation process. The GPU's shared cache can be an L2-level cache, in which case the write operation is an atomic addition operation, updating the gradient state of the target Gaussian sphere through atomic addition.

[0084] The aforementioned collaborative processing method significantly improves the efficiency of 3D generative models on GPUs. Specifically: First, viewpoint sensitivity modulation reduces the computational load in the diffusion processing stage, and the utilization of tensor cores accelerates low-precision calculations, thereby shortening the diffusion processing stage time. Second, the pixel block prediction engine balances the load between SMs, the thread bundle rearrangement engine reduces redundant alpha calculations, and the gradient accumulation unit reduces atomic operation latency, thus improving the efficiency of the Gaussian splashing rendering stage. Finally, this method significantly reduces the overall generation time of 3D generative models and improves the utilization of GPU hardware resources.

[0085] In some embodiments, predefined sensitivity evaluation parameters for assessing the sensitivity of the current processing viewpoint include: the number of Gaussian spheres accumulated in previous processing cycles and / or the accumulated rendering loss value associated with the spatial region to which the current processing viewpoint belongs. In this embodiment of the invention, viewpoint sensitivity assessment aims to quantify the computational resource requirements of the current viewpoint during 3D generation. This evaluation parameter directly affects the computational configuration of subsequent diffusion processing stages, thereby achieving adaptive allocation of computational resources and solving the problems of huge computational overhead (P1) and insufficient coordination between different computing units within the GPU (P6) in the prior art. Its technical principle lies in the fact that by statistically analyzing historical rendering data associated with a specific spatial region, the computational volume that the region may require in subsequent renderings can be effectively predicted.

[0086] Specifically, the number of Gaussian spheres (N_region) associated with the spatial region of the current processing viewpoint, accumulated over previous processing cycles, reflects the geometric complexity of that region. A higher number of Gaussian spheres indicates richer detail in the region, requiring more computational resources and higher precision for rendering and training. The accumulated rendering loss (L_region) reflects the convergence degree of that region. A higher loss value indicates that the generation quality of that region still needs improvement, requiring more time steps and higher precision for optimization.

[0087] In other embodiments, the predefined sensitivity evaluation parameters may also include, but are not limited to:

[0088] Region texture complexity: an indicator that measures the richness of texture details in a region, which can be calculated based on methods such as image gradient and frequency analysis.

[0089] Regional depth variation: measures the degree of change in depth information in a region. The greater the depth variation, the more complex the geometry usually is.

[0090] User interaction frequency: If the 3D generation process is interactive, the frequency of user interaction in a specific area can be counted. The higher the interaction frequency, the higher the user's attention to that area, and the higher the generation quality is required.

[0091] By using these predefined sensitivity assessment parameters and combining them with appropriate assessment and decision-making strategies, the sensitivity level of the current viewpoint can be determined more accurately, thereby allocating computational resources more effectively and improving 3D generation efficiency. For example, when a region has a small number of Gaussian spheres and low rendering loss, it can be identified as a low-sensitivity region, thus reducing the computational load of the diffusion process, increasing generation speed, and maintaining overall generation quality.

[0092] In some embodiments, the step of adjusting one or more computational configuration parameters for neural network computation in the diffusion processing stage includes: if the determined sensitivity level indicates low sensitivity, reducing the number of computation time steps to a first proportion of a baseline time step and setting the data processing precision to a first preset low precision. The aim is to reduce computational overhead and improve GPU resource utilization while ensuring generation quality. Adjusting the number of computation time steps and data processing precision according to the viewpoint sensitivity level is a key technical means. If the determined sensitivity level indicates low sensitivity, reducing the number of computation time steps to a first proportion of a baseline time step and setting the data processing precision to a first preset low precision.

[0093] In this invention, the baseline time step count refers to the total number of time steps typically used by the diffusion model without viewpoint sensitivity adjustment, such as 900 steps. The first ratio refers to the reduction ratio relative to this baseline time step count, which can be a preset fixed value or a value dynamically adjusted according to the sensitivity level. For example, the first ratio can be 30%-50%, meaning the time step count will be reduced to 70%-50% of the original. Reducing the time step count directly reduces the number of iterations in the diffusion process, thereby reducing computational load. Data processing precision refers to the precision of the data type used in neural network computation, such as FP32, FP16, INT8, etc. Reducing data processing precision can reduce memory usage and computational complexity, while also fully utilizing the GPU's tensor cores for acceleration. The first preset low precision refers to a lower precision than the standard precision, such as a lower precision of FP16, such as INT8 or INT4. The specific precision depends on the GPU hardware support and the requirements for generation quality. This technical feature aims to achieve a balance between computational complexity and generation quality by using fewer time steps and lower data precision in low-sensitivity viewpoints.

[0094] In some specific embodiments, the following more specific configurations may be adopted:

[0095] The baseline time step count is set to 900 steps.

[0096] The first ratio is set to a fixed value of 50%, which means that the time steps for the low-sensitivity viewpoint are reduced to 450 steps.

[0097] The first preset low precision setting is INT8.

[0098] The first proportion can also be dynamically adjusted based on the sensitivity level. For example, three sensitivity levels can be defined: high, medium, and low, corresponding to time step proportions of 100%, 75%, and 50%, respectively. The higher the sensitivity level, the more time steps are required.

[0099] In addition to INT8, the first preset low precision can also be selected as INT4 or other precisions supported by tensor cores. Choosing a lower precision can further accelerate computation, but may require more refined quantization and calibration strategies to ensure generation quality.

[0100] Adjusting computational configuration parameters can also include changing the batch size. For low-sensitivity viewing angles, the batch size can be increased appropriately to improve GPU utilization.

[0101] Adjusting computational configuration parameters can also include tweaking the optimizer's parameters. For example, for low-sensitivity perspectives, the learning rate can be increased to accelerate convergence.

[0102] By reducing the number of time steps and lowering data precision from a low-sensitivity perspective, the computational overhead of the diffusion processing stage can be significantly reduced, thereby improving the overall generation speed. Combined with GPU tensor cores, computational efficiency can be greatly improved while maintaining a certain generation quality, alleviating the problem of high computational overhead in the diffusion process. This makes it possible to run high-quality 3D generated models on resource-constrained devices and provides a potential technical solution for real-time applications. For example, in metaverse or game applications, frequent switching of the user's perspective may lead to lower generation quality requirements in some areas; the embodiments of this invention can significantly reduce resource overhead, thereby improving the user experience.

[0103] In some embodiments, when the data processing precision is not adjusted to a first preset low precision, the specific type of computing core performing the neural network computation in the diffusion processing stage is the CUDA core within the GPU. This means that the GenTC architecture of this embodiment does not solely rely on tensor cores for acceleration, but dynamically switches between CUDA cores and tensor cores based on viewpoint sensitivity. When the viewpoint is determined to be highly sensitive, requiring higher data precision to ensure generation quality, CUDA cores are still used for computation. As a general-purpose computing unit, CUDA cores can support a wider range of data types and operations, ensuring the accuracy and flexibility of high-precision computation. This dynamic switching strategy fully utilizes the advantages of different types of computing units within the GPU, maximizing acceleration while ensuring generation quality.

[0104] As an alternative to the CUDA core, the specific type of computing core in this embodiment can also be the latest generation of tensor cores supporting FP16 mixed-precision computing, or other types of general-purpose or special-purpose computing cores that may emerge in the future, as long as they can meet the requirements of high-precision computing. Even when the data processing precision is not adjusted to the first preset low precision, other suitable computing cores can still be selected based on actual hardware resources and computing needs. Furthermore, in specific implementations, a precision threshold can be set. For example, when the data processing precision is higher than INT8, the CUDA core is used; when the data processing precision is equal to or lower than INT8, the tensor core is used. This threshold can be adjusted according to actual hardware performance and model characteristics.

[0105] By employing the aforementioned techniques, it is ensured that GPU computing resources can be fully utilized even when high-precision calculations are required, avoiding the quality degradation that might result from relying solely on low-precision calculations. While maintaining the quality of generated 3D models, overall computational efficiency is improved, guaranteeing the rapid generation of high-quality 3D content.

[0106] In some embodiments, the step of assigning a pixel block to be processed to a selected SM for rendering based on the predicted rendering workload and the availability of multiple SMs within the GPU includes: when the predicted rendering workload exceeds a predetermined threshold, assigning the pixel block to the SM that first becomes idle among all available SMs. A high predicted rendering workload for a pixel block indicates that the pixel block contains a large number of Gaussian spheres to be processed or complex geometric structures, requiring more computational resources. To avoid bottlenecks caused by overloading a single SM, this solution employs a dynamic threshold judgment and priority scheduling strategy. Specifically, a workload threshold is preset, which can be adjusted based on the overall computing power of the GPU, the number of SMs, or historical statistical data. When the predicted workload of a pixel block exceeds this threshold, it indicates that the pixel block is a high-load task. At this time, the scheduler no longer simply assigns the task to any idle SM, but prioritizes selecting the SM that first completes its previous task and enters an idle state among all available SMs. This strategy can maximize the utilization of the GPU's parallel computing power, balance the load among SMs, and thus shorten the overall rendering time.

[0107] In this embodiment, the predetermined threshold can be a fixed value determined based on experience or experimentation, such as the maximum number of clock cycles required to process a pixel block, or the number of Gaussian spheres contained within the pixel block. In another implementation, the threshold can also be dynamically adjusted, for example, in real time based on factors such as the average GPU load and the busyness of the SM (Scheduler). The predicted rendering workload can be obtained in various ways, such as recording the actual rendering time of previously processed pixel blocks, the number of Gaussian spheres processed, or the computational complexity. The pixel block to be processed can be assigned to the SM with the lowest resource occupancy among all available SMs, or the SM with the longest idle time. This assignment process can be implemented using the GPU's hardware scheduler or simulated using a software algorithm.

[0108] Through the above technical solutions, high-load pixel block tasks can be allocated to idle SMs for execution more quickly, avoiding task backlog and SM idle waiting, significantly improving the overall parallel efficiency of the GPU, improving the throughput of the rendering pipeline, and thus shortening the generation time of 3D models.

[0109] In some embodiments, after controlling a group of rendering units to perform the rendering operation on the Gaussian sphere to be rendered according to a determined execution order, the method further includes:

[0110] If a scheduled rendering unit, after completing its alpha contribution calculation for the Gaussian sphere to be rendered, determines that the alpha contribution is below a preset pruning threshold, then for other rendering units in a group that have not yet been scheduled and whose spatial proximity to the geometric center of the Gaussian sphere is less than that of the scheduled unit, their rendering operations on the Gaussian sphere will be terminated. When scheduling thread bundles to render the Gaussian sphere in order of spatial distance from nearest to farthest, once the calculation result of a thread bundle indicates that the Gaussian sphere contributes very little to the final pixel color (alpha value below the preset threshold), the rendering of the Gaussian sphere by subsequent thread bundles will be immediately terminated, thereby avoiding unnecessary waste of computational resources. This scheme fully utilizes the radiation characteristics of the Gaussian sphere: the farther away from the center, the smaller the contribution. By terminating early, useless calculations can be significantly reduced, improving rendering efficiency.

[0111] Specifically, the preset pruning threshold can be flexibly set according to the application scenario and accuracy requirements. A common value range is between 0.001 and 0.1. For example, in applications with extremely high real-time requirements, a slightly larger threshold can be selected for more aggressive pruning, sacrificing a small amount of accuracy for higher speed; while in offline rendering with extremely high accuracy requirements, a smaller threshold can be selected to retain more detail. Furthermore, the alpha calculation method used for alpha contribution calculation can be the traditional alpha blending method or other equivalent alternatives well-known to those skilled in the art; this invention does not limit this.

[0112] In another embodiment, with finer control over the GPU architecture, it's possible to not only suspend unscheduled thread bundles but also attempt to interrupt currently executing thread bundles. This requires the GPU hardware to provide a corresponding thread interruption mechanism and to add monitoring of the thread bundle execution status to the thread bundle reordering engine. Once a Gaussian ball is determined to be invalid, the engine immediately sends an interrupt signal to the thread bundle processing that Gaussian ball, causing it to exit the rendering loop prematurely. This approach more thoroughly avoids redundant calculations but is also more complex to implement.

[0113] The aforementioned Alpha pruning mechanism effectively reduces the calculation of Gaussian spheres that contribute minimally to the final color, thereby significantly reducing the computational load on the 3DGS rendering pipeline, especially in areas with high Gaussian sphere density. Experiments show that this mechanism can increase the average activity of thread bundles within the SM by 10%-30%, improve rendering speed by 5%-15%, and has a negligible impact on the final rendering quality.

[0114] In some embodiments, the arithmetic logic resources built into the Streaming Processor (SM) include an addition tree composed of one or more parallel adders. During the backpropagation phase of the 3DGS rendering pipeline, the Streaming Processor (SM) is a key hardware acceleration unit for efficient gradient aggregation. During backpropagation, multiple threads within the same thread bundle compute different gradient components for the same Gaussian sphere parameter in parallel. In traditional methods, each thread individually uses atomic operations to accumulate its calculated gradient contribution into shared memory, such as the L2 cache. This results in a large number of atomic operation requests and potential memory access conflicts, becoming a performance bottleneck. This technical feature aims to significantly reduce the number of atomic operations on shared memory and alleviate atomic operation latency by integrating arithmetic logic resources—specifically, an addition tree composed of one or more parallel adders—within the SM to pre-aggregate these gradient components.

[0115] Parallel adder trees are efficient hardware adder structures that can accumulate multiple input values ​​in parallel over one or more clock cycles. For example, for an adder tree with 32 inputs, a 5-level (log2(32)) binary adder tree structure can be used, with each level containing multiple parallel adders. Such a structure can quickly accumulate gradient components from 32 threads and store the accumulated result in an accumulation result register. Then, only one atomic operation is needed to write the accumulated result from the accumulation result register to the gradient storage location in shared memory.

[0116] Parallel adders can be binary adder trees, Wallace trees, carry-lookahead adders, or other hardware structures with parallel accumulation capabilities. In terms of quantity, one or more parallel adder trees can be configured based on the size of the internal thread bundles of the SM (e.g., 32 threads) and the number of gradient components to be aggregated in each thread bundle. For different Gaussian sphere parameters such as position, scale, rotation, color, and transparency, multiple sets of parallel adder trees can be used, or a time-division multiplexing approach can be employed.

[0117] By employing the parallel adder tree built into SM, contention for the shared memory bus and atomic operation units is reduced, thereby significantly reducing the latency caused by atomic operations, accelerating the backpropagation process, and thus improving the overall training efficiency of the 3D generative model.

[0118] In some embodiments, the GPU's shared cache is an L2-level cache, and write operations are atomic addition operations. This invention relates to an update method after gradient aggregation during the backpropagation stage. Specifically, after aggregation by the gradient accumulation unit within the SM (Signal Builder), the aggregated gradient needs to be updated to the gradient storage location corresponding to the Gaussian sphere parameters for subsequent Gaussian parameter updates. This technical solution specifies an L2 cache as a shared cache and utilizes atomic addition operations to update the gradient. As a high-speed cache within the GPU, the L2 cache has lower access latency and higher bandwidth compared to main memory, making it suitable as a gradient storage location shared by multiple SMs. The atomic addition operation (atomicAdd) is a synchronization primitive that ensures data consistency; even if multiple threads attempt to update the same memory location simultaneously, it ensures the correctness of gradient updates and avoids data races.

[0119] As an alternative to L2 cache, shared cache can also be other levels of cache in the GPU, such as L1 cache, or directly use global memory. The reason for choosing L2 cache is its balance between capacity and access speed; L1 cache is typically small and may not be able to hold all the gradient information of the Gaussian ball, while direct access to global memory is too inefficient and impacts performance. Alternatively, other atomic operations can be used, such as atomicCAS (Compare and Swap), although atomicAdd is more suitable for accumulation scenarios.

[0120] The combination of L2 caching and atomic addition operations effectively addresses the gradient update bottleneck during backpropagation. Compared to direct atomic operations using main memory, L2 caching reduces memory access latency and improves update speed. Atomic addition operations, on the other hand, ensure the correctness of gradient updates, preventing training instability or model quality degradation caused by data contention. Atomic addition operations via L2 caching reduce memory access latency and improve data consistency, thereby enhancing backpropagation efficiency and model training stability.

[0121] This invention provides a GPU co-processing method for accelerating 3D generative models. The 3D generative model includes a diffusion processing stage and a Gaussian splash rendering stage. The method comprises the following steps:

[0122] S300. For the diffusion processing stage, adaptive computational resource allocation based on viewpoint characteristics is performed. This allocation includes dynamically setting the number of iterations and data representation precision for the neural network computation in this stage based on the assessment of the computational difficulty of the current processing viewpoint, and scheduling computing units within the GPU that match the set data representation precision to execute the neural network computation. Adaptive computational resource allocation based on viewpoint characteristics refers to dynamically adjusting the computational resources allocated to the diffusion processing stage, such as computational precision and the number of iterations, according to the complexity of different viewpoints and the requirements for generation quality. Viewpoint characteristics reflect the information complexity of the 3D scene observed from that viewpoint; high-complexity viewpoints typically require higher computational resources to ensure generation quality. The computational difficulty assessment result is a process of quantifying viewpoint complexity by analyzing indicators such as the number of Gaussian spheres within the viewpoint and rendering loss. Dynamically setting the number of iterations and data representation precision is an adjustment strategy based on the computational difficulty assessment result. For example, for low-complexity viewpoints, the number of iterations in the diffusion process, such as the time steps, can be reduced, and the data precision can be lowered, thereby reducing the computational load. Conversely, for high-complexity viewpoints, the number of iterations and data precision are maintained or increased to ensure generation quality. By matching computational units to the precision of data representation, different types of computational cores within the GPU can be fully utilized. For example, tensor cores are better suited for low-precision calculations, while CUDA cores can handle high-precision or complex logic. This adaptive adjustment fully leverages the heterogeneous computing capabilities of the GPU, reducing overall computational overhead. It particularly addresses the issues raised in P1 and P6 of the background technology, namely the enormous computational overhead of the diffusion process and insufficient coordination among different types of computational units within the GPU. More specifically, the assessment of computational difficulty can be achieved through spatial region partitioning and sensitivity index recording. For instance, the 3D observation space can be divided into several regions, and indicators such as the number of Gaussian spheres and rendering loss can be maintained for each region. Based on the comparison of these indicators with the global average, the sensitivity level of the current viewpoint can be determined, and the time step and data precision can be adjusted accordingly. The data representation precision can be adjusted between FP32, FP16, INT8, and INT4 precisions, and the corresponding computational unit scheduling can be either CUDA cores or tensor cores to fully utilize the GPU's computational power. Neural network computation can include, but is not limited to, convolutional layers and fully connected layers.

[0123] S400, for the Gaussian splash rendering stage, performs collaborative optimizations aimed at improving GPU parallel processing efficiency. These optimizations include:

[0124] S410. Implement a task load balancing scheduling strategy based on historical rendering data to even out the computational load of each SM. In this embodiment of the invention, the SM-level task load balancing scheduling strategy refers to intelligently allocating rendering tasks, typically in units of pixel blocks, to each stream processor (SM) in the GPU, so that the workload on each SM is as balanced as possible, avoiding the phenomenon that some SMs are overloaded while others are idle. Based on historical rendering data, the scheduling decision is not made blindly, but refers to data from previous rendering processes, such as the actual computation time consumed by each pixel block or the number of Gaussian spheres processed, to predict the workload of the current pixel block to be processed. This strategy aims to solve the problem raised by P3 in the background art, namely the problem of severe workload imbalance among SMs in the 3DGS rendering stage. More specifically, this scheduling strategy can be implemented through a pixel block prediction engine, which maintains a workload prediction table, records the actual workload information of pixel blocks that have recently been rendered, predicts the workload of the pixel blocks to be processed based on this information, and prioritizes allocating pixel blocks with high predicted workloads to idle SMs. The implementation of task load balancing scheduling can be based on a greedy scheduling algorithm that predicts workload, or a more complex scheduling algorithm that considers the current load of the task manager (SM) and task priorities.

[0125] S420. Within the SM (Rendering Service), a thread-beam-level rendering task prioritization mechanism based on geometric proximity is implemented to prioritize rendering interactions with high contribution probabilities and allow skipping of calculations based on early evaluation results. In this step, the thread-beam-level rendering task prioritization mechanism optimizes the interaction order of a given Gaussian sphere with multiple rendering patches within a pixel block, processed by different thread beams, within a single SM. Geometric proximity refers to the spatial distance between the center of the Gaussian sphere and the center of the rendering patch; closer distances generally mean greater contributions. By prioritizing rendering interactions with high contribution probabilities, it's possible to determine early on which Gaussian spheres have a significant impact on the final pixel color and which can be ignored. Allowing skipping of calculations based on early evaluation results means that if, after processing nearby rendering patches, it can be determined that the contribution of a certain Gaussian sphere is negligible (e.g., through alpha pruning), then calculations for more distant rendering patches can be skipped, thereby reducing redundant computation. This mechanism aims to address the problem raised in P4 in the background art, namely, the problem of excessive redundant computation in the 3DGS rendering stage. The thread-beam-level rendering task sorting mechanism can be implemented through a thread-beam reordering engine. This engine calculates the distance between the Gaussian sphere and each rendering patch, and sorts the thread bundles according to the distance, prioritizing the scheduling of thread bundles that are closer in distance. Calculation skipping can be implemented through the alpha pruning state interface. This interface interacts with the CUDA core that performs the alpha calculation to obtain the status of whether a specific Gaussian sphere has been pruned, thereby determining whether to skip the calculation of subsequent thread bundles.

[0126] S430. Within the SM (Session Manager), a hardware-assisted aggregation mechanism for gradient updates is implemented to reduce the frequency of atomic operations on shared memory resources. In this step, the hardware-assisted aggregation mechanism refers to using a dedicated hardware unit to pre-aggregate gradients generated by multiple threads within the SM, and then updating the aggregated gradients to shared memory. Gradient update refers to calculating the gradients of Gaussian sphere parameters, such as position, shape, and color, during backpropagation and accumulating them to the corresponding gradient storage locations. Reducing the frequency of atomic operations on shared memory resources means that by pre-aggregating, multiple independent atomic operations can be combined into one, thereby reducing contention for shared memory and improving the efficiency of backpropagation. This mechanism aims to solve the problem raised in P5 in the background technology, namely, the significant latency introduced by atomic operations in 3DGS backpropagation. The hardware-assisted aggregation mechanism can be implemented through a gradient accumulation unit, which includes a gradient input register, Gaussian ID verification logic, a parallel adder tree, and an atomic operation request interface. This unit can quickly accumulate gradients generated by multiple threads within a thread bundle within the SM, and then update them to the L2 cache or main memory through a single atomic operation. The adder tree can be implemented as a binary adder tree, a Wallace tree, etc., and the specific structure can be chosen based on the thread bundle size and hardware resources. Shared memory can be L2 cache or main memory, depending on the caching strategy.

[0127] The above technical solutions enable adaptive adjustment of computing resources based on viewpoint characteristics during the diffusion processing stage, reducing overall computing overhead. Furthermore, during the Gaussian splash rendering stage, the parallel processing efficiency of the GPU is effectively improved through SM-level task load balancing scheduling strategy, thread bundle-level rendering task sorting mechanism, and hardware-assisted aggregation mechanism. This significantly accelerates the training and inference process of the 3D generative model while ensuring the quality of the generated model.

[0128] In some embodiments, the evaluation result of the computational difficulty of the current processing viewpoint includes comparing the rendering complexity index of the spatial region to which the current viewpoint belongs with the global or neighborhood average complexity index. The evaluation result of the computational difficulty of the current processing viewpoint means that the complexity of the current viewpoint to be rendered needs to be pre-evaluated before the diffusion processing stage begins, so that computational resources can be adaptively adjusted subsequently. Specifically, this involves comparing the rendering complexity index of the spatial region to which the current viewpoint belongs with the global or neighborhood average complexity index. The rendering complexity index is used to quantify the visual complexity of the scene under a specific viewpoint. In this embodiment of the invention, the dimensions for measuring the rendering complexity index include, but are not limited to:

[0129] Gaussian point density within a region: A higher Gaussian point density generally indicates richer details in the region, requiring more computational resources for rendering. This density can be approximated by counting the number of Gaussian points projected onto the region from the current viewpoint.

[0130] Previous rendering error: If the rendering result of this region differs significantly from the reference image in previous rendering iterations, for example, if the loss function value is high, it indicates that this region requires more refined processing. This previous rendering error can be obtained by recording and analyzing historical rendering data.

[0131] Other metrics include: viewpoint direction, lighting intensity, texture complexity, and depth complexity. These metrics can be used individually or in combination.

[0132] The global or neighborhood average complexity metric is used as a benchmark to compare the complexity from the current perspective.

[0133] Specifically:

[0134] The global average complexity metric is the average rendering complexity across all viewpoints or all spatial regions. It can be the average Gaussian point density across all regions, or the average of previous rendering errors across all regions.

[0135] The neighborhood average complexity metric is the average rendering complexity within the neighborhood of the current viewpoint's spatial region. For example, it can be calculated as the average Gaussian point density of its neighboring regions. Using the neighborhood average allows for better adaptation to local variations.

[0136] The comparison operation involves comparing the rendering complexity metric of the current viewpoint with the global or neighborhood average complexity metric to determine whether the current viewpoint is a high-complexity or low-complexity viewpoint. Different comparison methods can be used, such as:

[0137] Threshold comparison: Set one or more thresholds. If the rendering complexity index of the current view exceeds the threshold, it is judged as a high-complexity view.

[0138] Ratio Comparison: Calculate the ratio of the rendering complexity metric of the current viewpoint to the global or neighborhood average complexity metric. If this ratio exceeds a threshold, the viewpoint is considered high-complexity.

[0139] Weighted comparison: The different rendering complexity metrics are weighted and summed, and then compared with a threshold or ratio.

[0140] In some embodiments, the spatial region may be divided in, but is not limited to, the following manner:

[0141] Uniform division: Divide the 3D viewing space (or the surface of the object's bounding box) into a grid of equal size.

[0142] Non-uniform partitioning: For example, based on the geometric features or texture complexity of the scene, detailed areas are divided into smaller areas, while flat areas are divided into larger areas.

[0143] Adaptive partitioning: During training, the spatial region partitioning method is dynamically adjusted to adapt to different scenarios and perspectives.

[0144] By employing the aforementioned techniques, we can more accurately assess the rendering complexity from different perspectives and provide a basis for subsequent adaptive allocation of computing resources, thereby effectively improving rendering efficiency while ensuring generation quality.

[0145] In some embodiments, the SM-level task load balancing scheduling strategy includes: using a table structure that maintains the actual execution cycle of the most recently processed pixel blocks to predict the execution cycle of the pixel blocks to be scheduled, and assigning pixel blocks with longer predicted execution cycles to currently or soon-to-be-idle SMs. This SM-level task load balancing scheduling strategy aims to solve the technical problem (P3) of workload imbalance among SMs in the 3DGS rendering pipeline. The table structure is the Workload Prediction Table (WPT) in the aforementioned TileSpeculation Engine. It utilizes the characteristic that adjacent or nearby pixel blocks have similar workloads during rendering, and predicts the workload of the pixel blocks to be processed based on the workload of the pixel blocks that have already been rendered, thereby achieving more intelligent SM task scheduling. More specifically, the WPT stores the actual workload information for recently rendered pixel blocks. For example, it can be organized by column (if the rendering order is column-wise), with each entry recording (Tile_Column_ID, Last_Completed_Row_ID, Actual_Workload). Actual_Workload can be the number of Gaussian spheres processed, the number of execution cycles of a SM (Streaming Manager) or other metrics reflecting workload. By querying the WPT, the predicted execution cycle of a pixel block to be scheduled can be obtained. During scheduling, pixel blocks with long predicted execution cycles are assigned to currently or soon-to-be-idle SMs, ensuring that computationally intensive tasks are processed promptly and preventing a particular SM from becoming a bottleneck due to handling heavily loaded pixel blocks, while other SMs remain idle or lightly loaded.

[0146] In some embodiments, the table structure can be implemented using on-chip SRAM to enable fast read and write access. In addition to the WPT, it may also include a Pending Tile Queue (PTQ) and a Comparison and Selection Unit (CSU). This strategy can be applied to column-by-column rendering, row-by-row rendering, or any other tile-based rendering method. The specific measurement method for Actual_Workload can be flexibly chosen, such as the actual number of Gaussian balls processed, the number of SM execution cycles, the number of instructions executed, etc. In some embodiments, the WPT can also store more historical information, such as workload data of multiple previously completed pixel blocks, or use methods such as weighted averaging to smooth historical data to improve prediction accuracy. As an alternative, instead of directly predicting the execution cycle, the number of Gaussian balls processed can be predicted and then converted into execution cycles according to a pre-calibrated coefficient. Furthermore, the prediction algorithm can be more complex, for example, using simple linear regression or more complex machine learning models. As a non-limiting statement, any reasonable prediction of the workload of the pixel blocks to be processed and scheduling based on the prediction results falls within the scope of protection of this invention.

[0147] By adopting a task load balancing scheduling strategy at the SM level, high-load pixel blocks are preferentially allocated to idle SMs, thereby avoiding the accumulation of high-load tasks on the same SM, effectively balancing the computational load among SMs, improving the overall parallel efficiency of the SM cluster, and shortening the 3D model generation time.

[0148] In some embodiments, the thread-beam-level rendering task sorting mechanism refers to calculating the distance between a Gaussian object to be rendered, such as a 3D Gaussian sphere, and the center point of the rendering region corresponding to all thread beams within the covered pixel block, and scheduling these thread beams in ascending order of distance. If any thread beam determines that the Gaussian object's contribution to the current region is negligible after processing, for example, by judging through an alpha value lower than a preset threshold, then subsequent thread beams that are farther away will no longer process the Gaussian object. This invention aims to solve the problem of redundant calculations in the 3DGS rendering stage. The contribution of a Gaussian sphere to different regions within a pixel block is closely related to its distance; the closer the distance, the greater the contribution, and the earlier the effectiveness of the Gaussian sphere can be determined. By prioritizing the processing of nearby regions, alpha pruning can be performed as early as possible, avoiding invalid calculations for distant regions. This invention can reduce redundant alpha calculations at the thread-beam level within the SM (Rendering Simulator), improve the internal calculation efficiency of the SM, and thus improve the overall rendering speed.

[0149] Rendering regions are typically handled by a single thread bundle; for example, a 32-thread bundle might handle a small pixel block (e.g., 4x8 pixels). When scheduling thread bundles, a priority queue can be maintained, adding thread bundles to the queue in order of distance. The Warp scheduler then retrieves the thread bundle from the head of the queue for scheduling.

[0150] Alpha pruning can be used to determine whether the contribution of a Gaussian object is negligible. The thread beam calculates the alpha value of the Gaussian sphere to the corresponding pixel region. If the alpha value is below a preset threshold, the contribution of the Gaussian sphere to that region is considered negligible. This threshold can be adjusted according to actual conditions, for example, set to 0.01, 0.001, or a smaller value. Furthermore, the alpha pruning threshold can be dynamically adjusted based on the current rendering quality. For example, if the rendering error is large, the threshold can be appropriately lowered to avoid over-pruning affecting the final rendering effect.

[0151] For example, in one specific embodiment, suppose a Gaussian sphere needs to be rendered by four thread bundles (Warp1, Warp2, Warp3, Warp4), with distances of 1, 2, 3, and 4 from the center of the Gaussian sphere, respectively. First, the thread bundles are sorted in ascending order of distance, resulting in a scheduling order of Warp1, Warp2, Warp3, Warp4. Warp1 is scheduled to perform alpha calculation first. If the alpha value calculated by Warp1 is less than a threshold, it is considered that the Gaussian sphere's contribution to the rendering area handled by Warp1 is negligible. In this case, the rendering of Warp2, Warp3, and Warp4 is skipped, and the next Gaussian sphere is processed directly.

[0152] By prioritizing the scheduling of nearby thread bundles and performing alpha pruning, redundant computations can be effectively reduced, improving the computational efficiency within the Render Stream (SM). Compared to schemes without a thread bundle-level rendering task sorting mechanism, this scheme can reduce alpha computation by approximately 15%-30% and improve rendering speed by approximately 5%-10%.

[0153] In some embodiments, the hardware-assisted aggregation mechanism utilizes a parallel adder array within the SM to sum the gradients generated by all threads within a single thread bundle for the same Gaussian object, forming a aggregated gradient. Then, a single memory transaction is used to update the gradient of the Gaussian object. This mechanism addresses the significant latency introduced by atomic operations in 3DGS backpropagation (P5), aiming to accelerate gradient aggregation through hardware and reduce the number of atomic operations on shared memory. In traditional backpropagation, each thread within a thread bundle independently calculates its gradient contribution to the parameters of a specific Gaussian object and accumulates these gradient increments in memory, typically the L2 cache or the gradient accumulation location corresponding to the Gaussian object in main memory, through atomic operations such as atomicAdd. This approach can lead to severe memory access conflicts and serialization issues in high-concurrency scenarios, becoming a performance bottleneck. The hardware-assisted aggregation mechanism of this invention integrates a parallel adder array within the SM to quickly sum the gradient contributions of all threads within the thread bundle, forming a aggregated gradient. Then, only one memory transaction, typically atomic addition, is needed to update the global gradient. This aggregation significantly reduces the frequency of atomic operations, alleviates memory access pressure, and significantly improves the efficiency of backpropagation.

[0154] The parallel adder array can be a variety of hardware adder structures, such as, but not limited to, carry-lookahead adders, carry-save adders, Wallace Tree adders, etc. The size of the adder array can be adapted to the size of the thread bundle. For example, for a 32-thread bundle, a 5-level (log2(32)) binary adder tree can be used, or other hardware circuits that can efficiently implement parallel summation can be used. For different parameters of the Gaussian object, such as position, scale, rotation, color, transparency, etc., multiple sets of parallel adder trees may be required, or the same adder tree may be time-division multiplexed. Memory transactions can be atomic addition operations or other types of memory write operations, depending on the underlying hardware architecture and cache coherence protocol. The target address of the memory transaction is usually the storage location of the gradient information of the Gaussian object, such as the address in the L2 cache or the address in main memory, which depends on the GPU's caching strategy and memory management method.

[0155] The embodiments of this invention can significantly reduce the latency of atomic operations, improve the efficiency of backpropagation, and alleviate the pressure on GPU memory bandwidth. Hardware-accelerated gradient aggregation can effectively alleviate the performance bottleneck in the backpropagation stage, thereby accelerating the entire 3D generation process.

[0156] As shown in the figure, this embodiment of the invention provides a GPU co-processing system for accelerating 3D generative models. This system aims to address the inefficiency of existing 3D generative models, especially when executed on GPUs. The 3D generative model includes a diffusion processing stage and a Gaussian splash rendering stage. This system significantly improves generation efficiency through co-optimization of these two stages. The system includes at least one graphics processing unit (GPU). As the core of modern graphics and parallel computing, the GPU is responsible for performing a large number of computational tasks in the 3D generative model.

[0157] The GPU is configured to perform the following operations during the diffusion processing phase:

[0158] The sensitivity level of the current processing viewpoint is determined based on at least one predefined sensitivity evaluation parameter associated with it: sensitivity level evaluation is crucial for optimization in the diffusion processing stage. Here, "sensitivity" refers to the viewpoint's sensitivity to the generated quality. The predefined sensitivity evaluation parameter can be various indicators within a spatial region associated with the current processing viewpoint. For example, if the number of Gaussian spheres in a region corresponding to a viewpoint is significantly higher than average, and the rendering loss is also high, then the viewpoint can be considered highly sensitive and requires more refined processing. This evaluation enables the system to adaptively adjust computational resources according to the importance of the viewpoint, thereby addressing the problems of huge computational overhead in the diffusion process (P1) and insufficient collaboration between different computing units within the GPU (P6). In practice, spatial region partitioning can employ various strategies, such as uniform partitioning, scene semantic-based partitioning, or importance-based sampling partitioning; the sensitivity index can also be calculated using various statistical methods, such as the mean, median, or weighted average.

[0159] Based on the determined sensitivity level, one or more computational configuration parameters for the neural network computation used in the diffusion processing stage are adjusted. These parameters include the number of computation time steps and the data processing precision. Dynamically adjusting the computational configuration according to the sensitivity level is one of the core strategies of this invention. For example, for a low-sensitivity perspective, the number of time steps in the diffusion process can be reduced, thereby reducing the computational load; simultaneously, the data processing precision can be reduced, for example, from FP16 to INT8, further reducing computational and storage overhead. For a high-sensitivity perspective, high precision and sufficient time steps are maintained to ensure generation quality. This adaptive adjustment effectively balances computational efficiency and generation quality. The reduction in time steps can be a fixed percentage, such as 30%-50%, or it can be dynamically adjusted based on the sensitivity level. Data precision reduction can employ various quantization techniques, such as linear quantization, nonlinear quantization, or mixed-precision quantization.

[0160] This approach leverages specific types of computing cores within the GPU to perform neural network computations based on one or more adjusted computational configuration parameters. When the data processing precision is adjusted to a first preset low precision, these specific computing cores are tensor cores. This fully utilizes the diverse computing resources of modern GPUs. Nvidia GPUs are equipped with both CUDA cores and tensor cores. Tensor cores are specifically designed for low-precision matrix operations, offering higher throughput and energy efficiency. When the data processing precision decreases to a first preset low precision, such as INT8, the system schedules tensor cores to perform neural network computations. For high-precision computations, either CUDA cores or high-precision-supporting tensor cores are used. This heterogeneous computing fully utilizes the advantages of different types of computing units to achieve optimal performance. In practice, programming models such as CUDA and OpenCL can be used, or specialized deep learning frameworks such as TensorFlow and PyTorch can be used to schedule different types of computing cores.

[0161] The GPU is also configured to perform the following operations during the Gaussian splash rendering phase:

[0162] Based on the rendering workload data recorded from at least one rendered pixel block, a predicted rendering workload for the pixel block to be processed is generated. To address the severe workload imbalance (P3) between SMs during the 3DGS rendering stage, this invention introduces a pixel block workload prediction mechanism. By recording the workload data of rendered pixel blocks, such as the number of Gaussian spheres processed or the number of SM execution cycles, the workload of the pixel block to be processed can be predicted. For example, assuming rendering is performed column-by-column, the workload of adjacent pixel blocks in the previous column can be used as the predicted workload of the current pixel block. This prediction provides a basis for subsequent task scheduling. In specific implementation, a Workload Prediction Table (WPT) can be used to store historical workload data, and a Comparison and Selection Unit (CSU) can be used to predict the workload of the pixel block to be processed.

[0163] Based on the predicted rendering workload and the availability of multiple Streaming Processors (SMs) within the GPU, pixel blocks to be processed are assigned to a selected SM for rendering: pixel blocks are allocated to appropriate SMs based on the predicted workload. When an SM is idle, the pixel block with the highest predicted workload is assigned to that SM. This prediction-based greedy scheduling strategy effectively balances the load among SMs, preventing some SMs from being overloaded while others are idle. In practice, a Pixel Block Queue (PTQ) can be used to cache pixel blocks to be processed, and an SM status monitoring interface can be used to obtain SM availability information.

[0164] Within a selected rendering SM (Screen Model), for a Gaussian sphere to be rendered within a pixel block to be processed, the geometric information of the Gaussian sphere and the geometric information of a group of rendering units within the pixel block to be processed are obtained. Each rendering unit corresponds to a thread bundle of the GPU. To reduce the problem of redundant rendering computation (P4) within the SM, this invention introduces a thread bundle rearrangement mechanism. For a Gaussian sphere to be rendered, its geometric information, such as its center coordinates, is first obtained, as well as the geometric information of a thread bundle corresponding to each rendering unit within the pixel block to be processed, such as the patch center coordinates.

[0165] Based on the geometric information of the Gaussian sphere and a set of rendering units, the execution order of rendering operations on the Gaussian sphere to be rendered is determined. This execution order prioritizes rendering units that are spatially closer to the geometric center of the Gaussian sphere. The distance between the Gaussian sphere and the rendering units is calculated based on their geometric information. Then, the rendering units are sorted in order of increasing distance. This sorting ensures that rendering units closer to the center of the Gaussian sphere are executed first, making alpha pruning easier.

[0166] According to a defined execution order, a group of rendering units are controlled to perform rendering operations on the Gaussian sphere to be rendered: the rendering units are scheduled to perform rendering operations in the sorted order.

[0167] In the backpropagation sub-stage of the Gaussian splash rendering phase, for multiple gradient components pointing to the same target Gaussian sphere generated by multiple threads in parallel computation within a single thread bundle: within the SM currently executing the thread bundle, the multiple gradient components are merged using the SM's built-in arithmetic logic resources to form a single aggregated gradient data; and the gradient state of the target Gaussian sphere is updated using the single aggregated gradient data through a single write operation to the GPU's shared cache for the target Gaussian sphere. To alleviate the significant latency (P5) introduced by atomic operations in 3DGS backpropagation, this invention introduces a gradient accumulation mechanism. During the backpropagation phase, multiple threads within a thread bundle may need to update the gradient of the same Gaussian sphere. Traditional methods involve each thread independently using atomic operations for updates, leading to numerous memory conflicts. This invention uses arithmetic logic resources (such as a parallel adder tree) within the SM to merge gradient components within the same thread bundle into a single aggregated gradient, and then performs only one atomic write operation, thereby significantly reducing memory conflicts.

[0168] Through the above technical solutions, this system can significantly improve the efficiency and throughput of 3D model generation and optimize GPU hardware resource utilization. Viewpoint sensitivity modulation and heterogeneous computing in the diffusion processing stage reduce computational load and memory bandwidth requirements, while fully utilizing the acceleration capabilities of the Tensor Core. Load balancing scheduling and thread bundle rearrangement in the Gaussian splash rendering stage improve the utilization of the SM (Spatial Modeling Tool) and reduce redundant computations. Gradient accumulation reduces the latency of atomic operations. These synergistic effects jointly optimize each key aspect of the 3D generation process.

[0169] In some embodiments, after determining the sensitivity level of the current processing viewpoint, the GPU is also configured to update predefined sensitivity evaluation parameters associated with the spatial region to which the current processing viewpoint belongs. This improves the accuracy of the viewpoint sensitivity evaluation by providing a feedback mechanism that adapts the viewpoint sensitivity evaluation to the model's training progress. Specifically, the predefined sensitivity evaluation parameters can take various forms, such as the number of Gaussian spheres associated with a region (N_region) and the region rendering loss value (L_region). Whenever a viewpoint is processed, the GPU updates the N_region and L_region of the region to which that viewpoint belongs. The N_region can be obtained by counting the number of Gaussian spheres actually involved in rendering during the rendering process. The L_region can be obtained by calculating the difference between the rendered result and the target image (such as the output of a diffusion model), such as mean squared error (MSE) or perceptual loss. Alternatively, other viewpoint-related metrics can be used, such as depth complexity in the viewing direction, color change gradient, etc. These predefined sensitivity evaluation parameters can be stored in on-chip SRAM or off-chip memory and maintained by hardware-accelerated update logic. In the hardware implementation, the update operation can be thread-safe through atomic operations or locking mechanisms.

[0170] Through this feedback mechanism, this technical feature enables more accurate assessment of viewpoint sensitivity and more rational allocation of GPU resources, thereby further improving the efficiency and quality of 3D generated models. For example, if the rendering quality of a certain region deteriorates (L_region increases) due to previous low-sensitivity processing, the system will automatically increase the sensitivity level of that region in subsequent iterations and allocate more computing resources to compensate for the quality loss.

[0171] In some embodiments, when the GPU adjusts one or more computational configuration parameters for neural network computation in the diffusion processing phase, if the determined sensitivity level indicates low sensitivity, the computation time steps are reduced to a first proportion of a baseline time steps, and the data processing precision is set to a first preset low precision. In this invention, the determination of the viewpoint sensitivity level is based on, for example, the number of Gaussian spheres in the region and the rendering loss, indicating the current viewpoint's focus on detail. Low sensitivity means that the viewpoint has lower detail requirements, and therefore a more aggressive computational configuration can be adopted to save computational resources.

[0172] By simultaneously reducing the number of computation time steps and lowering the data processing precision, the computational overhead of the diffusion processing stage in low-sensitivity perspectives can be significantly reduced, freeing up GPU resources to process higher-priority tasks.

[0173] In some embodiments, the time steps or the data precision can be adjusted only based on the viewpoint sensitivity level, or both can be adjusted simultaneously. For example, for a medium-sensitivity viewpoint, only the time steps can be reduced without reducing data precision. For a very low-sensitivity viewpoint, both the time steps and the data precision can be significantly reduced simultaneously. Furthermore, the proportion of time step reduction and the degree of data precision reduction can be dynamically adjusted based on the GPU's hardware characteristics and computational load.

[0174] By employing the above techniques, the computational overhead of the diffusion processing stage can be effectively reduced while ensuring generation quality, thereby improving the overall utilization and throughput of the GPU and accelerating the generation speed of 3D models. Specifically, by reducing the number of computation time steps and lowering the data processing precision, the computational load can be reduced by approximately 30%-70%, thus significantly accelerating the diffusion processing of low-sensitivity views and providing more resources for processing high-sensitivity views.

[0175] In some embodiments, during the Gaussian splash rendering stage, after a scheduled rendering unit completes the alpha contribution calculation for a Gaussian sphere to be rendered, if it is determined that the alpha contribution is lower than a preset pruning threshold, then for other rendering units in the same group that have not yet been scheduled and whose spatial proximity to the geometric center of the Gaussian sphere to be rendered is lower than that of the scheduled rendering unit, their rendering operations on the Gaussian sphere to be rendered are suspended. This aims to solve the problem of redundant calculations in the 3DGS rendering stage (P4). It utilizes the correlation between spatial proximity and alpha contribution. Since rendering units spatially closer to the center of the Gaussian sphere generally have a higher alpha contribution, if the alpha contribution of a nearby rendering unit is already very low, it can be inferred that the contribution of more distant rendering units to the Gaussian sphere is negligible, thus safely skipping the rendering calculations of these units.

[0176] In the implementation, this preset pruning threshold can be a fixed value or dynamically adjusted based on factors such as rendering quality and target frame rate. For example, the threshold can be set to 0.01, meaning that rendering units with an alpha contribution of less than 1% will be considered invalid. In the hardware implementation, the thread bundle rearrangement engine (B.2) inside SM queries the alpha pruning status interface before assigning each thread bundle (rendering unit) to check if the target Gaussian sphere has been marked as invalid by previously processed thread bundles. If so, the rendering of the current and subsequent thread bundles is skipped.

[0177] As an alternative measure of spatial proximity, the Euclidean distance (L2 norm), Manhattan distance (L1 norm), or other distance measures between the center of the Gaussian sphere and the center point of the pixel region covered by the rendering unit (thread bundle) can be used. Alternatively, the distance between the pixel coordinates of the rendering unit and the center point projected onto the image plane by the Gaussian sphere can also be used. The aforementioned "abortation of rendering operations on the Gaussian sphere" can be achieved by setting a flag to skip the corresponding alpha calculation and color blending code, or by directly exiting the execution of the thread bundle.

[0178] For example, in a specific implementation scenario, suppose a Render Stream (SM) is processing a Gaussian sphere within a pixel block. This Gaussian sphere is associated with eight rendering units (thread bundles) within the pixel block. The thread bundle reordering engine schedules these thread bundles sequentially according to their distance order. When the first thread bundle completes its alpha calculation, it finds that the alpha value is only 0.005, below the preset pruning threshold of 0.01. At this point, the Gaussian sphere is marked as pruned. Before scheduling subsequent thread bundles, the thread bundle reordering engine checks the pruning status of this Gaussian sphere. Since it has been pruned, all remaining seven thread bundles will be skipped, and redundant alpha calculations will no longer be performed.

[0179] Through the embodiments of the present invention, redundant alpha calculations for invalid Gaussian spheres can be significantly reduced, improving the computational efficiency of the internal thread bundle level of SM, thereby accelerating the overall rendering process.

[0180] This invention also provides a GPU co-processing system for accelerating 3D generative models. The GPU co-processing system of this embodiment is designed to execute a 3D generative model including a diffusion processing stage and a Gaussian splash rendering stage. The system is integrated on at least one graphics processing unit (GPU) and includes the following cooperating logical functional units or configurations:

[0181] Adaptive Diffusion Processing Unit: This unit is responsible for implementing adaptive computational resource allocation based on viewpoint characteristics during the diffusion processing stage of the 3D generated model. Specifically:

[0182] Viewpoint characteristic evaluation logic: This logic is configured to obtain computational difficulty evaluation parameters related to each new viewpoint before processing it. For example, it can query a table that maintains historical rendering statistics for each spatial region, such as the average Gaussian point density within the region and the average rendering loss of previous iterations, and compare the statistical data of the region to which the current viewpoint belongs with a preset global or neighborhood average complexity threshold, thereby determining whether the current viewpoint has high or low computational requirements.

[0183] The logic for dynamically setting computational parameters dynamically sets the number of iterations, such as time steps, and data representation precision, for the neural network computation in the current viewpoint diffusion processing stage based on the judgment result of the evaluation logic according to the viewpoint characteristics. For example, for viewpoints with low computational requirements, the number of time steps can be reduced by 30%-50%, and the data precision can be set to INT8; for viewpoints with high computational requirements, the standard number of time steps and FP16 precision are maintained.

[0184] Computation unit scheduling logic: This logic schedules appropriate computation units within the GPU based on the set data representation precision. When the data precision is set to INT8, the GPU's tensor cores are prioritized to perform the main matrix operations of the neural network; when the data precision is FP16, the CUDA cores or tensor cores that support FP16 are used for execution.

[0185] Collaborative Rendering Optimization Unit: This unit is responsible for implementing a series of collaborative optimizations aimed at improving GPU parallel processing efficiency during the Gaussian splash rendering stage of the 3D generated model, including:

[0186] SM-level task load balancer: This balancer contains a dynamic table structure or the number of Gaussian spheres processed to record the actual execution cycles of a certain number of recently processed pixel blocks. When a new pixel block task needs to be scheduled, it refers to the load data of processed pixel blocks that are spatially adjacent to the pixel block to be scheduled in this table to predict the execution cycle of the pixel block to be scheduled. Subsequently, it prioritizes assigning pixel blocks with longer predicted execution cycles to currently available or soon-to-be-idle stream processors (SMs) in order to even out the computational load across the SMs.

[0187] Rendering sorting and pruning logic within a SM: Within each SM, for each Gaussian object to be rendered, this logic first calculates the geometric distance between the Gaussian object and all interacting rendering units within the pixel block currently handled by the SM. Each rendering unit is processed by a thread bundle, corresponding to the geometric distance between the center point of the rendering region. Then, it schedules these thread bundles for rendering processing in order of increasing distance. Furthermore, if, after processing a relatively close thread bundle, the alpha value calculation determines that the Gaussian object's contribution to the current rendering region is lower than a preset pruning threshold, this logic controls all subsequent thread bundles that are further away to skip processing that Gaussian object.

[0188] Internal Gradient Hardware Aggregator within the SM: Within each SM, for the backpropagation sub-stage of Gaussian splash rendering, this aggregator utilizes an integrated array of parallel adders within the SM, such as an adder tree capable of processing 32 inputs simultaneously. Once all threads within a thread bundle have completed calculating their gradient components with respect to the same target Gaussian object, these gradient components are fed into the parallel adder array for rapid summation, forming a single aggregated gradient. This aggregated gradient is then updated to update the gradient state of the target Gaussian object through a single atomic write operation to the GPU's shared cache (such as L2 cache), significantly reducing the number of atomic operations.

[0189] Through the collaborative work of the above-mentioned units, the GPU co-processing system of this embodiment can effectively perform fine-grained resource allocation and computational optimization for different stages and characteristics of the 3D generation model, thereby achieving a significant acceleration effect.

[0190] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for accelerating the collaborative processing of graphics processing units to generate 3D models, wherein the 3D model includes a diffusion processing stage and a Gaussian splash rendering stage, characterized in that, The method includes the following steps: During the diffusion processing stage, the graphics processing unit performs the following operations: The sensitivity level of the current processing perspective is determined based on at least one predefined sensitivity assessment parameter associated with the current processing perspective. Based on the determined sensitivity level, one or more computational configuration parameters for the neural network computation used in the diffusion processing stage are adjusted, the computational configuration parameters including computation time steps and data processing precision; The neural network computation based on the adjusted one or more computation configuration parameters is performed using a specific type of computation core within the graphics processing unit. When the data processing precision is adjusted to a first preset low precision, the specific type of computation core is a tensor core. During the Gaussian splash rendering stage, the graphics processing unit performs the following operations: Based on the rendering workload data recorded by at least one rendered pixel block, generate the predicted rendering workload for the pixel block to be processed. Based on the predicted rendering workload and the availability of multiple stream processors within the graphics processing unit, the pixel block to be processed is assigned to a selected stream processor for rendering. Within the selected stream processor, for a Gaussian sphere to be rendered within the pixel block to be processed, the geometric information of the Gaussian sphere and the geometric information of a group of rendering units within the pixel block to be processed are obtained, each of the rendering units corresponding to a thread bundle of a graphics processing unit. Based on the geometric information of the Gaussian sphere and the geometric information of the group of rendering units, the execution order of the rendering operation of the group of rendering units on the Gaussian sphere to be rendered is determined, and the execution order makes the rendering unit that is closer to the geometric center of the Gaussian sphere to be rendered in space prioritized; according to the determined execution order, the group of rendering units is controlled to perform the rendering operation on the Gaussian sphere to be rendered. In the backpropagation sub-stage of the Gaussian splash rendering phase, multiple gradient components pointing to the same target Gaussian sphere are generated in parallel by multiple threads within a single thread bundle: Within the stream processor where the thread bundle is currently executing, the multiple gradient components are merged using the stream processor's built-in arithmetic logic resources to form a single aggregated gradient data. The gradient state of the target Gaussian sphere is updated using the single aggregated gradient data through a write operation initiated to the shared cache of the graphics processing unit for the target Gaussian sphere.

2. The method for collaborative processing of graphics processing units to accelerate the generation of 3D models according to claim 1, characterized in that, The predefined sensitivity evaluation parameters include: the number of Gaussian spheres accumulated in previous processing cycles and / or the accumulated rendering loss value associated with the spatial region to which the current processing viewpoint belongs.

3. The method for collaborative processing of graphics processing units to accelerate the generation of 3D models according to claim 1, characterized in that, The step of adjusting one or more computation configuration parameters for neural network computation in the diffusion processing stage includes: if the determined sensitivity level indicates low sensitivity, reducing the number of computation time steps to a first proportion of a baseline time step, and setting the data processing precision to the first preset low precision.

4. The method for collaborative processing of graphics processing units to accelerate the generation of 3D models according to claim 1, characterized in that, The step of assigning the pixel block to be processed to a selected stream processor for rendering processing based on the predicted rendering workload and the availability of multiple stream processors in the graphics processing unit includes: when the predicted rendering workload exceeds a predetermined threshold, assigning the pixel block to the stream processor that is the first to be converted to an idle state among all available stream processors.

5. The method for collaborative processing of graphics processing units to accelerate the generation of 3D models according to claim 1, characterized in that, After controlling the group of rendering units to perform the rendering operation on the Gaussian sphere to be rendered according to the determined execution order, the method further includes: If a scheduled rendering unit determines that its alpha contribution to the Gaussian sphere to be rendered is lower than a preset pruning threshold after completing its alpha contribution calculation, then for other rendering units in the group that have not yet been scheduled and whose spatial proximity to the geometric center of the Gaussian sphere to be rendered is lower than that of the scheduled rendering unit, their rendering operation on the Gaussian sphere to be rendered will be terminated.

6. A method for accelerating the collaborative processing of graphics processing units in a 3D generated model, wherein the 3D generated model includes a diffusion processing stage and a Gaussian splash rendering stage, characterized in that, The method includes the following steps: For the diffusion processing stage, adaptive computing resource allocation based on viewpoint characteristics is performed. The allocation includes dynamically setting the number of iterations and data representation precision for neural network computation in this stage based on the evaluation result of the computational difficulty of the current processing viewpoint, and scheduling computing units within the graphics processing unit that match the set data representation precision to perform the neural network computation. For the Gaussian splash rendering stage, collaborative optimization is performed with the goal of improving the parallel processing efficiency of the graphics processing unit. This collaborative optimization includes: Implement a stream processor-level task load balancing scheduling strategy based on historical rendering data to even out the computational load of each stream processor; Inside the stream processor, a thread bundle-level rendering task sorting mechanism based on geometric proximity is implemented to prioritize rendering interactions with high contribution probability and allow computations based on early evaluation results to be skipped. Inside the stream processor, a hardware-assisted aggregation mechanism for gradient updates is implemented to reduce the frequency of atomic operations on shared memory resources.

7. The method for collaborative processing of graphics processing units to accelerate the generation of 3D models according to claim 6, characterized in that, The evaluation results based on the computational difficulty of the current processing viewpoint include: comparing the rendering complexity index of the spatial region to which the current viewpoint belongs with the global or neighborhood average complexity index.

8. The method for collaborative processing of graphics processing units to accelerate the generation of 3D models according to claim 6, characterized in that, The stream processor-level task load balancing scheduling strategy includes: using a table structure that maintains the actual execution cycle of the most recently processed pixel blocks to predict the execution cycle of the pixel blocks to be scheduled, and allocating the pixel blocks with longer predicted execution cycles to the currently or soon-to-be-idle stream processors.

9. The method for collaborative processing of graphics processing units to accelerate the generation of 3D models according to claim 6, characterized in that, The hardware-assisted aggregation mechanism includes: using a parallel adder array within the stream processor to sum the gradients generated by all threads within a single thread bundle for the same Gaussian object, forming a summary gradient, and then updating the gradient of the Gaussian object through a single memory transaction.

10. A graphics processing unit collaborative processing system for accelerating 3D generated models, the 3D generated model comprising a diffusion processing stage and a Gaussian splash rendering stage, characterized in that, The system includes at least one graphics processing unit configured to implement the graphics processing unit collaborative processing method for accelerating three-dimensional generative models as described in any one of claims 1 to 5.

Images