A high-performance modular differentiable rendering method, apparatus, device, and storage medium based on DCU

Abstract

This invention relates to the field of computer graphics technology, specifically to a high-performance modular differentiable rendering method, apparatus, device, and storage medium based on a Digital Core Utility (DCU). The method includes: initializing a modular rendering pipeline; placing rendering data in DCU memory in tensor form; performing geometric transformations on vertex coordinates and outputting homogeneous coordinates, with delayed perspective division; generating pixel-level geometric information through a multi-layered pipeline of triangle setup, layered binning, coarse rasterization, and fine rasterization; and performing rendering and gradient backpropagation through a pluggable, differentiable modular rendering pipeline composed of rasterization, attribute interpolation, texture filtering, anti-aliasing, and shading modules. This invention deeply optimizes the domestic DCU architecture, solving the problems of difficult porting, poor non-modular scalability, and coarse gradient approximation in existing technologies. It achieves high-performance, high-precision differentiable rendering on domestic hardware, significantly improving rendering speed, system flexibility, and 3D reconstruction convergence.

Description

Technical Field

[0001] This invention relates to the field of computer graphics technology, and in particular to a high-performance modular differentiable rendering method, apparatus, device, and storage medium based on a DCU. Background Technology

[0002] In computer graphics and artificial intelligence, differentiable rendering technology can render a 3D scene as a 2D image while preserving the gradient information of scene parameters (such as geometry, materials, lighting, and camera pose) during the rendering process. This makes it possible to reverse engineer or optimize 3D scene parameters from a 2D image using optimization algorithms such as gradient descent. This technology has become core to fields such as reverse graphics, neural rendering, and 3D reconstruction.

[0003] Graphics processing units (GPUs), with their powerful parallel computing capabilities and high memory bandwidth, provide the hardware foundation for high-performance differentiable rendering. Traditional differentiable rendering solutions largely rely on mature general-purpose computing platforms such as NVIDIA CUDA. However, with the rapid development of domestically produced GPUs (such as DCUs), their importance in an independent and controllable ecosystem is becoming increasingly prominent. However, existing mainstream differentiable rendering methods are mostly designed and optimized based on the CUDA ecosystem, presenting challenges in terms of compatibility, performance tuning, and ecosystem support on domestically produced DCU platforms.

[0004] Currently, one of the most representative differentiable rendering solutions is the PyTorch 3D rendering library. As one of the existing technologies, its technical solution is as follows: leveraging the automatic differentiation mechanism of the PyTorch deep learning framework, it internally implements a simplified 3D rendering pipeline. This pipeline integrates rasterization and shading functions, and achieves differentiability of the rendering process based on triangle scan lines and gradient approximation algorithms.

[0005] However, after in-depth analysis, the existing technology has the following main drawbacks:

[0006] (1) Strong platform dependence and insufficient adaptation to domestic hardware: PyTorch3D is deeply coupled with the CUDA computing ecosystem, and its core computing kernel and memory model are designed for NVIDIA GPU architecture. This makes it impossible to run directly on domestic DCU platforms, the workload of porting and adaptation is large, and it is difficult to make full use of the unique hardware advantages of DCU (such as 64-bit wide thread bundles, specific synchronization primitives and memory hierarchy), and cannot unleash the full performance potential of domestic hardware.

[0007] (2) Monolithic architecture with poor scalability and flexibility: This technology adopts an integrated monolithic design, with high coupling between geometry processing, rasterization, shading and other links in the rendering pipeline. This tightly coupled architecture makes the system difficult to expand and cannot flexibly assemble or replace rendering modules according to different application requirements (such as whether complex texture filtering is required or whether anti-aliasing is enabled), which limits its applicability in the rapidly evolving research and application of neural rendering.

[0008] (3) Coarse gradient approximation limits optimization accuracy and stability: To address the non-differentiability problem caused by the discreteness in the rasterization process, this technique employs a simple gradient approximation method. Such approximations are often coarse and lack sensitivity to pixel coverage boundary regions, resulting in unsmooth reconstruction boundaries, slow convergence speed, and even difficulty in converging to the optimal solution when optimizing 3D geometric parameters, thus affecting the quality of gradient-based 3D reconstruction.

[0009] Therefore, there is an urgent need for a high-performance modular differentiable rendering solution based on DCU to solve the technical problems of existing technologies being unable to efficiently adapt to and utilize the performance of domestically produced DCU hardware, rigid rendering pipeline architecture lacking scalability, and insufficient gradient calculation accuracy affecting optimization results. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the prior art and provide a high-performance modular differentiable rendering method, device, equipment and storage medium based on DCU. By constructing a modular and pluggable rendering pipeline and deeply optimizing for DCU hardware characteristics, it achieves a significant improvement in rendering efficiency and accuracy, while supporting high-precision end-to-end gradient backpropagation, so as to better serve various reverse inference and generation tasks in the field of computer graphics technology.

[0011] To achieve the above objectives, this application proposes a high-performance modular differentiable rendering method based on DCU, comprising the following steps:

[0012] Step S1: Modular rendering pipeline initialization: Read the rendering data, copy the rendering data to the DCU's video memory through the automatic differentiation system, initialize the parameters of each rendering module on the CPU side, and set the mesh and block configuration for kernel function startup according to the characteristics of the DCU's computing unit.

[0013] Step S2, Geometric data preprocessing: The input three-dimensional vertex coordinates are processed by the automatic differentiation system to transform the original world coordinate vertices into homogeneous coordinates in clip space, and the perspective division operation is delayed.

[0014] Step S3, Layered rasterization processing: Receive the rendering data from step S2 and perform layered rasterization processing to provide pixel-level geometric information and processing targets for subsequent rendering stages; wherein, the layered rasterization processing includes four sub-stages in sequence: triangle setting, layered box rasterization, coarse rasterization, and fine rasterization.

[0015] Step S4, Modular rendering execution: Receive geometric information from step S3, and generate the final rendered image through a differentiable and pluggable dedicated processing module, supporting end-to-end backpropagation of gradients; wherein, the dedicated processing module includes at least one of a rasterization module, an attribute interpolation module, a texture filtering module, an anti-aliasing module, and a shading module.

[0016] As a further solution, step S1, which involves setting the grid and block configuration for kernel function startup based on the characteristics of the DCU's computing units, specifically includes:

[0017] The upper bound of the first grid size is determined based on the shared memory size required by the kernel function of each sub-stage of the hierarchical rasterization process and the maximum shared memory capacity of each computing unit of the DCU.

[0018] The upper bound of the second grid size is determined based on the thread count limit supported by each computing unit in the X and Y dimensions provided by the DCU runtime API.

[0019] The smaller value between the first grid size upper bound and the second grid size upper bound is taken as the final grid size startup configuration.

[0020] As a further solution, in step S2, the delayed perspective division specifically involves: outputting the homogeneous coordinates of the vertices during the geometric data preprocessing stage, and delaying the perspective division operation to the triangle setting stage in step S3.

[0021] It should be noted that in step S3, the pixel level is divided into sub-pixel level, pixel space, tile level and bin level.

[0022] As a further solution, the triangle setup in step S3 includes:

[0023] Receive the homogeneous coordinates of the three vertices that form the triangle in clip space;

[0024] Perform perspective division to convert vertex coordinates to normalized device coordinates;

[0025] The normalized device coordinates are transformed to screen space coordinates and converted to sub-pixel level representation. Based on the preset parameter subL, each pixel block is divided into subL*subL sub-pixel blocks.

[0026] Calculate the directed area of ​​the triangle and compare it with a preset threshold to perform degenerate triangle detection and back face removal;

[0027] Based on the depth values ​​of the three vertices, the plane equation coefficients used for depth interpolation are calculated.

[0028] As a further solution, the layered and boxed rasterization in step S3 includes:

[0029] According to preset parameters Divide the screen space into portions of size [size]. A grid area of ​​rectangular pixels;

[0030] For each valid triangle, compute its axially aligned bounding box in the mesh space;

[0031] For a triangle covering multiple grid regions, the exact intersection relationship between it and each grid region is determined by the side function test.

[0032] As a further solution, the bounding box consists of a minimum boundary. and maximum boundary If we represent the bounding box, then the boundary of the bounding box is:

[0033]

[0034]

[0035] in, , , These represent the coordinates of the triangle vertices in screen space; and These represent the corresponding edge vectors; , , These are the preset parameters.

[0036] As a further solution, the coarse rasterization in step S3 includes:

[0037] According to preset parameters The grid region is further divided into areas of size [size missing]. A pixel-sized tile;

[0038] Calculate the bounding box of the triangle in tile space;

[0039] Based on the bounding box size of the tile space, triangles are classified into small bounding box triangles, medium bounding box triangles, or large bounding box triangles, and different intersection testing strategies are adopted accordingly.

[0040] Filter out active tiles containing valid triangles and their associated triangle queues.

[0041] As a further solution, the classification of triangles based on bounding box size includes:

[0042] When the width and height of the bounding box are both less than or equal to 2 tile units, it is classified as a small bounding box triangle;

[0043] When the area of ​​the largest joint bounding box of all triangles within a thread bundle does not exceed twice the area of ​​a single triangle, it is classified as a medium bounding box triangle.

[0044] The remaining cases are classified as large bounding box triangles.

[0045] As a further solution, the different intersection testing strategies include:

[0046] For small bounding box triangles, the triangle index is directly written into the association list of the corresponding tile;

[0047] For a medium-sized bounding box triangle, first calculate the joint bounding box at the thread bundle level, and then determine the intersection relationship with each tile in parallel within the range of the joint bounding box;

[0048] For a large bounding box triangle, perform edge function tests on each tile level to determine the intersection relationship.

[0049] As a further solution, the fine rasterization in step S3 includes:

[0050] For each triangle within an active tile, a pre-computed overlay lookup table is used to quickly generate its precise overlay mask within an 8×8 pixel block;

[0051] Use a prefix sum scan algorithm to assign a segment index range to each triangle;

[0052] Perform depth testing and depth value updates on each generated fragment;

[0053] The color and depth data of the segments that pass the test are written to the global frame buffer.

[0054] As a further solution, the formula for generating the precise overlay mask is:

[0055] ;

[0056] Here, flag is the pre-computed edge function optimization flag. This is a lookup table-based coverage calculation function. , These are the coordinates of the triangle vertices in screen space. , and These represent the corresponding edge vectors; , , These are the pre-computed edge function optimization flags.

[0057] As a further solution, the lookup table-based coverage calculation function The pre-calculation is generated and stored in the constant memory or texture memory of the DCU; the pre-calculation includes: based on the preset line segment shape encoding and intercept parameters, the 8x8 pixel overlay mask under different edge function states is pre-calculated and stored offline.

[0058] As a further solution, the rasterization module, attribute interpolation module, texture filtering module, and anti-aliasing module all provide standardized forward data interfaces and backward gradient interfaces; the dedicated processing module is pluggably connected through the forward data interface and backward gradient interface, and its execution order can be dynamically arranged according to the rendering task requirements.

[0059] As a further solution, in the modular rendering execution stage of step S4, the intermediate data output by each dedicated processing module during the forward calculation process is temporarily stored as tensors in the DCU video memory; during the backward propagation process, the gradient signal is transmitted backward along the rendering pipeline, and the calculated parameter gradient is accumulated into the corresponding global gradient tensor by atomic addition operation.

[0060] As a further solution, the rasterization module includes a forward processing unit and a reverse processing unit;

[0061] For each valid pixel, the forward processing unit calculates the directed area vector and the perspective correction centroid coordinates based on its corresponding screen coordinates and the homogeneous coordinates of the vertices of the triangle to which it belongs. Normalized depth And the partial derivatives of the barycentric coordinates in screen space, the Jacobian matrix. ;

[0062] The reverse processing unit is used to receive gradient signals from the attribute interpolation module and calculate the gradient of the loss function with respect to the vertex position of the triangle based on the chain rule.

[0063] As a further solution, the forward processing unit processes the calculated centroid coordinates. Perform a clamping operation to confine it to the range. ; and / or the normalized depth Perform a clamping operation to confine it to the range. .

[0064] As a further solution, the attribute interpolation module includes a forward interpolation unit and a backward gradient unit;

[0065] The forward interpolation unit is used to calculate the centroid coordinates output by the rasterization module. Linear interpolation is performed on the vertex attributes to obtain pixel-level interpolated attributes. And calculate the partial derivative of the interpolation attribute in screen space;

[0066] The inverse gradient unit is used to receive feedback from subsequent modules on the interpolation attributes. The gradient signal, and the gradient signal according to the centroid coordinates The weights are backpropagated to the vertex attributes.

[0067] As a further solution, the texture filtering module includes a forward sampling unit and a reverse gradient unit;

[0068] The forward sampling unit is used to dynamically select the MIP level and perform trilinear interpolation sampling based on the texture coordinates and screen space derivatives output by the attribute interpolation module.

[0069] The inverse gradient unit is used to receive the gradient signal from the texture filtering module, distribute the gradient signal inversely to the texels participating in the interpolation according to the interpolation weights of the forward sampling, and calculate the gradient of the loss function with respect to the texture coordinates.

[0070] As a further solution, the anti-aliasing module includes a forward analysis unit and a backward gradient unit;

[0071] The forward analysis unit is used to identify visible discontinuous pixel pairs in the image caused by the contour edges of triangles, and to calculate the mixing weights. And mix the colors of adjacent pixels;

[0072] The inverse gradient unit is used to receive gradient signals in subsequent processing stages and calculate the loss function with respect to the mixed weights. The gradient is then calculated using the chain rule to determine the gradient at the vertex positions that constitute the contour edge.

[0073] As a further solution, the hybrid weights By calculating the intersection parameters of the line connecting the contour edge and the pixel center The intersection parameters Input a predefined differentiable saturated function The mixed weights are obtained. .

[0074] As a further solution, in the step of performing layered rasterization processing or modular rendering, thread bundle synchronization primitives of the DCU runtime are used for inter-thread communication and data consistency management; wherein, the synchronization primitives include at least one of any_sync, all_sync, or ballot_sync.

[0075] As a further solution, in the fine rasterization of step S3, the popcount primitive of the DCU runtime is used to count the number of bits that are 1 in the overlay mask to quickly obtain the number of pixels covered by the triangle.

[0076] On the other hand, the present invention also provides a high-performance modular differentiable rendering device based on a DCU, comprising:

[0077] Modular rendering pipeline initialization unit: used to read rendering data, copy the rendering data to the DCU's video memory through an automatic differentiation system, initialize the parameters of each rendering module on the CPU side, and set the mesh and block configuration for kernel function startup according to the characteristics of the DCU's computing unit.

[0078] Geometric data preprocessing unit: used to perform geometric transformation pipeline processing on the input three-dimensional vertex coordinates through the automatic differentiation system, transforming the original world coordinate vertices into homogeneous coordinates in clip space, and delaying perspective division operation;

[0079] Layered rasterization processing unit: used to receive rendering data from the geometry data preprocessing unit and perform layered rasterization processing to provide pixel-level geometric information and processing targets for subsequent rendering stages; wherein, the layered rasterization processing includes four sub-stages in sequence: triangle setting, layered box rasterization, coarse rasterization and fine rasterization.

[0080] Modular rendering execution unit: used to receive geometric information from the hierarchical rasterization processing unit, generate the final rendered image through differentiable and pluggable dedicated processing modules, and support end-to-end backpropagation of gradients; wherein, the dedicated processing modules include at least one of rasterization module, attribute interpolation module, texture filtering module, anti-aliasing module and shading module.

[0081] In another aspect, the present invention also provides a high-performance modular differentiable rendering device based on DCU, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement a high-performance modular differentiable rendering method based on DCU as described in any one of the above.

[0082] In another aspect, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a high-performance modular differentiable rendering method based on a DCU as described in any of the above.

[0083] Compared with related technologies, the high-performance modular differentiable rendering method, apparatus, device, and storage medium based on DCU provided by this invention have the following advantages:

[0084] 1. This invention addresses the core pain points of existing technologies (such as PyTorch3D) being deeply bound to the CUDA ecosystem, making porting them to domestic GPUs difficult and resulting in low performance. It has made a completely new design from the underlying computing model, memory access mode and parallel strategy. By deeply adapting to the unique 64-bit wide thread bundle, efficient synchronization primitives (such as any_sync) and hierarchical storage architecture of DCU, it has pioneered the realization of high performance and native support for differentiable rendering technology on domestic DCU hardware platforms.

[0085] Its technical advantages are as follows: it enables the direct and efficient operation of differentiable rendering functions on domestically produced DCUs, which not only solves the usability problem of "from scratch", but also fully releases the potential computing power of domestic hardware through a series of architecture-level optimizations (such as dynamic resource configuration and atomic operation conflict reduction). The rendering efficiency is improved by orders of magnitude compared with traditional porting solutions, laying the foundation for an independent and controllable graphics computing and AI integration ecosystem.

[0086] 2. This invention addresses the shortcomings of existing differentiable renderers, which are mostly "black box" monolithic architectures that are difficult to customize and expand. It decouples the complete rendering pipeline into a set of pluggable modules with well-defined functions and standard interfaces (such as rasterization, attribute interpolation, texture filtering, etc.). Each module independently performs forward computation and backward propagation and communicates through a standardized tensor interface.

[0087] The direct technical effect is that users can freely combine pipelines like building blocks according to specific tasks (such as those requiring only geometry optimization or those involving texture reconstruction). This design enables the system to have continuous evolution capabilities. New algorithms and optimizations can be easily integrated in the form of modules without reconstructing the entire system, which significantly extends the life cycle of the technical framework and reduces maintenance costs.

[0088] 3. This invention addresses the problem of excessive invalid computation in traditional rasterization when processing complex scenes. It adopts an original multi-level adaptive rasterization process, which achieves efficient spatial pruning through a three-layer processing mechanism of "Bin-Tile-Pixel" and a triangle adaptive classification and testing strategy based on bounding box size.

[0089] Its technical effects are reflected in the following aspects: through coarse-grained rapid screening, a large number of triangles and regions that do not intersect with pixels are effectively eliminated, and fine pixel-level calculations are concentrated on the truly needed local areas. This significantly reduces the computational load throughout the rasterization stage, especially when processing a large number of small triangles. The thread utilization and execution efficiency of the DCU are significantly optimized, thereby achieving a leap in rendering speed while ensuring accuracy.

[0090] 4. This invention addresses the problems of unstable optimization and rough reconstruction boundaries caused by gradient approximation in existing methods. It specifically implements precise gradient backpropagation based on analytical mathematical formulas in each core module. For example, the rasterization module directly calculates the Jacobian matrix of the centroid coordinates versus the screen coordinates, and the anti-aliasing module analytically calculates the gradient of the contour edge vertices by finding the intersection.

[0091] Its technical effects are reflected in the following aspects: the gradient transfer from image to scene parameters (geometry, material, etc.) is more accurate and stable; in inverse graphics tasks such as 3D reconstruction and material estimation, the convergence speed of the optimization process is faster, and the final model has richer details, smoother boundaries, and higher visual fidelity, solving the core problem of the rough effect of existing methods.

[0092] 5. This invention constructs an end-to-end, integrated automatic differential rendering workflow. By fully integrating the entire rendering process (from data input, intermediate calculations to gradient output) under the management of an automatic differential system (such as PyTorch), and with all data being DCU tensors, it greatly simplifies the development threshold of advanced vision applications.

[0093] Its technological advantages are reflected in the seamless integration of high-performance rendering into modern AI training frameworks. Users no longer need to manually implement complex gradient backpropagation; they only need to define forward rendering and loss functions, and the system can automatically perform optimization through differentiation. This significantly lowers the application threshold of differentiable rendering technology, allowing researchers and developers to focus more on upper-level algorithms and model design, and accelerating innovation and application in cutting-edge fields such as neural rendering and digital twins.

[0094] In summary, this invention not only successfully solves the technical challenge of achieving efficient differentiable rendering on a domestically produced DCU platform, but also provides a powerful foundational technology platform through its comprehensive advantages of modularity, high efficiency, high precision, and ease of use. This is of great value in promoting the cross-integration and development of independent and controllable computer vision, graphics, and artificial intelligence. Attached Figure Description

[0095] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0096] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0097] Figure 1 A schematic diagram illustrating the steps of a high-performance modular differentiable rendering method based on DCU provided by the present invention;

[0098] Figure 2 The forward and backward computation flowchart provided by this invention;

[0099] Figure 3 This is a schematic diagram illustrating how the modules provided by the present invention are arranged according to actual task requirements;

[0100] Figure 4 A schematic diagram of a high-performance modular differentiable rendering device based on DCU provided by the present invention;

[0101] Figure 5 A schematic diagram of a high-performance modular differentiable rendering device based on DCU provided by the present invention;

[0102] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0103] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0104] To better understand the technical solution of this embodiment, the abbreviations and key terms involved are defined as follows:

[0105] CUDA: Compute Unified Device Architecture, is a parallel computing platform and programming model developed by NVIDIA that allows developers to directly access the powerful computing capabilities of GPUs using C / C++ (as well as Python, Fortran, etc.) for general-purpose computing.

[0106] Differentiable rendering: refers to rendering a process that is differentiable with respect to scene parameters (such as geometry, materials, lighting, camera parameters, etc.), so that these parameters can be inferred or optimized from the image through optimization methods such as gradient descent.

[0107] Rasterization: refers to the process of mapping and filling 3D primitives (mainly triangles) composed of vertices onto screen pixels.

[0108] Tile-based: Divide the screen (or framebuffer) into several small rectangular regions (called tiles or bins), and then process the geometry and pixels within each tile block by block to optimize memory access, bandwidth and parallel computing efficiency.

[0109] The chain rule is a fundamental rule in calculus used to calculate the derivative of a composite function. In fields such as deep learning, machine learning, physics simulation, and differentiable rendering, the chain rule forms the mathematical basis for automatic differentiation and backpropagation.

[0110] Automatic differentiation mechanism: a technique for accurately and efficiently calculating the derivatives of functions in computer programs.

[0111] Automatic differentiation frameworks / systems are a class of software systems or libraries that support the automatic calculation of function gradients in programs. They achieve efficient and accurate gradient calculation for complex functions (such as neural networks, physics simulations, and differentiable renderers) by constructing computational graphs, tracing operations, and applying the chain rule. They are core infrastructures in modern AI, scientific computing, computer graphics, and other fields.

[0112] Thread bundle: also known as wavefront, is a basic scheduling and execution unit in DCU.

[0113] MIP Levels: Mipmap Levels refer to a set of pre-computed, progressively scaled-down texture images used for fast sampling at different drawing distances / scales, avoiding aliasing and improving caching efficiency.

[0114] block: A thread block refers to a larger unit of computation consisting of a group of threads.

[0115] Example 1

[0116] Please see Figure 1 This embodiment achieves accurate characterization and dynamic optimization of users' true query intent in the specific semantic space of education through a three-ring collaborative mechanism of data construction, model adaptation and reinforcement optimization, thereby significantly improving the generalization, accuracy and robustness of the educational resource and knowledge question answering retrieval system in real teaching deployment scenarios;

[0117] A high-performance modular differentiable rendering method based on DCU includes the following steps:

[0118] Step S1: Modular rendering pipeline initialization: Read the rendering data, copy the rendering data to the DCU's video memory through the automatic differentiation system, initialize the parameters of each rendering module on the CPU side, and set the mesh and block configuration for kernel function startup according to the DCU's computing unit characteristics; wherein, the rendering data refers to the geometric vertices and surface colors that constitute the rendering scene.

[0119] Step S2, Geometric Data Preprocessing: Receive the rendering data copied to the DCU during the modular rendering pipeline initialization phase in Step S, and perform geometric transformation pipeline processing on the input 3D vertex coordinates through the automatic differentiation system to transform the original world coordinate vertices into homogeneous coordinates in clip space, and delay the perspective division operation.

[0120] Step S3, Layered rasterization processing: Receive the rendering data from step S2 and perform layered rasterization processing to provide pixel-level geometric information and processing targets for subsequent rendering stages; wherein, the layered rasterization processing includes four sub-stages in sequence: triangle setting, layered box rasterization, coarse rasterization, and fine rasterization.

[0121] Step S4, Modular rendering execution: Receive geometric information from step S3, and generate the final rendered image through a differentiable and pluggable dedicated processing module, supporting end-to-end backpropagation of gradients; wherein, the dedicated processing module includes at least one of a rasterization module, an attribute interpolation module, a texture filtering module, an anti-aliasing module, and a shading module.

[0122] It should be noted that this embodiment provides a high-performance, modular, differentiable rendering method specifically customized for domestically produced DCU (Depth Computing Unit) hardware. Its core lies in reshaping the traditional graphics rendering pipeline into a differentiable computation pipeline that is deeply integrated with an automatic differentiation system and supports gradient backpropagation.

[0123] This method first prepares scene data as tensors in DCU memory through modular initialization and geometric preprocessing. Then, it uses a multi-level rasterization process that includes triangle setting, layer binning, coarse rasterization and fine rasterization to efficiently generate pixel-level geometric information. Finally, through an execution stage consisting of multiple pluggable and differentiable dedicated modules (such as rasterization, attribute interpolation, texture filtering, etc.), it completes the rendering from geometry to the final image, and uses an automatic differentiation system to realize end-to-end gradient calculation and optimization from image pixels to scene parameters (such as geometry and materials).

[0124] This method addresses three key challenges of porting existing technologies (such as PyTorch3D) to domestic DCU platforms: insufficient performance optimization and high architectural coupling with coarse gradient approximations. Its technical benefits are: 1) High-performance execution on domestic hardware is achieved through modular design and deep adaptation to DCU's native programming features (such as thread bundles and synchronization primitives); 2) The flexibility and scalability of the rendering system are significantly improved through pluggable modular design; 3) Smoother rendering boundaries and better optimization convergence are achieved through analytical or chain-law-based precise gradient calculation (rather than simple approximations), providing efficient underlying support for AI graphics applications such as 3D reconstruction and virtual humans that require inferring 3D parameters from images.

[0125] Furthermore, in step S1, setting the mesh and block configuration for kernel function startup based on the characteristics of the DCU's computing units specifically includes:

[0126] The upper bound of the first grid size is determined based on the shared memory size required by the kernel function of each sub-stage of the hierarchical rasterization process and the maximum shared memory capacity of each computing unit of the DCU.

[0127] The upper bound of the second grid size is determined based on the thread count limit supported by each computing unit in the X and Y dimensions provided by the DCU runtime API.

[0128] The smaller value between the first grid size upper bound and the second grid size upper bound is taken as the final grid size startup configuration.

[0129] Specifically, this embodiment proposes an adaptive automatic configuration method for kernel function parameters based on dual constraints of DCU hardware resources. This method does not empirically preset fixed parameters, but rather utilizes two parallel computation paths:

[0130] First, based on the shared memory size required by the kernel function and the maximum shared memory capacity of each computing unit in the DCU, a first upper bound on the grid size, constrained by memory resources, is calculated. Second, by querying the DCU runtime API, the physical limitations on the number of threads in each dimension are obtained, and a second upper bound on the grid size, constrained by parallel thread resources, is calculated. Finally, the system takes the smaller of these two upper bounds as the actual startup configuration for the kernel function.

[0131] This design addresses the performance bottleneck caused by improper parameter configuration in heterogeneous computing. Its technical benefits are twofold: firstly, by constraining shared memory capacity, it ensures that concurrently residing thread blocks on each computing unit do not become blocked or overflow due to contention for limited on-chip high-speed storage resources, thus optimizing memory access efficiency.

[0132] On the other hand, by constraining the number of threads, it is ensured that the scheduling system will not attempt to launch threads beyond the capacity of the hardware physical execution unit, thus maximizing hardware utilization. This dynamic, hardware-feedback-based configuration mechanism enables the rendering pipeline of this invention to intelligently adapt to different models or configurations of DCUs, which is a key underlying guarantee for achieving the goals of "high performance" and "platform adaptability".

[0133] In one specific embodiment, before starting the kernel function, in order to ensure that the computing resources of the DCU are fully utilized, the maximum number of blocks that each computing unit of the DCU can reside in at the same time is calculated based on the allocated computing resources, such as the size of shared memory, and this number of blocks is set as the grid's startup configuration.

[0134] Here's an example of setting the block size of the kernel function in the triangle setup phase (S3.1). That is, the block's The dimension is 64, which is the size of the thread bundle on a DCU. The dimension is 16, meaning each block contains a total of 16 thread bundles. The dimension is 1. And the calculated centroid coordinates are stored in shared memory during the triangle setting phase (S3.1). Therefore, each block contains a total of 64*16*1*12*4 bytes. Here, 12 means that each thread needs a maximum of 12 centroid coordinate component storage units, and 4 means that each 32-bit number needs 4 bytes.

[0135] The upper bound of the grid size based on the shared memory size can be determined by dividing the maximum shared memory capacity per compute unit of the DCU by the shared memory size required by each kernel function. Similarly, each compute unit of the DCU supports... , and The number of threads per dimension also has physical limitations. By calling the DCU runtime API, an upper bound for the mesh size can be obtained based on the current DCU thread count limit. The smaller of the two upper bounds determines the mesh size. After determining the mesh size, it can be configured to start the kernel function for the layered rasterization processing stage, and the kernel function can be started to execute the layered rasterization.

[0136] Furthermore, in step S2, the delayed perspective division specifically involves: outputting the homogeneous coordinates of the vertices during the geometric data preprocessing stage, and delaying the perspective division operation to the triangle setting stage in step S3.

[0137] It should be noted that this invention does not perform perspective division at this stage, thus maintaining the homogeneous coordinate form. This design has two advantages: First, it reduces the computational overhead of division operations, improving the efficiency of the preprocessing stage. Second, it postpones perspective division to the triangle setup step in the rasterization stage, aligning with the standard workflow of modern graphics hardware pipelines.

[0138] Furthermore, the present invention provides a triangle setting method, configured to receive triangle vertex data from a geometric data preprocessing stage (S2) to prepare for subsequent layered rasterization. The triangle setting in step S3 includes:

[0139] Receive the homogeneous coordinates of the three vertices of the triangle in clip space. ;

[0140] Perform perspective division to convert vertex coordinates to normalized device coordinates;

[0141]

[0142] in, , These are the perspective-corrected 2D screen projection coordinates, corresponding to the left, right, top, and bottom boundaries of the screen. The normalized depth value; The homogeneous coordinate projection factor;

[0143] The normalized device coordinates are transformed to screen space coordinates and converted to sub-pixel level representation. Based on the preset parameter subL, each pixel block is divided into subL*subL sub-pixel blocks.

[0144] Calculate the directed area of ​​the triangle and compare it with a preset threshold to perform degenerate triangle detection and back face removal;

[0145] Let the screen space coordinates of the three vertices of the triangle be respectively... , , Calculate the directed area of ​​a triangle. Used for degenerate triangle detection and back face rejection:

[0146]

[0147] Based on the depth values ​​of the three vertices, the coefficients of the plane equation used for depth interpolation are calculated, and the depth plane equation is expressed as:

[0148]

[0149] in, This represents the coordinates in screen space as... The pixel represents the depth value of the corresponding triangle surface point; A, B, and C are three constant coefficients that uniquely define a depth value in screen space. The inclined plane on the plane, that is, the plane in which the depth value of the current triangle is distributed in screen space.

[0150] In a more specific embodiment, the implementation of the triangle setting phase is as follows:

[0151] Based on the block configuration, the global index of the current thread is calculated. Each thread reads the vertex data of a triangle based on its thread index. First, perspective division is performed to convert the vertex coordinates to normalized device coordinates. Then, the value of subL is set to 4 to transform the normalized device coordinates to screen space. Assuming the output resolution is 256*256, the specific transformation steps are as follows: , The screen space coordinates here are Normalized device coordinates are .

[0152] Next, the directed area of ​​the triangle is calculated using the formula in S3.1. When the area is less than a threshold... At that point, the triangle was deemed necessary to be culled. Finally, the linear equations in S3.1 were solved using the screen coordinates of the triangle's three vertices and the normalized device coordinate depth values ​​to establish the depth plane equation, thus determining the variation of the depth value within screen space. The coefficients of the depth plane equation and other geometric information were written to the output buffer in global memory.

[0153] The aforementioned triangle setting method provides the geometric basis for subsequent rasterization by performing perspective division and screen space transformation on the clipping space coordinates; and by calculating the directed area and depth plane equations, it achieves early culling of invalid triangles and pre-calculation of depth information. The output of this stage contains a data structure containing screen space vertex coordinates, depth interpolation parameters, and triangle validity indicators, which will be used in the subsequent layered and boxed rasterization stage.

[0154] Specifically, the calculation steps described in this stage utilize an automatic differentiation system adapted to the DCU. Data is transferred to the DCU's video memory, and the calculation process occurs on the DCU. Each thread in the thread bundle is responsible for processing a triangle and the corresponding calculation steps.

[0155] Furthermore, this embodiment provides a hierarchical boxed rasterization method, configured to receive valid triangle data from the triangle setting stage, construct its index in screen space, wherein the meshes are all boxed meshes, and the hierarchical boxed rasterization method includes the following steps:

[0156] According to preset parameters and Divide the screen space into portions of size [size]. A rectangular pixel grid area, that is, each grid is composed of... Composed of pixel blocks;

[0157] For each valid triangle, compute its axially aligned bounding box in the mesh space;

[0158] For a triangle covering multiple grid regions, the exact intersection relationship between it and each grid region is determined by the side function test.

[0159] Specifically, the bounding box consists of a minimum boundary. and maximum boundary If we represent the bounding box, then the boundary of the bounding box is:

[0160]

[0161]

[0162] in, , , These represent the coordinates of the triangle vertices in screen space; and These represent the corresponding edge vectors; , , These are the preset parameters.

[0163] It should be noted that the bounding box defines the range of the mesh area that the triangle may cover, which limits the calculation range of subsequent intersection tests and avoids unnecessary computational overhead by traversing all mesh areas.

[0164] In a more specific embodiment, the implementation of layered and boxed rasterization is as follows:

[0165] First take The screen space is divided into rectangular grid areas, with each grid cell measuring 16*16 pixels. The current thread reads the triangle it is processing from the output buffer in S3.1 and calculates the bounding box of the current triangle in the grid space using the formula in S3.2. The value of is 3. The bounding box of the triangle is calculated. and Next, determine the intersection relationship between the triangle and each grid.

[0166] First, a quick check is performed, using the `__any_sync` primitive of the DCU runtime to determine whether the triangle of each thread in the current thread bundle satisfies the condition. equal or equal This process determines whether all triangles in the current thread bundle intersect with the current mesh. If not, it iterates through each mesh and uses the side function formula in S3.2 to determine the intersection relationship between the triangles held by the current thread and the mesh. All triangles that satisfy the intersection relationship are selected as candidate triangles, and their indices are written to the display memory that records the relationship between the mesh and the triangles.

[0167] Furthermore, for triangles covering multiple grid regions, edge function tests are used to determine their precise intersection relationships with each grid region; let the sampling points within the grid be... Calculate the directed distances of the point relative to the three sides of the triangle. :

[0168]

[0169] in, , This represents the cross product operation of two-dimensional vectors. When When all components are non-negative, the grid region is determined to intersect with the triangle;

[0170] The hierarchical and boxed rasterization method divides the screen space into rectangular grids and calculates the bounding boxes of triangles, enabling rapid filtering of the intersection relationships between triangles and grids, and establishing an accurate spatial index through edge function testing.

[0171] The output of this stage includes the mesh and triangle relationships, specifically a list of triangle indices and a preferred index mapping table for the mesh regions. Similarly, thanks to the automatic differentiation system, this stage is executed on the DCU, with each thread bundle handling a batch of triangles. Synchronization primitives within the DCU runtime reduce inter-thread communication latency, indirectly improving computational efficiency.

[0172] Furthermore, this embodiment provides a coarse rasterization method configured to receive mesh-triangle association data from the hierarchical binning rasterization stage (S3.2), construct a tile-level spatial index, wherein the coarse rasterization in step S3 includes:

[0173] According to preset parameters The grid region is further divided into areas of size [size missing]. A pixel-sized tile;

[0174] Calculate the bounding box of the triangle in tile space using the same formula as in the layered and boxed rasterization step. , It should be noted that the denominator is ;

[0175] Based on the bounding box size of the tile space, triangles are classified into small bounding box triangles, medium bounding box triangles, or large bounding box triangles, and different intersection testing strategies are adopted accordingly.

[0176] Specifically, when the width and height of the bounding box are both less than or equal to 2, it is classified as a small bounding box triangle; when the maximum bounding box area of ​​all triangles within the thread bundle does not exceed twice the area of ​​a single triangle, it is classified as a medium bounding box; the rest are large bounding box triangles.

[0177] Depending on the type of triangle, this step uses different intersection testing strategies:

[0178] For small bounding box triangles, the triangle index is directly written into the mask of the corresponding tile. For medium bounding box triangles, the joint bounding box at the thread bundle level is first calculated, and then the intersection state of each tile is determined in parallel within this range using the thread bundle voting mechanism of the DCU runtime. For large bounding boxes, edge function testing is performed tile by tile.

[0179] Filter out active tiles containing valid triangles and their associated triangle queues.

[0180] Specifically, the prefix sum and scanning algorithm are used to count the number of triangles within each tile, and the triangle indices are written to the corresponding storage segments according to the positions calculated by the prefix sum. Meanwhile, tiles containing valid triangles are marked as active tiles.

[0181] In a more specific embodiment, the implementation of coarse rasterization is as follows:

[0182] Similarly, according to the settings If the value is 3, the tile size is an 8x8 pixel block. Therefore, each grid is divided into 8x8 regions. The bounding box of the triangle in the tile space is calculated according to the formula in S3.3. The value is 3. The value is 4. Next, determine the triangle intersection relationship at the tile level.

[0183] Since the size of the output bounding box is in units of tiles, the __all_sync primitive used during DCU runtime first checks whether the width and height of the bounding boxes of all triangles in the current thread bundle are not greater than 2. If the condition is met, the triangles to which the bounding box belongs are considered to have an intersection relationship, and all triangles in the thread bundle are marked as valid triangles.

[0184] For bounding boxes that do not satisfy the condition that their width and height are both less than 2, further determine the intersection relationship. Merge the bounding boxes of all triangles in the thread bundle into a thread bundle-level bounding box. If the bounding box is not greater than twice the area of ​​any triangle in the thread bundle, traverse each tile in the bounding box. Using the __ballot_sync primitive of the DCU runtime, calculate all triangles in the thread bundle that intersect with the current tile, with the non-negativity of the edge function value as the judgment condition. These triangles are recorded as valid triangles in the tile triangle relationship.

[0185] Next, for larger bounding boxes, each thread traverses the tiles and determines intersection relationships based on the non-negativity of the edge function value. After completing the intersection relationship tests for all tiles and triangles, the active tile index containing valid triangles and the associated triangle indexes are written into video memory.

[0186] The coarse rasterization method provided in this implementation achieves fine-grained filtering of triangle coverage by further subdividing tiles within the mesh and calculating tile bounding boxes; and reduces the computational load of subsequent fine rasterization stages through an adaptive intersection testing strategy and active tile filtering. This stage outputs the relationship between tiles and triangles. Similarly, with the help of an automatic differentiation system, this step is executed on the DCU.

[0187] Furthermore, the present invention provides a fine rasterization method configured to receive active patches and their associated triangle queues from the coarse rasterization stage (S3.3) and perform a final pixel-level coverage calculation. The fine rasterization method includes the following steps:

[0188] For each triangle within an active tile, a pre-computed overlay lookup table is used to quickly generate its precise overlay mask within an 8×8 pixel block;

[0189] Let the vertex of the triangle be... , , The edge vectors are respectively , and Using a pre-computed overlay lookup table and flip flags, a 64-bit overlay mask can be quickly generated.

[0190] Specifically, the formula for generating an exact overlay mask is:

[0191] ;

[0192] in, Optimize the flag bits for pre-computed edge functions. This is a lookup table-based coverage calculation function, where each bit of the coverage mask corresponds to the coverage state of a pixel within a tile. , These are the coordinates of the triangle vertices in screen space. , and These represent the corresponding edge vectors; , , These are the pre-computed edge function optimization flags.

[0193] Use a prefix sum scan algorithm to assign a segment index range to each triangle;

[0194] Specifically, by calling the DCU runtime The interface counts the number of pixels covered by the mask. Then, a prefix sum scan algorithm is used to assign a segment index range to each triangle:

[0195]

[0196] in, This represents the number of pixels covered by the *th triangle. This represents the starting index of the *th triangle segment in the global segment array. This represents the ending index (excluding) of the *th triangle segment in the global segment array.

[0197] It should be noted that a fragment refers to the effective coverage of a certain pixel by a triangle within a tile.

[0198] Perform depth testing and depth value updates on each generated fragment;

[0199] Specifically, let the pixel coordinates be... The depth plane coefficient of the triangle is The interpolation depth is then:

[0200]

[0201] in, Indicates the interpolation depth.

[0202] Based on the interpolation depth, a maximum depth value for the tile will be dynamically maintained in this stage. Compare the minimum depth of the triangle and If the value is greater than the specified value, it will be directly discarded to avoid unnecessary overwrite calculations. The maximum depth is recalculated via thread bundle reduction during each depth buffer update:

[0203]

[0204] For segments that pass the depth test, an atomic operation sequence is used to ensure correct depth updates and triangle index writes: first, the depth buffer is updated using the atomic minimum operation provided by the DCU runtime; then, the winning thread is determined by thread bundle voting; and finally, the winning thread performs the triangle index write.

[0205] After processing all triangles within the tile, the color and depth data of the tested fragments are written to the global frame buffer.

[0206] In the process of refining the rasterization, this embodiment utilizes a lookup table to quickly generate 8 An 8-patch overlay mask enables highly efficient pixel-by-pixel overlay calculation; and through a prefix sum scanning algorithm and depth testing, it provides pixel space geometric information and processing targets for the subsequent modular rendering execution stage. The geometric information includes the index of the triangle covering each pixel and the shallowest depth of each pixel.

[0207] In a more specific embodiment, a fine rasterization implementation is described. This embodiment first calculates the precise overlay mask for an 8x8 tile. First, based on preset line segment shape encoding and y-intercept parameters, the range of pixel rows covered by the boundary lines in each column is calculated by scanning column by column. Then, the pixel coordinates are transformed according to whether the x-axis and y-axis coordinates are swapped, the x-axis direction is reversed, or inverted. Finally, the overlay result is encoded into a 64-bit mask and written into a lookup table. Then, the overlay mask is calculated according to the formula in S4.4.

[0208] The `__popcount` primitive used by the DCU runtime is used to calculate the number of overlaid bits in the mask. Then, the fragment index range for each triangle is calculated based on a prefix sum scan algorithm. The fragments are traversed, and the interpolated depth of each pixel is calculated according to the depth plane equation obtained in S3.1. Triangles with depth values ​​greater than the maximum allowed depth value are discarded. For fragments that pass the test, the depth value is updated and the color is written. After all fragments have been processed, the final color and depth values ​​are written to the video memory where the global frame resides.

[0209] Furthermore, this embodiment also includes the cover calculation function based on the lookup table. The pre-calculation is generated and stored in the constant memory or texture memory of the DCU; the pre-calculation includes: based on the preset line segment shape encoding and intercept parameters, the 8x8 pixel overlay mask under different edge function states is pre-calculated and stored offline.

[0210] Furthermore, the modular rendering execution stage of the present invention is configured to receive geometric information from the layered rasterization processing stage (S3) and generate the final rendered image through a series of interconnected dedicated processing modules, while supporting end-to-end backpropagation of gradients.

[0211] Furthermore, in the modular rendering execution stage of step S4, the intermediate data output by each dedicated processing module during the forward calculation process is temporarily stored as tensors in the DCU video memory; during the backward propagation process, the gradient signal is transmitted backward along the rendering pipeline, and the calculated parameter gradient is accumulated into the global gradient tensor of the corresponding candidate triangle by atomic addition operation.

[0212] This phase includes the following core modules:

[0213] S4.1: Rasterization Module

[0214] This embodiment provides a rasterization module configured to connect a fine rasterization stage (S3.4) and an attribute interpolation module (S4.2) for converting pixel-level triangle association information into differentiable geometric basis data.

[0215] The rasterization module includes a forward processing unit and a reverse processing unit; wherein...

[0216] For each valid pixel, the forward processing unit calculates the directed area vector and the perspective correction centroid coordinates based on its corresponding screen coordinates and the homogeneous coordinates of the vertices of the triangle to which it belongs. Normalized depth And the partial derivatives of the barycentric coordinates in screen space, the Jacobian matrix. ;

[0217] The reverse processing unit is used to receive gradient signals from the attribute interpolation module and calculate the gradient of the loss function with respect to the vertex position of the triangle based on the chain rule.

[0218] Specifically, during forward processing: the forward processing unit receives the pixel and triangle association information from the fine rasterization stage. For each valid pixel, it is based on screen coordinates. Homogeneous coordinates of the vertices of the triangle , , Calculate the directed area vector Each component The calculation method is as follows:

[0219] Here, Indicates the vertex The offset vector after the homogeneous coordinates are projected onto the current pixel position. This represents the cross product operation of two-dimensional vectors. for A cyclical arrangement.

[0220] Based on directed area vector Calculate the centroid coordinates for perspective correction. and normalized depth :

[0221]

[0222] To ensure numerical stability, Non-negative clamping is applied to the component, and the component is... The components are range-limited. Simultaneously, the partial derivative Jacobian matrix of the barycenter coordinates in screen space is calculated. Its elements are analytically derived from the derivative of the area vector and the coordinate scaling factor. The output includes... The geometric property tensor, consisting of the indices of the triangles to which the pixels belong, and the Jacobian matrix, which represents the partial derivatives of the barycenter coordinates in screen space. .

[0223] More specifically, during the reverse processing: the reverse processing unit receives a gradient signal from its associated attribute interpolation module (S4.2), the gradient signal including the loss function with respect to the centroid coordinates. The partial derivatives and the Jacobian matrix of the partial derivatives obtained in the forward processing stage The partial derivatives.

[0224] It uses the chain rule and intermediate variables from the forward computation to analytically calculate the position of each vertex. The gradient. This calculation process first uses the received gradient signal and the normalization factor calculated forward. The intermediate gradient variables are generated, and then, by combining the vertex coordinate difference, area derivative, and the forward-calculated centroid coordinates, the result is calculated via vector combination. The gradient on the x, y, w components.

[0225] In one specific embodiment, the rasterization module is implemented as follows:

[0226] The specific implementation of forward processing is as follows: Each thread calculates the linear index of the corresponding pixel, retrieves the index of the corresponding triangle based on the index, and determines the validity of the triangle based on the number of triangles. If the triangle corresponding to the current pixel is invalid, the buffer of the corresponding pixel is written with all zeros. Otherwise, the vertex coordinates of the triangle are read, and the directed area vector is first calculated according to the formula shown in S4.1. Then calculate the centroid coordinates for perspective correction. and normalized depth The calculated barycenter coordinates are clamped to a closed interval. In this case, the normalized depth value is constrained to the closed interval. Then, the values ​​are combined with the triangle index to form a quadruple and written to the corresponding pixel's video memory. The resulting centroid coordinates are then calculated. Partial derivative in screen space Then it can be stored in the video memory.

[0227] The specific implementation of the reverse process is as follows: Each thread calculates the linear index of the corresponding pixel, and reads the derivative information transmitted by the automatic differentiation system based on the index. and Next, the coordinates of the vertices of the triangle are read and the normalized centroid coordinates are calculated according to the same formula in the forward process. The derivative of the current pixel with respect to the three vertices is calculated according to the chain rule and accumulated into the gradient buffer through atomic operations.

[0228] In this embodiment, the rasterization module achieves efficient differentiable transformation from triangular primitives to pixel attributes by vectorizing the calculation of directed area and analytical derivative. It also accumulates the calculated vertex gradients into the global gradient buffer through atomic summation operations, ensuring the correct connection between the front-end module and the back-end attribute interpolation module.

[0229] S4.2, Attribute Interpolation Module:

[0230] This invention provides an attribute interpolation module configured to connect a rasterization module (S4.1) with a subsequent texture filtering module (S4.3) or a shading module (S4.5), used to perform pixel-level interpolation of vertex attributes based on the geometric information generated by the rasterization module. The attribute interpolation module includes a forward interpolation unit and a backward gradient unit.

[0231] The attribute interpolation module includes a forward interpolation unit and a backward gradient unit; wherein...

[0232] The forward interpolation unit is used to calculate the centroid coordinates output by the rasterization module. Linear interpolation is performed on the vertex attributes to obtain pixel-level interpolated attributes. And calculate the partial derivative of the interpolation attribute in screen space;

[0233] The inverse gradient unit is used to receive feedback from subsequent modules on the interpolation attributes. The gradient signal, and the gradient signal according to the centroid coordinates The weights are backpropagated to the vertex attributes.

[0234] Specifically, during forward interpolation: the forward interpolation unit receives the output from the rasterization module (S4.1), the output containing the centroid coordinates of the current pixel. And a triangle identifier. For each valid pixel, based on the triangle identifier, the attribute vectors of the three vertices constituting the triangle are obtained. , , ,in This is the attribute dimension. Based on the centroid coordinates, the interpolation attribute of the pixel is calculated. :

[0235]

[0236] This calculation process applies to attribute vectors Each component is executed in parallel, generating pixel-level attribute tensors. When derivative propagation is enabled, the forward interpolation unit also receives screen-space derivative information from its associated rasterization module (S4.1). It further calculates the selected attributes. Partial derivative in screen space:

[0237]

[0238] in The gradient of the attribute in the centroid coordinate space is the output of the forward processing stage, which is a pixel-level attribute map, that is, the attribute interpolation result and the gradient of the attribute in the centroid coordinate space for each pixel.

[0239] More specifically, during the inverse gradient process: the inverse gradient unit receives gradient signals from the subsequent texture filtering module (S4.3), the gradient signals including interpolation attributes The partial derivative of the gradient is used to backpropagate the gradient to the attribute vectors of the three vertices using the chain rule. Specifically, the gradient increment of the vertex attribute is calculated:

[0240]

[0241] in For the loss function to interpolation attributes The partial derivative. When derivative gradient propagation is initiated, the inverse gradient unit also receives the gradient of the attribute screen space derivative and backpropagates it to the derivative outputs of the vertex attributes and rasterization modules.

[0242] In a more specific embodiment, the attribute interpolation module is implemented as follows:

[0243] The specific implementation of forward interpolation is as follows: Each thread reads the linear index of the corresponding pixel, and based on the pixel index, reads the centroid coordinates and triangle identifier of the pixel output by the rasterization module. If the triangle identifier is -1, the current triangle is invalid and will not participate in subsequent calculations. The validity of all triangles in the current thread bundle is determined by the __all_sync primitive of the DCU runtime. If no valid triangle exists, the process returns directly. Otherwise, the vertex attributes of the current triangle are read, and the attributes of the vertex are interpolated using the centroid coordinates according to the formula described in S4.2. In particular, during the forward process, the partial derivatives of the selected interpolation attributes in screen space are calculated as needed, and the chain rule is applied according to the formula described in S4.2 and the values ​​passed by the rasterization module. Calculate the gradient of the property in the centroid coordinate space.

[0244] The specific implementation of the reverse process is as follows: Each thread calculates the linear index of the corresponding pixel, reads the rasterization result of the corresponding pixel based on the linear index, including the triangle index and the centroid coordinates of the pixel relative to the three vertices of the triangle. If the triangle index is -1, it is invalid, and the current thread will not participate in subsequent calculations. Otherwise, the centroid coordinates of the pixel are read, the attribute value corresponding to the vertex is read from the attribute tensor, and the partial derivative of the partial derivative transmitted by the current automatic differentiation system with respect to the vertex attribute can be calculated according to the formula in the reverse process in S4.2. Similarly, multiple threads may correspond to the same vertex, so they are accumulated in the vertex attribute gradient buffer through atomic summation operations.

[0245] The attribute interpolation module in this embodiment achieves efficient conversion from vertex attributes to pixel attributes through vectorized linear interpolation formulas. By directly using the interpolation weights as multipliers for gradient backpropagation, it achieves concise and accurate directional gradient calculation, ensuring seamless integration with the preceding rasterization module and the subsequent texture filtering module.

[0246] Furthermore, the present invention provides a texture filtering module configured to connect an attribute interpolation module (S4.2) and a subsequent shading module (S4.5), used to sample texture data based on the interpolated texture coordinates. The texture filtering module includes a forward sampling unit and a backward gradient unit.

[0247] The texture filtering module includes a forward sampling unit and a backward gradient unit; wherein...

[0248] The forward sampling unit is used to dynamically select the MIP level and perform trilinear interpolation sampling based on the texture coordinates and screen space derivatives output by the attribute interpolation module.

[0249] The inverse gradient unit is used to receive the gradient signal from the texture filtering module, distribute the gradient signal inversely to the texels participating in the interpolation according to the interpolation weights of the forward sampling, and calculate the gradient of the loss function with respect to the texture coordinates.

[0250] Specifically, during forward sampling: the forward sampling unit receives texture coordinates from the attribute interpolation module (S4.2). or (Cube map) and its screen space derivative. It samples the texture data stored in the texture pyramid according to a preset filtering mode. For the trilinear filtering mode, the forward sampling unit performs the following steps:

[0251] Based on texture coordinates The screen spatial derivative is used to calculate the continuous texture pyramid level that the current pixel should sample. :

[0252]

[0253] in The larger eigenvalues ​​of the Jacobian matrix, which is composed of screen-space derivatives, represent the maximum rate of change of the texture coordinates.

[0254] (b) Determine two discrete MIP levels and and the linear interpolation weights between them. .

[0255] (c) at the level Above, based on texture coordinates The four adjacent texels involved in the bilinear interpolation are identified, and their indices in texture memory are denoted as . , , , The values ​​of these four texels , , , Perform bilinear interpolation to obtain the sampling results. .

[0256] (d) at the level Perform the same bilinear interpolation process as in (c) to obtain the sampling results. .

[0257] (e) Based on weight Linear interpolation is performed to generate the final sampled result of pixel-level texture data. :

[0258]

[0259] More specifically, during the inverse gradient process: the inverse gradient unit receives gradient signals from the subsequent shading module (S4.5), the gradient signals including the sampling results partial derivatives Specifically, for trilinear filtering, the inverse gradient unit receives the gradient... The weights from the forward interpolation are then distributed backward to each texel participating in the interpolation. For the hierarchical level... and The gradient increment of the texels participating in bilinear interpolation is:

[0260]

[0261] in , hierarchical The weights of the bilinear interpolation are applied. All gradients are accumulated into the global texture gradient buffer through atomic summation. Simultaneously, the inverse gradient unit calculates the loss function with respect to texture coordinates. gradient The calculation process is based on the derivative of the weighted machine with respect to the texture coordinates used in forward interpolation, and takes into account the Jacobian matrix that maps from screen space to texture space.

[0262] In a more specific embodiment, the texture filtering module is implemented as follows:

[0263] The specific implementation of forward filtering is as follows: Each thread calculates the linear index of the corresponding pixel and obtains the input texture coordinates based on the index. Then, the level of the continuous texture pyramid to be sampled for the current pixel is calculated according to the formula in S4.3(a), and the two MIP levels are determined according to the formula in S4.3(b) and the interpolation weights are calculated. Assuming the spatial resolution of the input texture is 256*256, the number of MIP levels is 4, and the resolution of each level is the original resolution, 128*128, 64*64, and 32*32 respectively. The number of pixel channels is 3 for each level, and the MIP bias is 1.3.

[0264] Therefore, MIP level ,set up =1, It is 1.3. There are 4 The discrete layer index corresponding to 1.3 is 0.3. The coordinates are converted to texture coordinates for the corresponding layer. Bilinear interpolation is used to calculate the interpolation result for the corresponding texel, and finally, trilinear interpolation is performed. Bilinear interpolation is performed on the texels at these two layers to obtain two sampling results. These two sampling results are then further bilinearly interpolated according to the obtained weights to obtain the final sampling result.

[0265] The reverse process is implemented as follows: Each thread calculates the linear index of the corresponding pixel and obtains the corresponding texture coordinates based on the index. The MIP hierarchy and interpolation results are calculated identically to the front-line process, and then the contribution of the received gradient to the texel is calculated, in terms of texels. For example, let's assume spread to The gradient is: in , and All are interpolation weights.

[0266] Similarly, based on the texel values, the chain rule, and the interpolation weights, the partial derivative of the output pixel with respect to the texture coordinates can be calculated. The derivative of the texel interpolation result with respect to the texture coordinates can be directly calculated using the bilinear interpolation formula. Let the sample value be... The current texture layer width is ,So Thus, the derivative of the sampled value with respect to the texture coordinates can be obtained.

[0267] The texture filtering module in this embodiment dynamically selects the texture pyramid level by calculating the rate of change of texture coordinates in screen space, thereby achieving anti-aliasing sampling; and by using the interpolation weights as multipliers for gradient backpropagation, it realizes gradient calculation of texture coordinates and texture data.

[0268] S4.4, Anti-aliasing module

[0269] This invention provides an anti-aliasing module for post-processing images generated by deferred shading operations, specifically to eliminate aliasing of geometric edges and support differentiable optimization of scene geometric parameters. It is configured to be operatively coupled to the output of an attribute interpolation module (S4.2) or a texture filtering module (S4.3) to receive interpolated geometric attributes or filtered texture attributes as input. The anti-aliasing module includes a forward analysis unit and a backward gradient unit. The forward analysis process involves the forward analysis unit receiving image data output from the attribute interpolation module (S4.2) or the texture filtering module (S4.3) and vertex positions and indices output from the fine rasterization module (S3.4).

[0270] The anti-aliasing module includes a forward analysis unit and a backward gradient unit; wherein...

[0271] The forward analysis unit is used to identify visible discontinuous pixel pairs in the image caused by the contour edges of triangles, and to calculate the mixing weights. And mix the colors of adjacent pixels;

[0272] The inverse gradient unit is used to receive gradient signals in subsequent processing stages and calculate the loss function with respect to the mixed weights. The gradient is then calculated using the chain rule to determine the gradient at the vertex positions that constitute the contour edge.

[0273] Specifically, the anti-aliasing module performs the following steps to identify and handle visibility discontinuities:

[0274] (a) By detecting all horizontally or vertically adjacent pixel pairs, potential regions of visibility discontinuity are identified based on whether the two pixels in the pixel pair have different triangle identifiers.

[0275] (b) For each potential discontinuous region, a triangle associated with the surface closer to the camera is determined based on the normalized device coordinate depth value output by the rasterization stage.

[0276] (c) Examine the sides of the triangle to determine if there are any contour edges that cross the line connecting the centers of adjacent pixels.

[0277] (d) When a contour edge is detected to cross the pixel center line, a blending weight is calculated. Let the two endpoints of the contour edge be... and Then the parametric equation of the contour edge is: Let the center coordinates of adjacent pixels be... and Then the line connecting the centers of the pixels is Their intersection point is for:

[0278]

[0279] Mixed weights Depend on After passing through a pre-defined, differentiable saturation function It can be calculated that:

[0280]

[0281] (e) Based on the said mixed weights Adjust the color values ​​of adjacent pixels and Generates colors after anti-aliasing. and :

[0282]

[0283] Specifically, during the inverse gradient process: the inverse gradient unit receives gradient signals from subsequent processing stages, the gradient signals including partial derivatives with respect to the anti-aliased pixel color. and It utilizes the analysis results on visibility discontinuities stored during the forward analysis process to perform gradient calculations.

[0284] Specifically, for the processed pixel pairs, the loss function is applied to the mixed weights. The gradient is:

[0285]

[0286] Furthermore, the loss function applies to the vertex positions that constitute the contour edges. , The gradient is calculated using the chain rule:

[0287]

[0288] in , The gradient is obtained analytically using the intersection formula. Finally, the obtained gradient is accumulated in the gradient buffer of the corresponding vertex.

[0289] In a more specific embodiment, the anti-aliasing module is implemented as follows:

[0290] The specific implementation of forward parsing: The current thread acquires a pair of pixels, first determining the validity of the corresponding triangle, prioritizing the first valid triangle. If both are valid triangles, the depth values ​​are further compared, prioritizing the triangle with the smaller depth value. Cases where both triangles are invalid are skipped. After selecting a triangle, the contour edges within the current triangle are checked, ensuring that the current edge is not shared by either triangle. For the current sampled pixel position... and adjacent sampling pixel positions The system determines whether the triangles belong to the same triangle based on their indices. If they do not belong to the same triangle, a crossing has occurred, and the system then determines whether the crossing edge is an outline edge.

[0291] If so, calculate the mixed weights according to the formula in S4.4(d). , where the saturation function , in , , The value depends on whether the currently selected triangle is the center pixel triangle or an adjacent pixel triangle, and the final anti-aliased color value is calculated according to the formula in S4.4(e).

[0292] The specific implementation of the reverse process: The current thread obtains a pair of pixels and, based on the color mixing weights determined in the forward stage... Obtain the gradient signal of the corresponding pixel, traverse the channels, calculate and accumulate the gradient signal on the blending weights. The gradient of the input gradient signal is thus obtained. gradient .

[0293] judge If the value is zero, the current thread exits prematurely. Otherwise, obtain the vertex coordinates of the triangle edges of the contour edges confirmed in the forward phase, and calculate based on the forward phase. The formula and chain rule are used to calculate the gradient of a vertex. and ,in and These are the vertices of the outline edges. The optimization objective is defined by the automatic differentiation system.

[0294] The anti-aliasing module in this embodiment performs post-processing-based analytical anti-aliasing on the deferred shading results, transforming visible discontinuous regions into smooth coverage variations; and by storing the visibility analysis results during the forward process, it achieves efficient gradient calculation, thereby supporting end-to-end optimization of the vertex positions that constitute the contour edges.

[0295] Furthermore, the present invention provides a shading module configured to receive output data from an attribute interpolation module (S4.2), a texture filtering module (S4.3), and an anti-aliasing module (S4.4). The shading module organizes the output data into one or more tensors in image space, wherein the attribute tensors include interpolated geometric attributes, filtered texture values, and anti-aliased color values. It should be noted that the tensor here refers to any tensor supported by an automatic differentiation system adapted to the DCU.

[0296] Specifically, the shading module writes the color values ​​output from the constructed rendering pipeline into a tensor defined by the automatic differentiation system. In the reverse process, the shading module passes user-defined rendering optimization targets to the parent module in the rendering pipeline in the form of a tensor defined by the automatic differentiation system.

[0297] In this embodiment, the shading module does not perform specific shading calculations. Instead, it provides the attribute tensor as input to the automatic differentiation system. The automatic differentiation system executes the user-defined shading equation and uses its built-in automatic differentiation mechanism to calculate the gradient of the shading result with respect to the input parameters, thereby achieving end-to-end optimization of materials, textures, and lighting.

[0298] In one specific embodiment, the present invention provides a modular high-performance differentiable rendering method for DCU. The modular design improves the scalability of the method and makes full use of the parallel computing characteristics of DCU, such as 64-bit wide thread bundles, synchronization primitives and shared memory access optimization, thereby improving rendering efficiency. Furthermore, it can achieve differentiable rendering with the help of an automatic differentiation system and a custom inverse operator.

[0299] like Figure 2As shown, the scene's triangle mesh data, texture data, and camera parameters are used as inputs and managed by an automatic differentiation system (using PyTorch as an example); the main management method is as follows:

[0300] In the automatic differentiation system, scene data is defined as tensors on the DCU. By registering the forward and backward computation processes into the automatic differentiation framework, the rendering process can be completed on the DCU. The relevant data in the forward and backward computation processes reside in the DCU's video memory in the form of tensors. Among them, the backward process is mainly used for scene parameter optimization to achieve differentiability.

[0301] Furthermore, the rasterization module, attribute interpolation module, texture filtering module, and anti-aliasing module all provide standardized forward data interfaces and backward gradient interfaces; the dedicated processing module is pluggably connected through the forward data interface and backward gradient interface, and its execution order can be dynamically arranged according to the rendering task requirements.

[0302] like Figure 3 As shown, in the modular rendering execution phase (S4), the designed modules can be arranged according to the actual task requirements. Several possible combinations are given here.

[0303] Example 2

[0304] Please see Figure 4 This embodiment also provides a high-performance modular differentiable rendering device based on DCU, including:

[0305] Modular rendering pipeline initialization unit: used to read rendering data, copy the rendering data to the DCU's video memory through an automatic differentiation system, initialize the parameters of each rendering module on the CPU side, and set the mesh and block configuration for kernel function startup according to the characteristics of the DCU's computing unit.

[0306] Geometric data preprocessing unit: used to perform geometric transformation pipeline processing on the input three-dimensional vertex coordinates through the automatic differentiation system, transforming the original world coordinate vertices into homogeneous coordinates in clip space, and delaying perspective division operation;

[0307] Layered rasterization processing unit: used to receive rendering data from the geometry data preprocessing unit and perform layered rasterization processing to provide pixel-level geometric information and processing targets for subsequent rendering stages; wherein, the layered rasterization processing includes four sub-stages in sequence: triangle setting, layered box rasterization, coarse rasterization and fine rasterization.

[0308] Modular rendering execution unit: used to receive geometric information from the hierarchical rasterization processing unit, generate the final rendered image through differentiable and pluggable dedicated processing modules, and support end-to-end backpropagation of gradients; wherein, the dedicated processing modules include at least one of rasterization module, attribute interpolation module, texture filtering module, anti-aliasing module and shading module.

[0309] The overall process of this embodiment begins with the modular rendering pipeline initialization phase. In this phase, the scene's triangle mesh data, texture data, and camera parameters are used as input and managed by an automatic differentiation system (e.g., PyTorch). Specifically, the scene data is defined as tensors residing in the DCU memory, and the forward and backward computation processes are registered with the automatic differentiation framework, allowing the entire differentiable rendering pipeline to be executed on the DCU. All intermediate data and gradients are stored in tensor form. Next is the geometric data preprocessing phase. This phase uses the automatic differentiation system to perform a complete geometric transformation pipeline on the input vertex world coordinates, including world transformation, viewpoint transformation, and perspective transformation. It outputs the homogeneous coordinates of the vertices in clip space and intentionally delays the perspective division operation to subsequent stages to reduce computational overhead and align with the hardware pipeline.

[0310] The core layered rasterization process then begins. This stage starts with triangle setup. Each DCU thread processes one triangle, receiving its homogeneous vertex coordinates and performing perspective division to obtain normalized device coordinates. These coordinates are then transformed to screen space and converted to sub-pixel level representations. Next, the directed area of ​​the triangle is calculated for backface and degenerate culling, and the coefficients A, B, and C of the depth plane equation are solved, with the results written to a global buffer. Following this is layered binning rasterization. In this embodiment, the parameter binL is set to 4, dividing the screen into a 16x16 pixel grid. Each thread reads one triangle and calculates its axially aligned bounding box in grid space according to a formula. The bounding box boundary is determined by the minimum index lo and the maximum index hi. For triangles covering multiple grids, edge function testing is used for precise intersection determination, and the DCU's any_sync primitive is used for fast inter-thread collaborative judgment, associating the indices of intersecting triangles with the grid. Next is the coarse rasterization stage, where the parameter `tileL` is set to 3. Each grid is further divided into 8x8 pixel tiles. The bounding boxes of triangles in the tile space are calculated, and triangles are classified into small, medium, and large categories based on their bounding box dimensions (whether width and height are both less than or equal to 2). Adaptive strategies such as direct writing, thread-beam joint bounding box testing, and step-by-step edge function testing are used for intersection determination. Finally, a prefix sum scan algorithm is used to count the number of triangles within each tile and mark active tiles. Finally, in the fine rasterization stage, for each triangle within an active tile, a pre-computed cover lookup table is used to quickly generate its 64-bit cover mask within the 8x8 pixel block. The DCU's `popcount` primitive is used to count the number of covered pixels (`pop`), and a fragment index range is assigned to each triangle using a prefix sum scan algorithm. For each generated fragment, the depth value is calculated using the depth plane equation interpolation, tested against the tile depth buffer, and updated using atomic operations. The color is then written, and finally, the tile data is parsed back to the global frame buffer.

[0311] After rasterization, the process enters the modular rendering execution phase. This phase consists of a series of pluggable and differentiable modules, connected through standardized interfaces, whose order can be arranged according to task requirements. The rasterization module receives pixel-triangle association information from the fine rasterization. Its forward unit calculates the directed area using vector cross product based on pixel coordinates and homogeneous triangle vertex coordinates, thereby deriving the perspective correction barycentric coordinates (u, v), normalized depth z_w, and the partial derivative Jacobian matrix J_u of the barycentric coordinates in screen space. The calculation results are numerically clamped to ensure stability. Its backward unit receives gradients from the attribute interpolation module, uses the chain rule and stored intermediate variables to analyze and calculate the gradient of the loss function with respect to the triangle vertex positions, and accumulates it through atomic addition operations. The forward unit of the attribute interpolation module performs linear interpolation on vertex attributes based on barycentric coordinates to obtain pixel-level attributes and can calculate their derivatives in screen space. Its backward unit then feeds back the received attribute gradients to the vertex attributes according to the barycentric coordinate weights. The forward unit of the texture filtering module dynamically calculates the MIP level to be sampled based on the texture coordinates and their screen space derivatives, and performs trilinear filtering sampling. Its backward unit distributes the gradient of the sampling result to each participating texel according to the forward interpolation weights, and calculates the gradient with respect to the texture coordinates. The forward unit of the anti-aliasing module detects the differences in triangle identifiers between adjacent pixels, identifies contour edges, calculates the intersection parameter s* of the line connecting the contour edge and the pixel center, obtains the blending weight α through a differentiable saturation function, and blends the pixel colors. Its backward unit, based on the stored intersection information, backpropagates the color gradient to the blending weights, and then calculates the gradient with respect to the vertex position of the contour edge through the chain rule. The shading module itself does not perform specific calculations, but organizes the attribute tensors output by the aforementioned modules and provides them to the automatic differentiation system. The user-defined shading equations are executed within the automatic differentiation framework, and the gradients are automatically calculated. The backward gradient units of all modules accumulate the gradients to the corresponding global gradient tensors through atomic operations, and finally, the automatic differentiation system completes the end-to-end optimization of parameters such as scene geometry, materials, and lighting. This embodiment achieves high-performance, highly scalable, and boundary-smooth differentiable rendering through the above-mentioned modular design, parallel optimization of the DCU architecture, and accurate differentiable computation.

[0312] The high-performance modular differentiable rendering device based on DCU provided in this application employs a high-performance modular differentiable rendering method based on DCU as described in Embodiment 1 above. It enables high-performance, high-precision differentiable rendering on domestically produced hardware, significantly improving rendering speed, system flexibility, and the convergence effect of 3D reconstruction. Compared with related technologies, the beneficial effects of the high-performance modular differentiable rendering device based on DCU provided in this application are the same as those of the high-performance modular differentiable rendering method based on DCU provided in Embodiment 1 above. Furthermore, other technical features in the high-performance modular differentiable rendering device based on DCU are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0313] Example 3

[0314] Please see Figure 5 This embodiment also provides a high-performance modular differentiable rendering device based on DCU. The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. The computer program is configured to implement a high-performance modular differentiable rendering method based on DCU as described in any one of the above embodiments.

[0315] Figure 5 This illustration shows a structural diagram of a high-performance modular differentiable rendering device based on a DCU, suitable for implementing embodiments of this application. The high-performance modular differentiable rendering device based on a DCU in the embodiments of this application can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The high-performance modular differentiable rendering device based on DCU shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0316] like Figure 5As shown, a DCU-based high-performance modular differentiable rendering device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the DCU-based high-performance modular differentiable rendering device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the DCU-based high-performance modular differentiable rendering device to wirelessly or wiredly communicate with other devices to exchange data. Although the figure shows a DCU-based high-performance modular differentiable rendering device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented alternatively.

[0317] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0318] The high-performance modular differentiable rendering device based on DCU provided in this application employs a high-performance modular differentiable rendering method based on DCU as described in Embodiment 1 above. It achieves high-performance, high-precision differentiable rendering on domestically produced hardware, significantly improving rendering speed, system flexibility, and the convergence effect of 3D reconstruction. Compared with related technologies, the beneficial effects of the high-performance modular differentiable rendering device based on DCU provided in this application are the same as those of the high-performance modular differentiable rendering method based on DCU provided in the above embodiments. Furthermore, other technical features of this high-performance modular differentiable rendering device based on DCU are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0319] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute a high-performance modular differentiable rendering method based on a DCU as described in the above embodiments.

[0320] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0321] The aforementioned computer-readable storage medium may be included in a DCU-based high-performance modular differentiable rendering device; or it may exist independently and not be assembled into a DCU-based high-performance modular differentiable rendering device.

[0322] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a high-performance modular differentiable rendering device, the DCU-based high-performance modular differentiable rendering device achieves a significant improvement in rendering efficiency and accuracy by constructing a modular, pluggable rendering pipeline and deeply optimizing it for DCU hardware characteristics. It also supports high-precision end-to-end gradient backpropagation to better serve various reverse inference and generation tasks in the field of computer graphics technology.

[0323] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0324] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0325] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples; wherein, the name of a module does not, in some cases, constitute a limitation on the unit itself.

[0326] The above are only some embodiments of this application and do not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A high-performance modular differentiable rendering method based on DCU, characterized in that, Includes the following steps: Step S1: Modular rendering pipeline initialization: Read the rendering data, copy the rendering data to the DCU's video memory through the automatic differentiation system, initialize the parameters of each rendering module on the CPU side, and set the mesh and block configuration for kernel function startup according to the characteristics of the DCU's computing unit. Step S2, Geometric data preprocessing: The input three-dimensional vertex coordinates are processed by the automatic differentiation system to transform the original world coordinate vertices into homogeneous coordinates in clip space, and the perspective division operation is delayed. Step S3, Layered rasterization processing: Receive the rendering data from step S2 and perform layered rasterization processing to provide pixel-level geometric information and processing targets for subsequent rendering stages; wherein, the layered rasterization processing includes four sub-stages in sequence: triangle setting, layered box rasterization, coarse rasterization, and fine rasterization. Step S4, Modular rendering execution: Receive geometric information from step S3, and generate the final rendered image through a differentiable and pluggable dedicated processing module, supporting end-to-end backpropagation of gradients; wherein, the dedicated processing module includes at least one of a rasterization module, an attribute interpolation module, a texture filtering module, an anti-aliasing module, and a shading module.

2. The high-performance modular differentiable rendering method based on DCU according to claim 1, characterized in that, In step S1, setting the mesh and block configuration for kernel function startup based on the characteristics of the DCU's computing units specifically includes: The upper bound of the first grid size is determined based on the shared memory size required by the kernel function of each sub-stage of the hierarchical rasterization process and the maximum shared memory capacity of each computing unit of the DCU. The upper bound of the second grid size is determined based on the thread count limit supported by each computing unit in the X and Y dimensions provided by the DCU runtime API. The smaller value between the first grid size upper bound and the second grid size upper bound is taken as the final grid size startup configuration.

3. The high-performance modular differentiable rendering method based on DCU according to claim 1, characterized in that, In step S2, the delayed perspective division specifically involves outputting the homogeneous coordinates of the vertices during the geometric data preprocessing stage and delaying the perspective division operation until the triangle setting stage in step S3.

4. The high-performance modular differentiable rendering method based on DCU according to claim 3, characterized in that, The triangle setup in step S3 includes: Receive the homogeneous coordinates of the three vertices that form the triangle in clip space; Perform perspective division to convert vertex coordinates to normalized device coordinates; The normalized device coordinates are transformed to screen space coordinates and converted to sub-pixel level representation. Based on the preset parameter subL, each pixel block is divided into subL*subL sub-pixel blocks. Calculate the directed area of ​​the triangle and compare it with a preset threshold to perform degenerate triangle detection and back face removal; Based on the depth values ​​of the three vertices, the plane equation coefficients used for depth interpolation are calculated.

5. The high-performance modular differentiable rendering method based on DCU according to claim 1, characterized in that, The layered and boxed rasterization in step S3 includes: According to preset parameters Divide the screen space into portions of size [size]. A grid area of ​​rectangular pixels; For each valid triangle, compute its axially aligned bounding box in the mesh space; For a triangle covering multiple grid regions, the exact intersection relationship between it and each grid region is determined by the side function test.

6. The high-performance modular differentiable rendering method based on DCU according to claim 5, characterized in that, The bounding box is composed of a minimum boundary. and maximum boundary If we represent the bounding box, then the boundary of the bounding box is: in, , , These represent the coordinates of the triangle vertices in screen space; and These represent the corresponding edge vectors; , , These are the preset parameters.

7. The high-performance modular differentiable rendering method based on DCU according to claim 6, characterized in that, The coarse rasterization in step S3 includes: According to preset parameters The grid region is further divided into areas of size [size missing]. A pixel-sized tile; Calculate the bounding box of the triangle in tile space; Based on the bounding box size of the tile space, triangles are classified into small bounding box triangles, medium bounding box triangles, or large bounding box triangles, and different intersection testing strategies are adopted accordingly. Filter out active tiles containing valid triangles and their associated triangle queues.

8. The high-performance modular differentiable rendering method based on DCU according to claim 7, characterized in that, The classification of triangles based on bounding box size includes: When the width and height of the bounding box are both less than or equal to 2 tile units, it is classified as a small bounding box triangle; When the area of ​​the largest joint bounding box of all triangles within a thread bundle does not exceed twice the area of ​​a single triangle, it is classified as a medium bounding box triangle. The remaining cases are classified as large bounding box triangles.

9. A high-performance modular differentiable rendering method based on DCU according to claim 7, characterized in that, The different intersection testing strategies employed include: For small bounding box triangles, the triangle index is directly written into the association list of the corresponding tile; For a medium-sized bounding box triangle, first calculate the joint bounding box at the thread bundle level, and then determine the intersection relationship with each tile in parallel within the range of the joint bounding box; For a large bounding box triangle, perform edge function tests on each tile level to determine the intersection relationship.

10. A high-performance modular differentiable rendering method based on DCU according to claim 1, characterized in that, The fine rasterization in step S3 includes: for each triangle within an active tile, quickly generating its precise overlay mask within an 8×8 pixel block using a pre-computed overlay lookup table; assigning a segment index range to each triangle using a prefix sum scan algorithm; performing depth testing and depth value updates on each generated segment; and writing the color and depth data of the tested segments into the global frame buffer.

11. A high-performance modular differentiable rendering method based on DCU according to claim 10, characterized in that, The formula for generating the exact overlay mask is: ;in, Optimize the flag bits for pre-computed edge functions. This is a lookup table-based coverage calculation function. , These are the coordinates of the triangle vertices in screen space. , and These represent the corresponding edge vectors; , , These are the pre-computed edge function optimization flags.

12. The high-performance modular differentiable rendering method based on DCU according to claim 11, characterized in that, The lookup table-based coverage calculation function The pre-calculation is generated and stored in the constant memory or texture memory of the DCU; the pre-calculation includes: based on the preset line segment shape encoding and intercept parameters, the 8x8 pixel overlay mask under different edge function states is pre-calculated and stored offline.

13. The high-performance modular differentiable rendering method based on DCU according to claim 1, characterized in that, The rasterization module, attribute interpolation module, texture filtering module, and anti-aliasing module all provide standardized forward data interfaces and backward gradient interfaces; the dedicated processing module is pluggably connected through the forward data interface and backward gradient interface, and its execution order can be dynamically arranged according to the rendering task requirements.

14. The high-performance modular differentiable rendering method based on DCU according to claim 1, characterized in that, In the modular rendering execution stage of step S4, the intermediate data output by each dedicated processing module during the forward computation process is temporarily stored as tensors in the DCU video memory. During backpropagation, the gradient signal is transmitted backward along the rendering pipeline, and the calculated parameter gradients are accumulated into the corresponding global gradient tensor through atomic summation operations.

15. A high-performance modular differentiable rendering method based on DCU according to claim 3, characterized in that, The rasterization module includes a forward processing unit and a backward processing unit. For each valid pixel, the forward processing unit calculates the directed area vector and the centroid coordinates for perspective correction, based on its corresponding screen coordinates and the homogeneous coordinates of the vertices of the triangle to which it belongs. Normalized depth And the partial derivatives of the barycentric coordinates in screen space, the Jacobian matrix. The reverse processing unit is used to receive gradient signals from the attribute interpolation module and calculate the gradient of the loss function with respect to the vertex position of the triangle based on the chain rule.

16. A high-performance modular differentiable rendering method based on DCU according to claim 15, characterized in that, The forward processing unit processes the calculated centroid coordinates. Perform a clamping operation to confine it to the range. ; and / or the normalized depth Perform a clamping operation to confine it to the range. .

17. A high-performance modular differentiable rendering method based on DCU according to claim 15, characterized in that, The attribute interpolation module includes a forward interpolation unit and a backward gradient unit; the forward interpolation unit is used to calculate the centroid coordinates output by the rasterization module. Linear interpolation is performed on the vertex attributes to obtain pixel-level interpolated attributes. And calculate the partial derivative of the interpolation attribute in screen space; The inverse gradient unit is used to receive feedback from subsequent modules on the interpolation attributes. The gradient signal is then processed according to the centroid coordinates. The weights are backpropagated to the vertex attributes.

18. A high-performance modular differentiable rendering method based on DCU according to claim 17, characterized in that, The texture filtering module includes a forward sampling unit and a backward gradient unit. The forward sampling unit is used to dynamically select the MIP level and perform trilinear interpolation sampling based on the texture coordinates and their screen space derivatives output by the attribute interpolation module. The backward gradient unit is used to receive the gradient signal from the texture filtering module, distribute the gradient signal inversely to the texels participating in the interpolation according to the interpolation weights of the forward sampling, and calculate the gradient of the loss function with respect to the texture coordinates.

19. A high-performance modular differentiable rendering method based on DCU according to claim 1, characterized in that, The anti-aliasing module includes a forward analysis unit and a backward gradient unit; the forward analysis unit is used to identify visible discontinuous pixel pairs in the image caused by the contour edges of triangles, and calculate the mixing weights. And mix the colors of adjacent pixels; The inverse gradient unit is used to receive gradient signals in subsequent processing stages and calculate the loss function with respect to the mixed weights. The gradient is then calculated using the chain rule to determine the gradient at the vertex positions that constitute the contour edge.

20. A high-performance modular differentiable rendering method based on DCU according to claim 19, characterized in that, The mixed weight By calculating the intersection parameters of the line connecting the contour edge and the pixel center The intersection parameters Input a predefined differentiable saturated function The mixed weights are obtained. .

21. The high-performance modular differentiable rendering method based on DCU according to claim 1, characterized in that, In the step of performing layered rasterization processing or modular rendering, thread bundle synchronization primitives of the DCU runtime are used for inter-thread communication and data consistency management; wherein, the synchronization primitives include at least one of any_sync, all_sync, or ballot_sync.

22. The high-performance modular differentiable rendering method based on DCU according to claim 1, characterized in that, In the fine rasterization of step S3, the popcount primitive of the DCU runtime is used to count the number of bits that are 1 in the overlay mask in order to quickly obtain the number of pixels covered by the triangle.

23. A high-performance modular differentiable rendering device based on a DCU, characterized in that, include: Modular rendering pipeline initialization unit: used to read rendering data, copy the rendering data to the DCU's video memory via an automatic differentiation system, initialize the parameters of each rendering module on the CPU side, and set the mesh and block configuration for kernel function startup according to the DCU's computing unit characteristics; Geometric data preprocessing unit: used to perform geometric transformation pipeline processing on the input 3D vertex coordinates through the automatic differentiation system, transforming the original world coordinate vertices into homogeneous coordinates in clip space, and delaying perspective division operations; Layered rasterization processing unit: used to receive rendering data from the geometric data preprocessing unit and perform layered rasterization processing to provide pixel-level geometric information and processing targets for subsequent rendering stages; wherein, the layered rasterization processing includes four sub-stages in sequence: triangle setting, layered box rasterization, coarse rasterization, and fine rasterization; Modular rendering execution unit: used to receive geometric information from the hierarchical rasterization processing unit, generate the final rendered image through differentiable and pluggable dedicated processing modules, and support end-to-end backpropagation of gradients; wherein, the dedicated processing modules include at least one of rasterization module, attribute interpolation module, texture filtering module, anti-aliasing module and shading module.

24. A high-performance modular differentiable rendering device based on a DCU, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement a high-performance modular differentiable rendering method based on a DCU as described in any one of claims 1 to 22.

25. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the high-performance modular differentiable rendering method based on DCU as described in any one of claims 1-22.

Images