Graphical rendering method, graphical processor and electronic device

Abstract

The application provides a graphics rendering method, a graphics processor and an electronic device; the graphics rendering method is applied to a graphics processor, the graphics processor comprises a geometry processing unit for executing a geometry pipeline and a fragment processing unit for executing a fragment pipeline, the geometry processing unit comprises a vertex shader, the fragment processing unit comprises an interpolation unit and a fragment shader, and the graphics rendering method comprises the following steps: determining a projection result of a target three-dimensional Gaussian body in an image plane based on parameters of the target three-dimensional Gaussian body by using the vertex shader; determining an attenuation coefficient corresponding to the target three-dimensional Gaussian body based on the projection result of the target three-dimensional Gaussian body and position information of at least one pixel covered by the target three-dimensional Gaussian body by using the interpolation unit; and determining a rendering result of the at least one pixel based on the attenuation coefficient corresponding to the target three-dimensional Gaussian body by using the fragment shader.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of computer technology, and in particular to a graphics rendering method, a graphics processor, and an electronic device. Background Technology

[0002] A Graphics Processing Unit (GPU) is a specialized computing hardware component used to perform graphics processing tasks. GPUs typically possess parallel computing capabilities and high-throughput memory access. They are widely used in electronic devices such as computers, servers, and mobile devices to accelerate tasks such as image rendering, video decoding, and artificial intelligence inference.

[0003] The GPU's rendering pipeline (RP) converts a 3D scene into pixels visible on the screen. The RP mainly consists of two core stages: the Geometry Stage (GS) and the Fragment Stage (FS). In related technologies, to accelerate 3D Gaussian Splatting (3DGS), a dedicated calculator is typically added to the rasterization section of the FS to calculate Gaussian values. This not only increases hardware footprint but also has poor compatibility with traditional rendering pipelines. Summary of the Invention

[0004] This application provides a graphics rendering method, a graphics processor, and an electronic device.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides a graphics rendering method applied in a graphics processor. The graphics processor includes a geometry processing unit for executing a geometry pipeline and a fragment processing unit for executing a fragment pipeline. The geometry processing unit includes a vertex shader, and the fragment processing unit includes an interpolation unit and a fragment shader. The graphics rendering method includes: The projection of the target 3D Gaussian volume onto the image plane is determined using a vertex shader based on the parameters of the target 3D Gaussian volume. Using the interpolation unit based on the projection result of the target 3D Gaussian body and the position information of at least one pixel covered by the target 3D Gaussian body, the attenuation coefficient corresponding to the target 3D Gaussian body is determined. The attenuation coefficient corresponding to the target 3D Gaussian body characterizes the degree of attenuation of the deposition intensity of each pixel in at least one pixel in the image plane relative to the center point of the target 3D Gaussian body. The rendering result of at least one pixel is determined by using the fragment shader based on the attenuation coefficient corresponding to the target 3D Gaussian volume.

[0006] This application provides a graphics processor including a geometry processing unit for executing a geometry pipeline and a fragment processing unit for executing a fragment pipeline. The geometry processing unit includes a vertex shader, and the fragment processing unit includes an interpolation unit and a fragment shader, wherein: Vertex shaders are used to determine the projection of the target 3D Gaussian volume onto the image plane based on the parameters of the target 3D Gaussian volume. An interpolation unit is used to determine the attenuation coefficient corresponding to the target 3D Gaussian body based on the projection result of the target 3D Gaussian body and the position information of at least one pixel covered by the target 3D Gaussian body. The attenuation coefficient corresponding to the target 3D Gaussian body represents the degree of attenuation of the deposition intensity of each pixel in the image plane relative to the center point of the target 3D Gaussian body. A fragment shader is used to determine the rendering result of at least one pixel based on the decay coefficient corresponding to the target 3D Gaussian volume.

[0007] This application provides an electronic device including the aforementioned graphics processor.

[0008] The embodiments of this application have the following beneficial effects: First, since vertex shaders can support large-scale parallel computing, reusing vertex shaders for projection of 3D Gaussian bodies can not only significantly accelerate the rendering process, but also reduce the parameters. Compared to transmitting all the parameters of a 3D Gaussian body, this reduces data transmission overhead and bandwidth usage. In addition, compared to adding other hardware separately, it greatly reduces the area of ​​the graphics processor. Secondly, since the interpolation unit integrates adders and multipliers, hardware-level parallelism and acceleration are achieved by reusing the inherent interpolation unit to calculate the attenuation coefficient corresponding to the exponential part of Gaussian, which significantly reduces the load on the fragment shader. At the same time, compared with adding a dedicated calculator in the rasterizer, since this solution directly utilizes existing hardware, it will not increase the area of ​​the graphics processor. Furthermore, since this solution does not rely on a specific calculator or complex block logic, it can be well compatible with traditional pipelines. Finally, since the fragment shader integrates an arithmetic logic unit (ALU), which typically provides exponential calculation functionality, reusing the fragment shader for the core exponential function operations and color blending achieves the goal of leveraging the existing hardware's native support for mathematical functions. This significantly improves rendering real-time performance while maintaining high-quality rendering. Furthermore, by converting the exponential part into interpolation operations in the interpolation unit, the number of instructions in the fragment shader can be significantly reduced, and there's no need to occupy a general-purpose ALU, thus allowing for a higher fill rate. Attached Figure Description

[0009] Figure 1This is a schematic diagram illustrating the implementation process of a graphics rendering method provided in an embodiment of this application; Figure 2 This is a schematic diagram of the composition structure of a graphics processor provided in an embodiment of this application; Figure 3 This is a schematic diagram of a three-dimensional Gaussian sputtering method provided in an embodiment of this application; Figure 4 This is a schematic diagram of a conventional rendering pipeline provided in an embodiment of this application; Figure 5 This is a schematic diagram of a rendering pipeline compatible with 3D Gaussian provided in an embodiment of this application; Figure 6 This is a schematic diagram of a point element provided in an embodiment of this application.

[0010] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0013] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0015] The technical solutions in the embodiments of this application will now be clearly and completely described with reference to the accompanying drawings.

[0016] The graphics rendering method provided in this application embodiment can be executed by the graphics processor of an electronic device. This electronic device can be various types of terminals such as laptops, tablets, desktop computers, set-top boxes, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), or it can be implemented as a server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0017] The graphics processor includes at least a geometry processing unit and a fragment processing unit. The geometry processing unit can be any suitable hardware unit for executing a geometry pipeline. The fragment processing unit can be any suitable hardware unit for executing a fragment pipeline.

[0018] The Geometry Pipeline (GP) is a highly efficient parallel processing pipeline that transforms 3D scene data into screen space coordinates through a series of transformations and operations. It determines the position, shape, and visibility of objects in the scene. GP can also be called the Geometry Processing Pipeline (GPP) or geometry pipeline. A GPU can have at least one GP built-in. It's understandable that GP focuses on the internal hardware-level pipeline structure and execution flow; GS is a logical stage of RP.

[0019] GP (Graphics Processing) can include, but is not limited to, primitive assembly, vertex shading, tessellation (optional), geometry shading (optional), clipping, and viewport transformation. Primitive assembly organizes vertices into complete geometric primitives according to primitive type (e.g., GL_TRIANGLES), preparing for subsequent clipping. Vertex shading uses the vertex shader to execute shader programs on each vertex, performing coordinate transformations (e.g., model → world → view → clip space) and calculating attributes such as normals and texture coordinates. Tessellation uses shell shaders, tessellation machines, and domain shaders to subdivide low-poly triangles into high-poly meshes, achieving level of detail control. Geometry shading uses the geometry shader with complete primitives (e.g., points, lines, triangles) as input to generate new primitives or modify structures, such as expanding a point into a quadrilateral. Clipping removes primitive portions located outside the view frustum, avoiding rasterization of invisible areas and improving efficiency. Viewport transformation maps normalized device coordinates in clip space to screen pixel coordinates, determining the primitive's position in the framebuffer. Therefore, this geometry processing unit includes at least a vertex shader.

[0020] The Fragment Pipeline (FP) is the second stage of the rendering pipeline (RP). It primarily handles the rasterization output, calculating the final color of each pixel and determining the visual quality of the image. FP can include, but is not limited to, rasterization operations, interpolation operations, pixel shading operations, pixel operations (optional), and pixel output. Rasterization converts geometric primitives into pixels on the screen and determines the position, color, and other attributes of each pixel. Interpolation is a crucial step in GPU rendering, smoothly transitioning the attribute values ​​(such as color, texture coordinates, normals, etc.) output by the vertex shader according to pixel position, ensuring visual continuity of the image. In the geometry pipeline, after the vertex shader processes the vertices of the primitives, these vertex attributes are passed to the rasterization stage. When a triangle is rasterized into multiple fragments, each fragment does not directly possess attribute values ​​but is calculated from the attributes of the three vertices of its parent triangle through interpolation. Pixel shading runs the fragment shader, calculating the final color of each fragment, including advanced visual effects such as texture sampling, lighting calculation, and shadow processing. Pixel operations involve blending, depth testing, and stencil testing of the final pixel color. Pixel output operations write the final pixel color to the frame buffer to generate the final image. Therefore, this fragment processing unit includes at least an interpolation unit and a fragment shader (or pixel shader), with the interpolation unit integrating arithmetic logic units such as multipliers and adders.

[0021] It is understandable that GS and FP work together in RP to generate the final rendered image. In some implementations, after the GS of a task is completed, the FP of that task can be executed immediately, or the FPs of each task can be executed sequentially.

[0022] In some implementations, in a tile-based rendering GPU, the GPU also includes a tile-based unit (Tiler), located between the geometry processing unit and the fragment processing unit. The Tiler is a core hardware module in the GPU responsible for dividing the screen into multiple small regions (i.e., "tiles") and managing their independent rendering processes, optimizing memory bandwidth and power consumption. The Tiler divides the framebuffer into multiple tiles of a fixed size (e.g., 32×32 or 16×16 pixels), each tile can be processed independently, reducing frequent access to main memory. After GP (Geometry Processing), the Tiler analyzes the primitives (e.g., triangles) covered by each tile, generates a Primitive List (PL), and stores it in on-chip memory, preparing for subsequent FP (Fragment Processing).

[0023] In some implementations, the GPU may also include a computing unit located prior to the geometry processing unit. This computing unit typically refers to a compute shader in the application phase, which is responsible for data preparation and parallel computation before the GPU starts.

[0024] The computation shader is the only GPU-native computation module located before the geometry processing unit. It is independent of the rendering pipeline, can directly read and write buffers, and is often used for point cloud to Gaussian volume conversion, view-dependent Gaussian visibility culling, Gaussian volume parameter calculation, etc.

[0025] The point cloud to Gaussian volume conversion transforms sparse geometric point clouds into a 3D Gaussian distribution set with volume, shape, and appearance attributes, serving as an explicit representation of the scene. This transforms discrete "points" into a continuous "probability distribution," enabling end-to-end scene optimization through differential rendering while preserving the sparsity and efficiency of point clouds.

[0026] Because the scene contains millions of Gaussian volumes, and only a portion of these volumes are visible to the current camera viewpoint in each frame, efficient culling is necessary to reduce rendering load. Viewpoint-related Gaussian visibility culling includes frustum culling, near / far plane culling, and tile-based culling. Frustum culling determines whether the center of a Gaussian volume or its bounding box intersects with the camera's frustum. If it is completely outside the frustum, it is discarded and not included in subsequent sorting or projection. Near / far plane culling removes Gaussian volumes that are too close to the camera (potentially causing projection instability) or too far away (contributing minimal pixel value). Tile-based culling removes Gaussian volumes that do not overlap with any tiles.

[0027] The parameters of a Gaussian body can include, but are not limited to, position. Covariance matrix Opacity Color / Appearance Factor wait.

[0028] It represents the center coordinates of the Gaussian distribution in 3D space, which can determine the specific spatial position of the Gaussian volume in the scene. It is the reference point for projection transformation and determines the projection center of the Gaussian volume on the screen.

[0029] It is a 3×3 positive semi-definite matrix that controls the shape, magnitude, and direction of the Gaussian distribution. In some implementations, It can be decomposed into rotation matrices and scaling matrix . It is usually represented by a quaternion to describe the orientation of the Gaussian ellipsoid in space. Using three-dimensional vectors This indicates the extent to which the Gaussian body extends along its three principal axes, therefore This anisotropic property allows Gaussian bodies to be stretched into flat or elongated shapes, thus more precisely conforming to the geometry of an object's surface.

[0030] It is a scalar value. It is used to control the transparency of the Gaussian body. During the rendering process, The contribution weight of the Gaussian body to the final pixel color is determined. In some implementations, low-opacity Gaussian bodies are easily pruned (removed) during optimization to remove redundant data.

[0031] It is a set of spherical harmonic (SH) coefficients used to represent view-dependent color information. Lower-order coefficients represent the basic diffuse color, while higher-order coefficients capture non-Lambertian properties such as highlights and reflections, allowing objects to exhibit realistic color changes under different viewing angles.

[0032] Understandably, in a standard GPU rendering pipeline, GP is the first internal stage, and its input is vertex and graph data. However, before GP, there is a crucial pre-processing stage (i.e., the application stage). The application stage mainly runs on the CPU, but can achieve high-performance parallel preprocessing with the help of the GPU's compute shaders.

[0033] Figure 1 This is a schematic diagram illustrating the implementation flow of a graphics rendering method provided in an embodiment of this application, applied to the aforementioned graphics processor, such as... Figure 1 As shown, the graphics rendering method includes steps S11 to S13, wherein: Step S11: Using the vertex shader, determine the projection result of the target 3D Gaussian volume onto the image plane based on the parameters of the target 3D Gaussian volume.

[0034] Here, a Gaussian volume is a fundamental primitive in 3DGS, used to explicitly represent the geometric and appearance information of each local region in a 3D scene. It is not a simple point, but an anisotropic 3D Gaussian distribution with spatial distribution characteristics, its shape and visual attributes precisely defined by a set of learnable parameters. The 3D Gaussian volume has a Gaussian center point as its core, with color density decaying outwards in a Gaussian distribution. From an engineering perspective, after decaying to a certain range, it is considered to have left the range of the 3D Gaussian volume. Therefore, a 3D Gaussian volume can also be understood as an ellipsoid in space.

[0035] The parameters of this Gaussian body may include, but are not limited to, position. Covariance matrix Opacity Color / appearance coefficients, etc. It's understandable that in 3DGS, the parameters of each Gaussian volume are determined before entering the rendering pipeline (i.e., the graphics pipeline).

[0036] The parameters of a Gaussian body can be obtained in any suitable way. In some implementations, the parameters of the Gaussian body can be stored in a cache or memory, and can be obtained by reading the cache or memory. In some implementations, since the RP cannot recognize the Gaussian body, it is necessary to convert the Gaussian body into primitives (such as points, lines, triangles, etc.) that the RP can recognize. In this case, the parameters of the Gaussian body can be used as attributes of the primitives, so the parameters of the Gaussian body can be obtained based on the attributes of the primitives.

[0037] The projection of a Gaussian volume refers to the process of mapping a three-dimensional Gaussian distribution in 3DGS onto a two-dimensional screen plane through camera projection transformation, ultimately forming an elliptical splat with position, shape, and transparency. In essence, after projection, a 3D Gaussian volume can be represented as a 2D Gaussian ellipse on the image plane.

[0038] The projection result of the Gaussian solid may include, but is not limited to, the projection point corresponding to the center point of the Gaussian solid, and the projection matrix corresponding to the covariance matrix of the Gaussian solid. The projection point is the point formed after the center point is projected onto the image plane, that is, the center point of the 2D Gaussian ellipse.

[0039] Projection matrix It is from 3x3 Extracting the 2x2 submatrix from the top left corner, also known as the two-dimensional covariance, determines the major and minor axes and rotation angle of the 2D Gaussian ellipse. In some implementations, It can be represented as: .in, Represents the covariance matrix of a 3D Gaussian body. This represents the view transformation matrix (world coordinates → camera coordinates), which is usually the rotation part of the camera's extrinsic parameters. It is the Jacobian matrix of the projection transformation, representing the local linear approximation of the perspective projection near the current point.

[0040] The projection result of the Gaussian body can be determined in any suitable way.

[0041] In some implementations, the correspondence between each parameter and each projection result can be established in advance. Based on this correspondence, a projection result that matches the parameters of the target three-dimensional Gaussian body can be obtained.

[0042] In some implementations, since the RP does not support Gaussian volumes, the 3D Gaussian volume can first be converted into primitives that the RP can recognize. Projection processing is then performed based on the parameters of the 3D Gaussian volume carried in these primitives to obtain the projection result of the 3D Gaussian volume. In practice, this projection processing includes at least perspective projection of the center point of the 3D Gaussian volume and affine approximate projection of the covariance matrix of the 3D Gaussian volume.

[0043] Step S12: Using the interpolation unit, based on the projection result of the target 3D Gaussian body and the position information of at least one pixel covered by the target 3D Gaussian body, determine the attenuation coefficient corresponding to the target 3D Gaussian body. The attenuation coefficient corresponding to the target 3D Gaussian body represents the degree of attenuation of the deposition intensity of each pixel in the image plane relative to the center point of the target 3D Gaussian body.

[0044] Here, in 3DGS, although each 3D Gaussian volume may be treated as a point primitive in GP, ​​it is not rendered as a single pixel. Its coverage is determined by factors such as projection and opacity. Understandably, whether a Gaussian volume covers multiple pixels depends on its projection size on the screen. In practice, the closer the Gaussian volume is to the camera and the larger its covariance, the more pixels it covers; conversely, it may only affect a few pixels or even a single pixel.

[0045] In the 3DGS rendering system, the attenuation factor essentially describes the degree of attenuation of the deposition intensity of a pixel relative to the center point of a Gaussian volume. The deposition intensity can be the density or opacity contributed by that pixel. Taking the center point of the Gaussian projection onto the screen as the anchor point, the farther the pixel is from the center point of the screen, the greater the attenuation, and the lower the deposition intensity of the Gaussian volume on that pixel.

[0046] In 3DGS, each Gaussian volume is an anisotropic three-dimensional Gaussian distribution. Therefore, the exponential term of the three-dimensional Gaussian distribution determines the "weight strength" of a spatial point's contribution to rendering. The larger the absolute value, the further the location is from the center of the Gaussian volume, and the faster the light intensity attenuates. Projecting the three-dimensional Gaussian volume onto the image plane forms a two-dimensional Gaussian ellipse. The attenuation coefficient corresponding to this three-dimensional Gaussian volume can include the attenuation coefficients of each pixel covered by the three-dimensional Gaussian volume. It can be understood that if the three-dimensional Gaussian volume covers only one pixel, the interpolation unit only needs to determine the attenuation coefficient of that pixel; if the three-dimensional Gaussian volume covers at least two pixels, the interpolation unit needs to determine the attenuation coefficients of all pixels it covers. The process of determining the attenuation coefficient of each pixel is similar.

[0047] The pixel attenuation coefficient can be considered as the exponential term of the two-dimensional Gaussian ellipse corresponding to the three-dimensional Gaussian volume. It determines the "probability density attenuation" of the pixel relative to the center of the Gaussian volume in two-dimensional space. The larger its absolute value, the further away from the center the location is, and the lower the light intensity or probability of its occurrence. Understandably, the exponential term of the two-dimensional Gaussian ellipse is a key component determining the distribution shape. It measures the standardized distance of the pixel from the mean, taking into account the directional and scale differences described by the covariance matrix. Therefore, this exponential term can be analogous to spatial attention weights, focusing on regions close to the center and conforming to the distribution direction, while quickly suppressing the influence of points deviating from the principal axis or too far away.

[0048] The attenuation coefficient of this pixel can be determined in any suitable way.

[0049] In some implementations, a correspondence between each projection result, each position information and each attenuation coefficient can be established in advance. Then, the interpolation unit can obtain an attenuation coefficient that is compatible with both the projection result and the position information of the pixel based on the correspondence.

[0050] In some implementations, the interpolation unit may first determine the common information between pixels based on the projection results; and then determine the attenuation coefficient of each pixel based on the common information and the position information of each pixel.

[0051] Step S13: Using the fragment shader, determine the rendering result of at least one pixel based on the attenuation coefficient corresponding to the target 3D Gaussian volume.

[0052] Here, the rendering result of a pixel includes its color. It is understandable that if the 3D Gaussian volume covers only one pixel, then only the rendering result of that pixel needs to be determined; if the 3D Gaussian volume covers at least two pixels, then the rendering results of all pixels covered by the 3D Gaussian volume need to be determined, and the process for determining the rendering result of each pixel is similar.

[0053] The rendering result of this pixel can be determined in any suitable way.

[0054] In some implementations, a correspondence between each attenuation coefficient and each color can be established in advance. Based on this correspondence, a color that matches the attenuation coefficient of the pixel can be obtained.

[0055] In some implementations, the density of each pixel within a 3D Gaussian volume can be determined first based on the attenuation coefficient and the opacity of the 3D Gaussian volume. Then, the color of each pixel can be determined based on the pixel density and the color of the 3D Gaussian volume. The opacity of the 3D Gaussian volume is used to control its transparency. The color of the 3D Gaussian volume is primarily a view-dependent color, which can be determined using a spherical harmonic function.

[0056] In this embodiment, firstly, since the vertex shader supports large-scale parallel computing, reusing the vertex shader for projection of a 3D Gaussian body not only significantly accelerates the rendering process but also reduces parameter weight. Compared to transmitting all parameters of the 3D Gaussian body, this reduces data transmission overhead and bandwidth usage. Furthermore, compared to adding other hardware separately, it greatly reduces the area of ​​the graphics processor. Secondly, since the interpolation unit integrates adders and multipliers, reusing the inherent interpolation unit to calculate the attenuation coefficient corresponding to the exponential part of the Gaussian achieves hardware-level parallelism and acceleration, significantly reducing the load on the fragment shader. Moreover, compared to adding a dedicated calculator to the rasterizer, this solution directly utilizes existing hardware, thus not increasing the area of ​​the graphics processor. Additionally, since this solution does not rely on a specific calculator or complex block logic, it is well-compatible with traditional pipelines. Finally, since the fragment shader integrates an arithmetic logic unit (ALU), which typically provides exponential calculation functionality, reusing the fragment shader for the core exponential function operations and color blending achieves the goal of leveraging the existing hardware's native support for mathematical functions. This significantly improves rendering real-time performance while maintaining high-quality rendering. Furthermore, by converting the exponential part into interpolation operations in the interpolation unit, the number of instructions in the fragment shader can be significantly reduced, and there's no need to occupy a general-purpose ALU, thus allowing for a higher fill rate.

[0057] In some implementations, step S11 includes: using a vertex shader to obtain the target primitive corresponding to the target 3D Gaussian volume; and using the vertex shader to perform projection processing on the target primitive to obtain the projection result of the target 3D Gaussian volume.

[0058] Here, primitives can be any suitable primitive that can be recognized by RP, such as points, lines, triangles, etc. The attributes of primitives include at least the parameters of the Gaussian volume.

[0059] The primary tasks of a vertex shader are coordinate transformation (such as model-view-projection transformation), lighting calculation (vertex lighting), and vertex animation. The vertex shader does not know which triangle or line a vertex belongs to. Understandably, the input to a vertex shader is typically the attributes of each vertex of a primitive, and the output is the transformed individual vertex. In implementation, the vertex shader reads the vertex data (such as position, normals, texture coordinates, etc.) to be processed from the vertex buffer using the vertex attributes. Therefore, the parameters of a 3D Gaussian volume can also be stored in the vertex buffer, allowing the parameters of the 3D Gaussian volume to be obtained through the vertex attributes.

[0060] The projection processing of this primitive mainly involves projecting at least some parameters (such as center point, depth, covariance matrix, etc.) of a 3D Gaussian volume. Therefore, the projection result of the 3D Gaussian volume includes the result of the parameters projected from the 3D Gaussian volume. It can be understood that the result of projecting the 3D Gaussian volume is a two-dimensional ellipse, called a splat. This splat is not a single pixel, but a continuous distribution covering multiple pixels.

[0061] In some implementations, the projection process includes at least perspective projection of the center point of a 3D Gaussian volume, resulting in a projection of the center position of a 2D Gaussian ellipse. The projection of the center point follows a standard computer graphics perspective projection process, converting 3D world coordinates to 2D screen pixel coordinates. In practice, a view matrix can be used to transform the center of the Gaussian volume from the world coordinate system to the camera coordinate system; for a pinhole camera model, the projected coordinates are normalized to device coordinates; finally, camera intrinsics are used to map the normalized coordinates to pixel space to obtain the center position of the 2D Gaussian ellipse.

[0062] In some implementations, the projection process includes at least affine approximation projection of the covariance matrix of a 3D Gaussian body, resulting in a 2D covariance for the projection of the 3D Gaussian body. The covariance matrix describes the shape and orientation of the Gaussian body. Since perspective projection is non-linear, linear transformation formulas cannot be used directly. 3DGS uses a first-order Taylor expansion for local linear approximation and achieves projection through the Jacobian matrix, which describes the effect of small displacements in camera space on screen pixel coordinates. In practice, the 3D covariance can be transformed to camera space first; then the Jacobian matrix can be calculated; and using the propagation law of covariance under linear transformation, the covariance of camera space can be projected to 2D screen space to obtain the 2D covariance. The 2D covariance can be a 2×2 symmetric positive definite matrix that defines the shape, size, and rotation angle of the elliptical splash on the screen.

[0063] In the embodiments of this application, the vertex shader performs projection processing based on the parameters of the Gaussian volume carried by the primitive, realizing the transformation between model-view-projection, achieving the purpose of efficient coordinate transformation and rendering optimization, and laying the foundation for subsequent rasterization and differentiable rendering.

[0064] In some embodiments, the graphics processor further includes a computing unit, and the graphics rendering method further includes: using the computing unit to obtain parameters of a target three-dimensional Gaussian body; wherein the parameters of the target three-dimensional Gaussian body include the center point of the target three-dimensional Gaussian body and the scaling matrix of the target three-dimensional Gaussian body, the scaling matrix of the target three-dimensional Gaussian body being used to indicate the degree of extension of the target three-dimensional Gaussian body in the three principal axis directions; using the computing unit to determine a target primitive based on the center point of the target three-dimensional Gaussian body and the scaling matrix of the target three-dimensional Gaussian body; wherein the center point of the target primitive is determined by the center point of the target three-dimensional Gaussian body, and the size of the target primitive is determined by the scaling matrix of the target three-dimensional Gaussian body.

[0065] Here, before entering PR, the Gaussian volume can be converted into primitives that PR can recognize (such as points, lines, triangles, etc.) using the CPU or computing unit.

[0066] In some implementations, the CPU or computing unit may first represent the 3D Gaussian volume as primitives. In practice, the 3D Gaussian volume can be represented as point primitives for the following reasons: (1) Because a 3D Gaussian body is an ellipsoid with spatial extensibility, not a path or outline, using lines to represent it will completely lose its volume and color distribution characteristics; at the same time, the rasterization result of the line is a one-dimensional thin strip, which cannot achieve the two-dimensional smooth diffusion effect required by Splat, and multiple lines are difficult to cover complex surfaces, and will cause rendering discontinuity and gaps. Therefore, line primitives are not used for this target primitive. (2) Although triangles are the most commonly used primitives in computer graphics, if a Gaussian ellipsoid is approximated by a triangular mesh, dozens or even hundreds of triangles are needed, which greatly increases the amount of data and rendering overhead. At the same time, once it is converted into a triangle, the direct control of the original parameters such as covariance, transparency, and spherical harmonic color is lost, so the target primitive is not a triangle.

[0067] (3) Point primitives are one of the most efficient primitives of GPU. They have fast rasterization speed and can process millions of Gaussian volumes in parallel to achieve real-time rendering. At the same time, each point carries information such as position, covariance, color, and transparency, which can fully express the 3D Gaussian properties. Moreover, the entire projection and rendering process is differentiable, which allows the Gaussian parameters to be optimized through gradient descent to achieve high-quality reconstruction. Therefore, three-dimensional Gaussian volumes can be represented as point primitives.

[0068] In some implementations, the 3D Gaussian volume can first be converted into a square or rectangular bounding box in screen space. This is to determine the specific pixel range affected by the Gaussian volume on the image plane, thereby enabling efficient rasterization and culling. Then, the square or rectangular bounding box is converted into point primitives. It can be understood that the bounding box itself is a region, while a point primitive is a zero-dimensional location. Therefore, the "conversion" is not a direct equivalent transformation of geometry, but rather a process of data extraction and attribute binding; that is, extracting the geometric center of the bounding box as the center point of the primitive, and using the size of the bounding box as the size of the point primitive.

[0069] In some implementations, a square bounding box with the center point of the three-dimensional Gaussian body as the midpoint and the length of the long side of the three-dimensional Gaussian body as the side length is used to enclose the three-dimensional Gaussian body. The square bounding box is then converted into a point primitive. That is, the center point of the point primitive is the center point of the three-dimensional Gaussian body, and the size of the point primitive indicates the size of the square bounding box.

[0070] In some implementations, a rectangular bounding box is used to enclose the three-dimensional Gaussian solid, with the center point of the three-dimensional Gaussian solid as the midpoint and the length of the long and short sides of the three-dimensional Gaussian solid as the side length. The rectangular bounding box is then converted into a point primitive, that is, the center point of the point primitive is the center point of the three-dimensional Gaussian solid, and the size of the point primitive indicates the size of the rectangular bounding box.

[0071] In the embodiments of this application, converting the three-dimensional Gaussian volume into primitives before entering the rendering pipeline is a prerequisite for realizing 3DGS, so as to ensure efficient, differentiable, real-time, and high-quality rendering.

[0072] In some implementations, the graphics processor further includes a computing unit, and the graphics rendering method further includes, prior to rendering: using the computing unit to determine the order of each three-dimensional Gaussian volume based on the depth information of each three-dimensional Gaussian volume in the scene; and the computing unit feeding each three-dimensional Gaussian volume into the geometry pipeline according to the order of each three-dimensional Gaussian volume.

[0073] Here, in 3DGS, the scene is not composed of traditional triangular meshes or voxels, but rather a collection of point clouds consisting of a large number of 3D Gaussian volumes. Each Gaussian volume is essentially an ellipsoidal probability distribution with directionality and volume, used to represent the radiation characteristics of a point in space. The number of Gaussian volumes in a scene depends on the reconstruction accuracy and complexity, typically ranging from hundreds of thousands to tens of millions. During implementation, each Gaussian volume carries parameters such as position, covariance, color, and opacity, working together to render a realistic image.

[0074] In 3DGS, weighted alpha blending is typically used to synthesize the final pixel color. If nearby objects are drawn first, light from distant objects will not be able to penetrate the lens, leading to occlusion errors and color distortion. Only by drawing objects from far to near according to depth can the physical accuracy of light propagation be guaranteed. If the rendering order is not sorted, random drawing will cause visual artifacts such as flickering, jagged edges, and abnormal overlapping of transparent areas. Therefore, sorting is necessary before rendering, which can significantly improve image stability and realism.

[0075] During implementation, the Gaussian bodies can be sorted according to their depth information (e.g., from far to near, from near to far) to determine their order, and then sent to the subsequent RP in the order of the Gaussian bodies.

[0076] In this embodiment, before the rendering pipeline is executed, the mechanical energy of each Gaussian body in the scene is sorted in order to achieve a realistic transparent overlay effect, restore the sense of layering and lighting details of complex scenes, and ensure visual accuracy.

[0077] In some implementations, the attenuation coefficient corresponding to the target 3D Gaussian volume includes the attenuation coefficient of each pixel in at least one pixel within the target 3D Gaussian volume; the interpolation unit in step S12 includes steps S121 and S122, wherein: Step S121: Using the interpolation unit based on the projection result of the target three-dimensional Gaussian volume, determine the common information between at least one pixel; wherein, the common information is information shared by each pixel and decoupled from each pixel.

[0078] Here, the projection result of the target three-dimensional Gaussian body may include, but is not limited to, at least one of the following: the projection point corresponding to the center point of the target three-dimensional Gaussian body, the projection matrix corresponding to the covariance matrix of the target three-dimensional Gaussian body.

[0079] The projection point includes the third coordinate value on the first direction (such as the U or X axis) of the image plane. and the fourth coordinate value in the second direction (such as the V or Y axis) of the image plane. The second direction is perpendicular to the first direction.

[0080] The projection matrix may include, but is not limited to, a first element A, a second element B, a third element C, and a fourth element D. B and C are the two elements on the diagonal of the projection matrix; A and D are the two elements on the other diagonal. A indicates the degree of diffusion of the projection result of the target 3D Gaussian solid in a first direction of the image plane. B and C both indicate the rotational tilt angle of the projection result of the target 3D Gaussian solid in the image plane. D indicates the degree of diffusion of the projection result of the target 3D Gaussian solid in a second direction of the image plane.

[0081] This common information is shared by all pixels within the same Gaussian volume to determine the attenuation coefficients of at least one pixel. This information is determined by the properties of the Gaussian volume itself (such as its center point and covariance matrix). Once the Gaussian volume is projected onto the screen, this information is a fixed constant for all pixels covered by the Gaussian volume, independent of the pixel's specific location. In some implementations, this common information can be determined by decomposing the exponential portion of the two-dimensional Gaussian ellipse. For example, this common information includes a first common coefficient and a second common coefficient. The first common coefficient can be a coefficient associated with the projection point; that is, its determination depends on the projection point. The second common coefficient is independent of the projection point; that is, its determination does not depend on the projection point. In some implementations, the first common coefficient can further include a first coefficient H1, a second coefficient H2, and a third coefficient H3. H1 and H2 are also related to some elements of the projection matrix, and H3 is related to all elements of the projection matrix.

[0082] The determination of this public information can be done in any suitable way. In some implementations, a correspondence between each projection result and each piece of public information can be established in advance, and the public information adapted to the projection can be obtained based on this correspondence. In some implementations, a first common coefficient can be determined based on the projection points and the projection matrix, and then a second common coefficient can be determined based on the projection matrix.

[0083] Step S122: For each pixel, the attenuation coefficient of the pixel in the target three-dimensional Gaussian volume is determined by the interpolation unit based on common information and the pixel's position information.

[0084] Here, the position information of the pixel includes the first coordinate value of the pixel in the first direction of the image plane. and the second coordinate value of the pixel in the second direction of the image plane .

[0085] It is understandable that the attenuation coefficient of this pixel in each three-dimensional Gaussian volume can be the same or different.

[0086] The attenuation coefficient of the pixel in this three-dimensional Gaussian can be determined in any suitable way.

[0087] In some implementations, a correspondence between each piece of public information, each piece of location information, and each attenuation coefficient can be established in advance. Based on this correspondence, an attenuation coefficient that is compatible with both the public information and the location information can be obtained.

[0088] In some implementations, the attenuation coefficient can be determined based on a first value, a second value, a third value, and a third coefficient, where the first value is determined based on a first element, a first coefficient, and a first coordinate value, such as... The second value is determined based on the fourth element, the second coefficient, and the second coordinate value. The third value is determined based on the second common coefficient, the first coordinate value, and the second coordinate value, such as... For example, the sum of the first value, the second value, the third value, and the third coefficient, as well as the weighted sum of the sum, can be used as the attenuation coefficient of the pixel in a three-dimensional Gaussian volume.

[0089] It is understandable that if the three-dimensional Gaussian volume covers only a single pixel, then step S122 can be executed only once to obtain the attenuation coefficient of the single pixel in the three-dimensional Gaussian volume; if the three-dimensional Gaussian volume covers at least two pixels, then step S122 can be executed at least twice to obtain the attenuation coefficient of each pixel in the three-dimensional Gaussian volume.

[0090] In this embodiment, on the one hand, determining common information based on the projection of a three-dimensional Gaussian volume avoids repeatedly performing expensive matrix multiplication and exponential operations on each pixel, significantly reducing computational complexity and maximizing the efficiency of the graphics processor. On the other hand, since each pixel has a different offset relative to the center of the Gaussian volume, the attenuation coefficient of each pixel in the three-dimensional Gaussian volume can be determined based on the common information and the position of each pixel. This allows for mapping the pixel position to the shape space of the Gaussian volume, accurately measuring the distance between the pixel and the center, and providing a solid foundation for determining the color of subsequent pixels.

[0091] In some embodiments, the projection of the target 3D Gaussian body includes the projection point corresponding to the center point of the target 3D Gaussian body and the projection matrix corresponding to the covariance matrix of the target 3D Gaussian body; the common information includes a first common coefficient and a second common coefficient, the first common coefficient being a constant coefficient shared by all pixels and coupled to the projection result of the target 3D Gaussian body, and the second common coefficient being a constant coefficient shared by all pixels and decoupled from the projection point corresponding to the center point of the target 3D Gaussian body; this step S121 includes steps S1211 and S1212, wherein: Step S1211: Determine the first common coefficient using the interpolation unit based on the projection points corresponding to the center point of the target 3D Gaussian body and the projection matrix corresponding to the covariance matrix of the target 3D Gaussian body; Step S1212: Determine the second common coefficient using the projection matrix corresponding to the covariance matrix of the target 3D Gaussian body based on the interpolation unit.

[0092] Here, the first common coefficient is generated after the Gaussian volume is projected. It is shared by all pixels covered by the Gaussian volume and does not change with pixel changes. The value of the first common coefficient is completely determined by the projection result of the Gaussian volume; that is, the first common coefficient is related to both the projection points and the projection matrix.

[0093] The first common coefficient can be determined in any suitable way.

[0094] In some implementations, a correspondence between each projection point, each projection matrix, and each first common coefficient can be established in advance. Based on this correspondence, a first common coefficient that fits both the projection point and each projection matrix can be obtained.

[0095] In some implementations, the projection matrix corresponding to the covariance matrix of the target 3D Gaussian body includes a first element, a second element, a third element, and a fourth element; the first common coefficient includes a first coefficient, a second coefficient, and a third coefficient; this step S1211 includes steps S141 to S143, wherein: Step S141: Determine the first coefficient using the interpolation unit based on the projection point corresponding to the center point of the target three-dimensional Gaussian body, the first element, the second element, and the third element.

[0096] Here, the first element A indicates the degree of diffusion of the projection result of the target 3D Gaussian body in a first direction of the image plane. The second element B and the third element C indicate the rotational tilt angle of the projection result of the target 3D Gaussian body in the image plane. B and C are two elements on the diagonal of the projection matrix. In some embodiments, B=C. The fourth element D indicates the degree of diffusion of the projection result of the target 3D Gaussian body in a second direction of the image plane. D and A are two elements on the diagonal of the projection matrix. In practice, these four elements together define the shape, orientation, and extent of the two-dimensional Gaussian ellipsoid.

[0097] The first coefficient H1 is used to correct the offset of each pixel in the first direction. In other words, during subsequent pixel-by-pixel calculations, this constant first coefficient is used in conjunction with the coordinates of the current pixel in the first direction to quickly offset the center offset term in the first direction within the exponential expansion. The center offset term directly determines the distance of the pixel from the Gaussian center. In other words, H1 represents the global linear offset reference for the weight attenuation contribution of the Gaussian projection center to all covered pixels in the first direction.

[0098] H1 can be determined in any suitable way.

[0099] In some implementations, a correspondence between each projection point, each first element, each second element, each third element, and each H1 can be established in advance. Based on this correspondence, an H1 that is compatible with the projection point, the first element, the second element, and the third element can be obtained.

[0100] In some implementations, H1 can be determined based on the sum of the first product, the second product, and the third product, and a weighted average of that sum. The first product is based on... The product between A and A is determined, such as The second product is based on B and The product between them is determined, such as The third product is based on C and The product between them is determined, such as In practice, H1 can be obtained by reusing the adders and multipliers in the interpolation unit.

[0101] Step S142: Determine the second coefficient using the interpolation unit based on the projection point corresponding to the center point of the target 3D Gaussian body, the second element, the third element, and the fourth element.

[0102] Here, the second coefficient is used to correct the offset of each pixel in the second direction. That is, in subsequent pixel-by-pixel calculations, this constant second coefficient is used to calculate the coordinates of the current pixel in the second direction, which can quickly offset the center offset term in the second direction in the exponential expansion. In other words, H2 represents the global linear offset reference for the weight attenuation contribution of the Gaussian projection center to all covered pixels in the second direction.

[0103] H2 can be determined in any suitable way.

[0104] In some implementations, a correspondence between each projection point, each second element, each third element, each fourth element, and each H2 can be established in advance. Based on this correspondence, an H2 that is compatible with the projection point, the second element, the third element, and the fourth element can be obtained.

[0105] In some implementations, H2 can be determined based on the sum of the fourth, fifth, and sixth products, and a weighted average of that sum. The fourth product is based on... The product between and D is determined, such as The fifth product is based on B and The product between them is determined, such as The sixth product is based on C and The product between them is determined, such as In practice, H2 can be obtained by reusing the adders and multipliers in the interpolation unit.

[0106] Step S143: Determine the third coefficient using the interpolation unit based on the projection point corresponding to the center point of the target 3D Gaussian body, the first element, the second element, the third element, and the fourth element.

[0107] Here, the third coefficient is used to correct the offset of the projection point corresponding to the center point of the target 3D Gaussian volume for each pixel. In other words, the third coefficient, as an offset constant, directly corrects the origin offset error introduced by the expansion of the exponential term. In other words, H3 represents the distance reference of the Gaussian projection center itself relative to the origin of the graphic.

[0108] H3 can be determined in any suitable way.

[0109] In some implementations, a correspondence between each projection point, each first element, each second element, each third element, each fourth element, and each H3 can be established in advance. Based on this correspondence, an H3 that is compatible with the projection point, the first element, the second element, the third element, and the fourth element can be obtained.

[0110] In some implementations, H3 can be determined based on the sum of the seventh, eighth, ninth, and tenth products, and a weighted average of that sum. The seventh product is based on... The product between and D is determined, such as: The eighth product is based on A and The product between them is determined, such as The ninth product is based on base B. and The product between them is determined, such as: The tenth product is based on C. and The product between them is determined, such as: In practice, H3 can be obtained by reusing the adders and multipliers in the interpolation unit.

[0111] In this way, the interpolation unit can determine the first common coefficient by projecting the points and the elements in the projection matrix, thus enabling the complex matrix operations to be performed in advance. This avoids the need for general matrix construction and multiplication instructions in the subsequent fragment stage, significantly reducing the load on the arithmetic logic unit in the fragment stage and significantly reducing the number of instruction cycles per pixel.

[0112] The second common coefficient is also generated after the Gaussian volume is projected. It is shared by all pixels covered by the Gaussian volume and does not change with the number of pixels. The value of the second common coefficient is completely determined by the projection matrix of the Gaussian volume; that is, the second common coefficient is only related to the projection matrix and not to the projection points.

[0113] The second common coefficient can be determined in any suitable way.

[0114] In some implementations, step S1212 includes: determining a second common coefficient based on the second and third elements using an interpolation unit.

[0115] Here, the second and third elements are the two elements on the diagonal of the projection matrix. The second common coefficient can be determined in any suitable way.

[0116] In some implementations, a correspondence between each second element, each third element, and each second common coefficient can be pre-established. Based on this correspondence, a second common coefficient that fits both the second element and each third element can be obtained. In some implementations, the second common coefficient can be obtained based on the sum of the second and third elements, or a weighted sum of those sums. In practice, the second common coefficient is obtained using adders, multipliers, or other multiplexed interpolation units. For example, the second and third elements can be used as inputs to an adder, and the output is the second common coefficient.

[0117] In this way, the interpolation unit can determine the second common coefficient based on the two elements in the projection matrix, thereby decoupling the coefficients from the projection points and achieving the goal of simplification and fast computation.

[0118] In some implementations, a correspondence between each projection and each second common coefficient can be established in advance. Then, the interpolation unit can obtain the second common coefficient that matches the projection matrix based on the correspondence.

[0119] In the embodiments of this application, in 3DGS, a Gaussian volume typically covers multiple pixels. If common coefficients are not extracted before entering the fragment stage, the same matrix operation will be repeatedly performed on each pixel. Therefore, by distinguishing between "Gaussian volume inherent properties" and "pixel-related variables", and determining different common coefficients based on the projection points and projection matrices, redundant matrix operations can be eliminated. Furthermore, subsequent calculations only require loading a small number of common coefficients and pixel coordinates, thereby optimizing register allocation and thread utilization and significantly improving the processor's concurrency capabilities.

[0120] In some implementations, the pixel's position information includes a first coordinate value of the pixel in a first direction of the image plane and a second coordinate value of the pixel in a second direction of the image plane; the common information includes a first common coefficient and a second common coefficient; the first common coefficient includes a first coefficient, a second coefficient, and a third coefficient; the projection matrix corresponding to the covariance matrix of the target 3D Gaussian volume includes a first element and a fourth element; the step S122, "using the fragment shader to determine the attenuation coefficient of the pixel in the target 3D Gaussian volume based on the common information and the pixel's position information," includes steps S1221 to S1224, wherein: Step S1221: Determine the first value using the interpolation unit based on the first element, the first coefficient, and the first coordinate value.

[0121] Here, the first value represents the attenuation contribution of a pixel in the first direction; it is the independent attenuation component of the pixel in the first direction. The first value can be determined in any suitable way.

[0122] In some implementations, a correspondence between each A, each H1, each first coordinate value and each first value can be established in advance. Based on this correspondence, a first value that matches A, H1 and the first coordinate value can be obtained.

[0123] In some implementations, the first value can be determined based on a first difference, such as using the first difference as the first value. The first difference is the difference between the fourth value and the fifth value. The fourth value is determined based on A and the first coordinate value, such as... The fifth value is determined based on H1 and the first coordinate value, that is: In practice, the first value is obtained by reusing the adders and multipliers in the interpolation unit.

[0124] Step S1222: Determine the second value using the interpolation unit based on the fourth element, the second coefficient, and the second coordinate value.

[0125] Here, the second value characterizes the pixel's attenuation contribution in the second direction; it is the pixel's independent attenuation component in the first direction. The second value can be determined in any suitable way.

[0126] In some implementations, a correspondence between each D, each H2, each second coordinate value and each second value can be established in advance. Based on this correspondence, a second value that matches both D, H2 and the second coordinate value can be obtained.

[0127] In some implementations, the second value can be determined based on the second difference, such as using the second difference as the second value. The second difference is the difference between the sixth and seventh values. The sixth value is determined based on D and the second coordinate value, such as... The seventh value is determined based on H2 and the second coordinate value, that is: In practice, the second value is obtained by reusing the adders and multipliers in the interpolation unit.

[0128] Step S1223: Determine the third value using the interpolation unit based on the second common coefficient, the first coordinate value, and the second coordinate value.

[0129] Here, the third value characterizes the attenuation contribution of a pixel in a non-orthogonal direction. A non-orthogonal direction is a direction that is not parallel to either the first or second direction; that is, a non-orthogonal direction is any tilted direction that is not parallel to the orthogonal coordinate axes. The third value is the cross-coupling component of the pixel in the first and second directions. The third value can be determined in any suitable way.

[0130] In some implementations, a correspondence between each second common coefficient, each first coordinate value, each second coordinate value, and each third value can be established in advance. Based on this correspondence, a third value that matches both the second common coefficient, the first coordinate value, and the second coordinate value can be obtained.

[0131] In some implementations, the third value can be the second common coefficient, the product of the first coordinate value and the second coordinate value, and the weighted sum of the product.

[0132] Step S1224: Using the interpolation unit, determine the attenuation coefficient of the pixel in the target three-dimensional Gaussian volume based on the first value, the second value, the third value, and the third coefficient.

[0133] Here, the attenuation coefficient of the pixel can be determined in any suitable way.

[0134] In some implementations, a correspondence between each first value, each second value, each third value, each H3 and each attenuation coefficient can be established in advance. Based on this correspondence, an attenuation coefficient that is compatible with the first value, the second value, the third value and H3 can be obtained.

[0135] In some implementations, the sum of the first value, the second value, the third value, and the third coefficient, as well as the weighted sum of these values, can be used as the attenuation coefficient of the pixel in the target 3D Gaussian volume. In practice, this attenuation coefficient can be obtained by reusing the adder in the interpolation unit.

[0136] In this embodiment, the pixel attenuation coefficient is calculated based on each common coefficient and pixel coordinate by reusing the existing hardware in the interpolation unit. This fully utilizes the dedicated hardware resources of the GPU to achieve high-efficiency, low-latency parallel computing, significantly improving computing efficiency and avoiding wasting computing power on repetitive logic.

[0137] In some embodiments, step S13 includes steps S131 and S132, wherein: Step S131: Using the fragment shader, determine the density of each pixel in the target 3D Gaussian volume based on the attenuation coefficient corresponding to the target 3D Gaussian volume.

[0138] Here, the attenuation coefficient corresponding to the target 3D Gaussian volume includes the attenuation coefficient of each pixel in at least one pixel in the target 3D Gaussian volume.

[0139] The density of this pixel in a three-dimensional Gaussian volume This density, used to describe the effect of a pixel's position relative to the center of a Gaussian volume, is also known as spatial weight. It decays in a Gaussian distribution, rapidly approaching zero at the edges. In practice, the pixel's density can be the same or different in different three-dimensional Gaussian volumes. Density determines the attenuation (or transmittance) of light in the medium.

[0140] The density of this pixel in a three-dimensional Gaussian volume can be determined in any suitable way.

[0141] In some implementations, a correspondence between each attenuation coefficient and each density can be established in advance. Based on this correspondence, a density that matches the attenuation coefficient of the pixel can be obtained.

[0142] In some implementations, step S131 includes: for each pixel, using a fragment shader to determine the density of the pixel in the target three-dimensional Gaussian volume based on the pixel's attenuation coefficient and the opacity of the target three-dimensional Gaussian volume.

[0143] Here, the parameters of a 3D Gaussian solid also include opacity, which controls the transparency of the Gaussian solid, i.e., the degree of solidity of the Gaussian solid itself. The greater the opacity, the stronger its ability to block light from behind.

[0144] The density of this pixel can be determined in any suitable way. In some implementations, a correspondence between each attenuation coefficient, each opacity, and each density can be established in advance, and a density that matches both the attenuation coefficient and the opacity can be obtained based on this correspondence. In some implementations, the density can be the product of the attenuation coefficient and the opacity, or a weighted average of the product.

[0145] In this way, the density of pixels in the Gaussian volume is determined based on the attenuation coefficient and opacity, so that the density integral along the light rays has an analytical solution or an efficient numerical approximation. This decouples the geometry from the physical material, reduces aliasing during rendering, and ensures that the rendering results conform to the laws of physical optics. Especially when dealing with semi-transparent objects (such as smoke and glass), it can correctly accumulate occlusion relationships rather than simply layer overlay, which is the key to achieving high-quality, real-time new perspective compositing.

[0146] Step S132: For each pixel, the fragment shader is used to determine the rendering result of the pixel based on the pixel's density in the target 3D Gaussian volume and the color of the target 3D Gaussian volume.

[0147] Here, the color of the three-dimensional Gaussian solid is mainly the view-dependent color of the three-dimensional Gaussian solid, which can be determined by the spherical harmonic function.

[0148] The rendering result of a pixel includes the color of the pixel. .

[0149] The color of this pixel can be determined in any suitable way.

[0150] In some implementations, a correspondence between each density, each Gaussian body color, and each pixel color can be established in advance. Based on this correspondence, the color of the pixel that matches both the density and the color of the target three-dimensional Gaussian body can be obtained.

[0151] In some implementations, since a pixel can be covered by at least one Gaussian body, the contribution of the target 3D Gaussian body can be determined first, and then the color of the pixel can be determined based on the contribution of each 3D Gaussian body covering the pixel. The weight corresponding to the target 3D Gaussian body is determined based on the pixel density within the target 3D Gaussian body and the color of the target 3D Gaussian body.

[0152] In this embodiment, on the one hand, determining the pixel density in a 3D Gaussian volume based on the attenuation coefficient not only ensures physical consistency and accurate ray transmission modeling, enabling precise handling of subsequent occlusion relationships, but also captures geometric details with high fidelity, utilizes complex structures, and avoids detail loss due to over-smoothing. On the other hand, determining the pixel rendering result based on density and color achieves decoupling of geometric and appearance attributes, reduces mutual interference between parameters, and adaptively renders and restores details with high quality.

[0153] In some implementations, the rendering result of a pixel includes the pixel's color; the step S132, "determining the rendering result of a pixel using a fragment shader based on the pixel's density in the target 3D Gaussian volume and the target 3D Gaussian volume's color," includes steps S1321 and S1322, wherein: Step S1321: Using the fragment shader, determine the contribution of the target 3D Gaussian volume based on the pixel density in the target 3D Gaussian volume, the color of the target 3D Gaussian volume, and the transmittance of the target 3D Gaussian volume.

[0154] Here, in 3DGS, the contribution of a 3D Gaussian volume is the combined weight of the influence of a single 3D Gaussian volume on the final pixel color. It's understandable that the contribution of a single Gaussian volume to the pixel color is not directly assigned, but determined through a depth-based sorting and transparency composition process. In implementation, different 3D Gaussian volumes can have the same or different contribution values.

[0155] The transmittance corresponding to this three-dimensional Gaussian body This is a cumulative transmittance, representing the proportion of light that is not absorbed or blocked by any preceding Gaussian bodies before reaching the current 3D Gaussian volume. In practice, the more opaque the preceding Gaussian body is, the more completely the light is blocked by the objects in front, resulting in a lower transmittance for that Gaussian body; in other words, the current Gaussian body contributes nothing to the image. Conversely, the more transparent the preceding Gaussian body is, the less light is blocked, resulting in a higher transmittance for that Gaussian body; in other words, the current Gaussian body has high visibility and contributes significantly to the image. This is understandable. .

[0156] In implementation, for the [number]th [item] sorted by depth from near to far... A Gaussian body, its cumulative transmittance It can be represented as: ; in, Indicates the first The spatial attenuation weight (i.e., density) of a Gaussian volume at the pixel location. It is understandable that the cumulative transmittance for the first Gaussian volume is 1.

[0157] The contribution of a Gaussian volume is determined by its spatial attenuation weight (i.e., density), color, and cumulative transmittance. The contribution depends not only on the color and size of the Gaussian volume itself, but also on its depth order and occlusion relationship. Even if a Gaussian volume is brightly colored or large in volume, if it is completely occluded by an opaque object in front of it, its actual contribution will approach zero.

[0158] The contribution of the Gaussian body can be determined in any suitable way.

[0159] In some implementations, a correspondence between each density, color, transmittance, and contribution can be established in advance. Based on this correspondence, a contribution that matches the density, color, and transmittance can be obtained.

[0160] In some implementations, the contribution can be the product of density, color, and transmittance, or a weighted average of that product.

[0161] Step S1322: Determine the color of a pixel using a fragment shader based on the contribution of at least one three-dimensional Gaussian body, wherein the at least one three-dimensional Gaussian body includes a target three-dimensional Gaussian body, and each three-dimensional Gaussian body in the at least one three-dimensional Gaussian body covers the pixel.

[0162] Here, the contribution values ​​of different Gaussian bodies can be the same or different.

[0163] In practice, if the number of at least one three-dimensional Gaussian body is one, then the at least one three-dimensional Gaussian body includes only the target three-dimensional Gaussian body; if the number of at least one three-dimensional Gaussian body is at least two, then the at least one three-dimensional Gaussian body includes not only the target three-dimensional Gaussian body but also other three-dimensional Gaussian bodies.

[0164] The color of this pixel can be determined in any suitable way. In some implementations, the sum of the contributions of each three-dimensional Gaussian volume, or a weighted average of those sums, can be used as the color of the pixel. In practice, if the pixel is covered by only one three-dimensional Gaussian volume, then the color of the pixel can be the contribution of that three-dimensional Gaussian volume.

[0165] In this embodiment, on the one hand, the contribution of the 3D Gaussian volume is determined based on density, color, and transmittance, achieving physical realism and accurate occlusion modeling. This allows for the natural handling of semi-transparent or complex geometric structures such as smoke, glass, and leaves, avoiding the need for complex sorting or deep stripping to achieve transparency effects in traditional mesh rendering. It balances quality, speed, and storage efficiency, offering significant advantages in physical plausibility, computational efficiency, and optimization flexibility. On the other hand, determining the final color of a pixel based on the contribution of each 3D Gaussian volume covering it is not only a discrete implementation of the volumetric rendering equation but also a crucial bridge connecting geometric representation, appearance modeling, and efficient computation, representing the core advantage of 3DGS's high-quality, high-efficiency rendering.

[0166] In some implementations, the graphics processor further includes a tileization unit. Before the target primitive corresponding to the target 3D Gaussian volume enters the fragment pipeline, the graphics rendering method further includes: using the tileization unit to determine the intersection result between the target primitive and at least one tile; and using the tileization unit to send the target primitive into the fragment pipeline when the intersection result indicates that the target primitive intersects with the target tile and the opacity of the target 3D Gaussian volume is greater than a preset threshold.

[0167] Here, before entering the fragment pipeline, that is, before step S12, the Gaussian bodies need to be screened. Only Gaussian bodies that meet the screening conditions will be further sent into the subsequent fragment pipeline to ensure that only geometry that makes a significant contribution to the final pixel color participates in the expensive shading calculation, thereby greatly improving efficiency while ensuring rendering quality.

[0168] The intersection results may include, but are not limited to, the first intersection result and the second intersection result. The first intersection result indicates that the primitive intersects with at least one block, and the second intersection result indicates that the primitive does not intersect with any other primitive.

[0169] The intersection result can be determined in any suitable manner. In some implementations, the coverage area of ​​a primitive can be compared with the coverage area of ​​each tile to determine whether the primitive is located in at least one tile.

[0170] In some implementations, since only Gaussian bodies that intersect with the tile can be projected onto the corresponding area on the screen, non-intersecting Gaussian bodies are eliminated in advance to avoid invalid calculations. Therefore, it is necessary to determine whether the primitives corresponding to the Gaussian bodies intersect with the tile.

[0171] In some implementations, since opacity directly reflects the visual contribution of a Gaussian volume to a pixel, its impact on the final color is negligible if the opacity is below a preset threshold. Therefore, Gaussian volumes with opacities greater than the preset threshold must be included in the subsequent fragment stage. This eliminates a large number of "almost transparent" or deeply occluded Gaussian volumes, significantly reducing the fragment shader load.

[0172] The preset threshold can be any suitable, sufficiently small value, such as 0.005 or 0.001. Its purpose is to remove Gaussian bodies that contribute very little to the final image. This preset threshold can be fixed or dynamically adjusted, such as based on the average opacity of a local area of ​​the image.

[0173] Understandably, one can first use opacity for initial screening, then determine the intersection results, and further screen based on the intersection results.

[0174] In the embodiments of this application, Gaussian volumes are filtered based on their intersection with the tiles and their opacity. This means that Gaussian volumes are eliminated by utilizing geometric visibility and visual contribution. This is a core performance optimization mechanism in the 3DGS rendering pipeline, which can achieve efficient and accurate rendering optimization, significantly reduce computational load, and maintain high-quality visual output.

[0175] Based on the above embodiments, this application also provides a graphics processor. Figure 2 This is a schematic diagram of the composition structure of a graphics processor provided in an embodiment of this application, such as... Figure 2 As shown, the graphics processor 200 includes a geometry processing unit 21 for executing a geometry pipeline and a fragment processing unit 22 for executing a fragment pipeline. The geometry processing unit 21 includes a vertex shader 211, and the fragment processing unit 22 includes an interpolation unit 221 and a fragment shader 222, wherein: Vertex shader 211 is used to determine the projection of the target 3D Gaussian volume onto the image plane based on the parameters of the target 3D Gaussian volume; Interpolation unit 221 is used to determine the attenuation coefficient corresponding to the target three-dimensional Gaussian body based on the projection result of the target three-dimensional Gaussian body and the position information of at least one pixel covered by the target three-dimensional Gaussian body. The attenuation coefficient corresponding to the target three-dimensional Gaussian body characterizes the degree of attenuation of the deposition intensity of each pixel in the image plane relative to the center point of the target three-dimensional Gaussian body. Fragment shader 222 is used to determine the rendering result of at least one pixel based on the attenuation coefficient corresponding to the target 3D Gaussian volume.

[0176] Here, the parameters of the Gaussian body may include, but are not limited to, position. Covariance matrix Opacity Color / appearance factors, etc.

[0177] The projection result of the Gaussian body may include, but is not limited to, the projection point corresponding to the center point of the Gaussian body, the projection matrix corresponding to the covariance matrix of the Gaussian body, etc. The process of the vertex shader determining the projection result can be referred to the specific implementation of step S11 above.

[0178] The attenuation coefficient corresponding to the three-dimensional Gaussian volume can include the attenuation coefficient of each pixel covered by the three-dimensional Gaussian volume. It is understood that if the three-dimensional Gaussian volume covers only one pixel, then the interpolation unit only needs to determine the attenuation coefficient of that pixel; if the three-dimensional Gaussian volume covers at least two pixels, then the interpolation unit needs to determine the attenuation coefficient of each pixel it covers. The process by which the interpolation unit determines the attenuation coefficient of a pixel can be found in the specific implementation of step S12 described above.

[0179] The rendering result of this pixel includes the pixel's color value. The process by which the fragment shader determines the rendering result of this pixel can be found in the specific implementation of step S13 above.

[0180] In this embodiment, firstly, since the vertex shader supports large-scale parallel computing, reusing the vertex shader for projection of a 3D Gaussian body not only significantly accelerates the rendering process but also reduces parameter weight. Compared to transmitting all parameters of the 3D Gaussian body, this reduces data transmission overhead and bandwidth usage. Furthermore, compared to adding other hardware separately, it greatly reduces the area of ​​the graphics processor. Secondly, since the interpolation unit integrates adders and multipliers, reusing the inherent interpolation unit to calculate the attenuation coefficient corresponding to the exponential part of the Gaussian achieves hardware-level parallelism and acceleration, significantly reducing the load on the fragment shader. Moreover, compared to adding a dedicated calculator to the rasterizer, this solution directly utilizes existing hardware, thus not increasing the area of ​​the graphics processor. Additionally, since this solution does not rely on a specific calculator or complex block logic, it is well-compatible with traditional pipelines. Finally, since the fragment shader integrates an arithmetic logic unit (ALU), which typically provides exponential calculation functionality, reusing the fragment shader for the core exponential function operations and color blending achieves the goal of leveraging the existing hardware's native support for mathematical functions. This significantly improves rendering real-time performance while maintaining high-quality rendering. Furthermore, by converting the exponential part into interpolation operations in the interpolation unit, the number of instructions in the fragment shader can be significantly reduced, and there's no need to occupy a general-purpose ALU, thus allowing for a higher fill rate.

[0181] In some implementations, vertex shader 211 is used to obtain the target primitives corresponding to the target 3D Gaussian volume; and to perform projection processing on the target primitives to obtain the projection result of the target 3D Gaussian volume.

[0182] In some embodiments, the graphics processor further includes a computing unit for acquiring parameters of a target 3D Gaussian solid, the parameters of which include the center point of the target 3D Gaussian solid and the scaling matrix of the target 3D Gaussian solid, the scaling matrix of the target 3D Gaussian solid indicating the extent of extension of the target 3D Gaussian solid in the three principal axis directions; and determining a target primitive based on the center point of the target 3D Gaussian solid and the scaling matrix of the target 3D Gaussian solid, the center point of the target primitive being determined by the center point of the target 3D Gaussian solid, and the size of the target primitive being determined by the scaling matrix of the target 3D Gaussian solid.

[0183] In some implementations, the attenuation coefficient corresponding to the target three-dimensional Gaussian volume includes the attenuation coefficient of each pixel in the target three-dimensional Gaussian volume; the interpolation unit 221 is used to determine the common information between at least one pixel based on the projection result of the target three-dimensional Gaussian volume, the common information being information shared by each pixel and decoupled from each pixel; for each pixel, the attenuation coefficient of the pixel in the target three-dimensional Gaussian volume is determined based on the common information and the position information of the pixel.

[0184] In some implementations, the projection result of the target 3D Gaussian body includes the projection point corresponding to the center point of the target 3D Gaussian body and the projection matrix corresponding to the covariance matrix of the target 3D Gaussian body; the common information includes a first common coefficient and a second common coefficient, wherein the first common coefficient is a constant coefficient shared by each pixel and coupled to the projection result of the target 3D Gaussian body, and the second common coefficient is a constant coefficient shared by each pixel and decoupled from the projection point corresponding to the center point of the target 3D Gaussian body; the interpolation unit 221 is used to determine the first common coefficient based on the projection point corresponding to the center point of the target 3D Gaussian body and the projection matrix corresponding to the covariance matrix of the target 3D Gaussian body; and to determine the second common coefficient based on the projection matrix corresponding to the covariance matrix of the target 3D Gaussian body.

[0185] In some implementations, the projection matrix corresponding to the covariance matrix of the target 3D Gaussian body includes a first element, a second element, a third element, and a fourth element; the first common coefficient includes a first coefficient, a second coefficient, and a third coefficient; the interpolation unit 221 is used to determine the first coefficient based on the projection point corresponding to the center point of the target 3D Gaussian body, the first element, the second element, and the third element, wherein the first element indicates the degree of diffusion of the projection result of the target 3D Gaussian body in a first direction of the image plane, and the second and third elements indicate the rotation tilt angle of the projection result of the target 3D Gaussian body in the image plane, and the first coefficient is used to correct the offset of each pixel in the first direction; the second coefficient is determined based on the projection point corresponding to the center point of the target 3D Gaussian body, the second element, the third element, and the fourth element, wherein the fourth element indicates the degree of diffusion of the projection result of the target 3D Gaussian body in a second direction of the image plane, the second direction being perpendicular to the first direction, and the second coefficient is used to correct the offset of each pixel in the second direction; the third coefficient is determined based on the projection point corresponding to the center point of the target 3D Gaussian body, the first element, the second element, the third element, and the fourth element, and the third coefficient is used to correct the offset of each pixel at the projection point corresponding to the center point of the target 3D Gaussian body.

[0186] In some implementations, the projection matrix corresponding to the covariance matrix of the target three-dimensional Gaussian body includes a second element and a third element; interpolation unit 221 is used to determine a second common coefficient based on the second element and the third element.

[0187] In some implementations, the pixel's position information includes a first coordinate value of the pixel in a first direction of the image plane and a second coordinate value of the pixel in a second direction of the image plane; common information includes a first common coefficient and a second common coefficient; the first common coefficient includes a first coefficient, a second coefficient, and a third coefficient; the projection matrix corresponding to the covariance matrix of the target three-dimensional Gaussian volume includes a first element and a fourth element; interpolation unit 221 is used to determine a first value based on the first element, the first coefficient, and the first coordinate value, the first value representing the attenuation contribution of the pixel in the first direction; determine a second value based on the fourth element, the second coefficient, and the second coordinate value, the second value representing the attenuation contribution of the pixel in the second direction; determine a third value based on the second common coefficient, the first coordinate value, and the second coordinate value, the third value representing the attenuation contribution of the pixel in a non-orthogonal direction, the non-orthogonal direction being a direction that is not parallel to either the first or second direction; and determine the attenuation coefficient of the pixel in the target three-dimensional Gaussian volume based on the first value, the second value, the third value, and the third coefficient.

[0188] In some implementations, fragment shader 222 is used to determine the density of each pixel in the target three-dimensional Gaussian volume based on the attenuation coefficient corresponding to the target three-dimensional Gaussian volume; and for each pixel, to determine the rendering result of the pixel based on the pixel density in the target three-dimensional Gaussian volume and the color of the target three-dimensional Gaussian volume.

[0189] In some implementations, the attenuation coefficient corresponding to the target three-dimensional Gaussian volume includes the attenuation coefficient of each pixel in the target three-dimensional Gaussian volume; the rendering result of the pixel includes the color of the pixel; and the fragment shader 222 is used to determine the density of the pixel in the target three-dimensional Gaussian volume for each pixel based on the attenuation coefficient of the pixel in the target three-dimensional Gaussian volume and the opacity of the target three-dimensional Gaussian volume.

[0190] In some implementations, fragment shader 222 is configured to determine the contribution of a target three-dimensional Gaussian body based on the pixel density in the target three-dimensional Gaussian body, the color of the target three-dimensional Gaussian body, and the transmittance of the target three-dimensional Gaussian body, and to determine the color of the pixel based on the contribution of at least one three-dimensional Gaussian body, wherein the at least one three-dimensional Gaussian body includes the target three-dimensional Gaussian body, and each three-dimensional Gaussian body in the at least one three-dimensional Gaussian body covers the pixel.

[0191] In some implementations, the graphics processor further includes a tileization unit, which determines the intersection result between the target primitive and at least one tile before the target primitive corresponding to the target 3D Gaussian volume enters the fragment pipeline; if the intersection result indicates that the target primitive intersects with the target tile and the opacity of the target 3D Gaussian volume is greater than a preset threshold, the target primitive is sent into the fragment pipeline.

[0192] In some implementations, the graphics processor further includes a computing unit that, prior to rendering, determines the order of each three-dimensional Gaussian volume based on the depth information of each volume in the scene, and feeds each three-dimensional Gaussian volume into the geometry pipeline in that order.

[0193] The description of the graphics processor embodiments above is similar to the description of the method embodiments above, and has similar beneficial effects. For technical details not disclosed in the graphics processor embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0194] The technical solution of this application is described in detail below.

[0195] 3D Gaussian sputtering is a relatively new technology that has emerged in recent years. Compared to traditional rendering methods, Gaussian sputtering is a differentiable rendering method that incorporates the advantages of Neural Radiance Field Rendering (NRFR) while avoiding the implicit use of neural networks to represent the rendered scene, thus allowing for better integration with methods that explicitly represent the scene. 3D sputtering is therefore widely used in scene reconstruction, world model generation, and other fields.

[0196] The concept of three-dimensional Gaussian sputtering, as follows Figure 3 As shown, the scene in space is represented by a large number of three-dimensional Gaussian spheres. The three-dimensional Gaussian spheres have a Gaussian center point as their core, and the color density decays outwards in a Gaussian distribution. From an engineering perspective, after decaying to a certain range, it is considered to have left the range of the three-dimensional Gaussian sphere. Therefore, the three-dimensional Gaussian sphere can also be understood as an ellipsoid in space. During rendering, rays are emitted sequentially from the viewpoint towards each pixel on the screen. The intersections of the rays and all the Gaussian spheres they pass through are calculated. The color and transparency of the intersections are calculated, and all intersection values ​​are blended to obtain the final rendered color of the pixel corresponding to that ray.

[0197] Mathematically, a three-dimensional Gaussian can be represented by the following formula: ; in For rays and Gaussians The coordinates of the sampling points, For Gauss The center coordinates. It is Gauss The covariance matrix.

[0198] In engineering, rotation and scaling matrices are typically used to represent this. .

[0199] In engineering, Gaussian 3D calculations need to be projected to 2D before further calculations. If the projection matrix is... , If we consider the radial approximation of the projection transformation, then the two-dimensional projection of the three-dimensional Gaussian covariance (corresponding to the aforementioned projection matrix) is: .

[0200] Therefore, finally, the Gaussian projection into two dimensions... The density at that intersection point (corresponding to the density of the aforementioned pixel) is: .in For the Gaussian Opacity. This represents the Gaussian value calculated from the intersection of the Gaussian and ray projections onto a two-dimensional plane. This value is also the Alpha value required for the blend operation (i.e., alpha mixing).

[0201] After obtaining the intersection values ​​of the ray and each Gaussian point, the color of a pixel P is... Accumulate in the following way: ; After unfolding: ; in, Representing Gauss The color value. Representing Gauss The density.

[0202] Because the cumulative equation depends on the order, before rendering, the 3D Gaussian sphere needs to be sorted by depth value Z from farthest to nearest before calculation. During calculation, the 3D Gaussian sphere is calculated in patches. Only 3D Gaussian spheres that intersect with a patch within a certain density threshold will participate in the calculation for that patch.

[0203] like Figure 4 As shown, in a traditional graphics rendering engine, the graphics data that the software inputs to the hardware processor via API is processed in the order of geometry pipeline (i.e., geometry pipeline) and fragment pipeline (i.e. fragment pipeline) and finally presented on the screen.

[0204] The geometry pipeline includes operations such as primitive assembly, vertex shading, surface tessellation, geometry shading, clipping, and viewport transformation. The fragment pipeline includes operations such as rasterization and pixel shading.

[0205] Direct-rendering-based graphics systems transfer primitives to the primitive setup (i.e., after clipping and viewport transformation) Figure 4 The processing includes tile-based and rasterization modules.

[0206] Tiled-based graphics systems also require a tile-based operation after the geometry pipeline, assigning primitives to tiles they can cover. The fragment pipeline processes each tile. Multiple processing hardware components can be connected to the fragment pipeline, allowing for parallel processing of different tiles.

[0207] After vertex shading, optional tessellation, and optional geometry shading stages in the geometry pipeline, primitives are transformed to clip space. Dividing the primitive coordinates in clip space by the w-component value transforms them to normalized device coordinate space (NDC space). Primitives in NDC space undergo further transformations, ultimately transforming to viewport space. Primitives in viewport space are then rasterized, meaning the pixels covered by the primitives are defined.

[0208] The attribute data carried by primitives and other vertex-based data are interpolated to sampled pixels. This interpolation is passed to the fragment shader in the form of a plane equation or sampled values, and participates in pixel shading calculations in the fragment shader.

[0209] Related technologies, by adding a Gaussian calculation module calculator to the rasterizer, can accelerate the calculation in Gaussian sputtering. However, the added calculator also increases the area, and since the calculator is only used for Gaussian calculation, the utilization rate of this increased area is not high.

[0210] At the same time, this technology is not compatible with the geometry part of the traditional graphics pipeline. It requires that the triangle / line / point primitives, as well as the corresponding vertex coordinates and attribute values, of the traditional graphics input be directly used as the input of the rasterizer.

[0211] This application accelerates Gaussian sputtering in a graphics GPU. Simultaneously, while accelerating Gaussian sputtering calculations, it makes the most of the existing computing modules in the graphics pipeline.

[0212] To accelerate 3D Gaussian rendering while minimizing modifications to the existing graphics rendering pipeline, the following technical solution is proposed, utilizing existing computational modules (including vertex shaders, interpolation shaders, and fragment shaders): The interpolation and pixel shaders need to be adjusted, as does the supported point primitive size. The vertex shader needs to perform some calculations for 3D Gaussians, but the shader hardware itself does not need to be modified.

[0213] like Figure 5 As shown, point primitives with dimensions are used as square bounding boxes to represent the range of a three-dimensional Gaussian.

[0214] First, the 3D Gaussian is sorted before entering the rendering pipeline. Then, the 3D Gaussian is represented as a point primitive using its center point and long side dimensions (obtained from the scaling matrix). The parameters required for the 3D Gaussian calculation, including the scaling / rotation matrix, opacity, and spherical harmonic parameters (or color values), are passed in as primitive attributes.

[0215] In the vertex shader, projection is calculated based on the 3D Gaussian center point, spherical harmonics, and scaling / rotation matrices. The projection yields 2D points (i.e., projected points), depth, and a 2D covariance matrix. These results are then passed as attribute values ​​to the pixel shader.

[0216] The 3D Gaussian representation of point primitives is processed in the clipping and tileization units. Pieces outside the view frustum and screen protector ring are culled, and the remaining point primitives are assigned to individual tiles based on their intersection with each tile. After the primitives in the tile are calculated by the rasterizer, the multipliers and adders in the interpolation module are reused in the interpolation stage to pre-calculate the exponential part of the Gaussian calculation. The subsequent exponential function part is performed in the pixel shader. The resulting density is output as alpha, and the color as color. Through a blend operation, the final 3D Gaussian rendering result is obtained.

[0217] As mentioned above The formula expansion form of output merge follows the blend formula: srcColor srcAlpha + destColor (1-srcAlpha); The src prefix indicates the output of a pixel shader at a certain time, and the dest prefix indicates the color value of the previous time stored in the framebuffer (which can be 0 for the first time).

[0218] It can be seen that as long as the 3D Gaussian enters the rendering pipeline for sorting, the regular blend operation can meet the requirements of the 3D Gaussian.

[0219] like Figure 6 As shown, a 3D Gaussian sphere projected onto a 2D plane can actually be represented as an ellipse. Using a square bounding box, with the center point of the 3D Gaussian sphere as the midpoint, we can obtain... Figure 6 The second row shows an example with a bounding box.

[0220] If the center point of a 3D Gaussian is represented as the midpoint of a point primitive, and the size of the point primitive indicates the size of the square bounding box, then the range of the square bounding box can be represented by points.

[0221] When such point primitives are input into the traditional graphics pipeline, normal clipping, culling, and blockization functions will be able to conservatively send Gaussians that intersect with the blocks to the corresponding fragment stage, and then Gaussian calculations can be performed.

[0222] Using a square bounding box will waste some space for the ellipse shown in the figure, but it can change the flow of traditional primitives to a minimum.

[0223] Alternatively, two sizes can be used to represent a 3D Gaussian using a rectangular bounding box, but this method would require significant modifications to the traditional graphics pipeline.

[0224] To leverage the existing graphics pipeline, point primitives are used to convey the center point information of the Gaussian. The Vulkan API allows point primitives to support point sizes of at least 64 pixels or 256 pixels. Therefore, by further supporting point sizes in the 16.8 range, it is possible to support representing 3D Gaussians using points.

[0225] Because the attribute information of a 3D Gaussian follows the center point, both the vertex projection calculation and the covariance matrix projection calculation can be performed in the vertex shader. The results are then passed as attributes to the fragment stage.

[0226] Conservative rasterization calculations are performed using point primitives. This may result in extra pixels, but no modifications to the existing rasterizer are required.

[0227] Since it has already been converted to two dimensions in the vertex shader, the exponent part can be decomposed as follows: ; ; in , These are the coordinates of the intersection point (i.e., pixel) in a two-dimensional plane. A, B, C, and D are obtained after projection in the vertex shader, corresponding to the Gaussian coordinates. The attribute value. Then the exponent part can be reduced to: ; For multiple pixels covered by the same Gaussian curve, the following common parts can be calculated during the interpolation stage, and the adder and multiplier can be shared with the traditional interpolation part. Only some control logic needs to be adjusted (i.e., the logic of the operation order): (Corresponding to the aforementioned first coefficient); (Corresponding to the second coefficient mentioned above); (Corresponding to the aforementioned second common coefficient); (Corresponding to the aforementioned third coefficient); and The values ​​are combined and substituted sequentially into the coordinates of each pixel within the same Gaussian coverage to obtain the final exponential calculation result (corresponding to the attenuation coefficient of the aforementioned pixels). The final result is sent to the fragment shader, where the exponential calculation in the fragment shader's ALU (Arithmetic Logic Unit) is used to obtain the remaining part of the Gaussian calculation.

[0228] Depending on the level of support for exponentiation instructions in the ALU, most ALUs provide exponentiation calculation functionality, so there is no need to add an additional exponent calculator to support Gaussian calculations.

[0229] In practice, the 3D Gaussian scene needs to be sorted by the CPU or computation shader before it enters the graphics pipeline for rendering.

[0230] During implementation, the pixel shader needs to output color and alpha values. The pipeline needs to enable blend operations and disable early-z culling and early clipping features such as HSR.

[0231] This application accelerates Gaussian processing while reducing the need for additional hardware. It improves performance while minimizing chip area increase.

[0232] This application breaks down Gaussian calculation into two stages: interpolation and pixel shader. By reusing the original multipliers and adders of each stage, as well as the exponentiation operation of the pixel shader, it accelerates Gaussian calculation while reducing the introduction of additional hardware. This improves performance while reducing the increase in chip area.

[0233] This application improves overall parallel performance by adjusting the point size and incorporating the projection part into the vertex shader calculation.

[0234] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of software products. These software products are stored in a storage medium and include several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0235] This application provides an electronic device including any of the aforementioned graphics processors.

[0236] It should be noted that the description of the above device embodiments is similar to the description of the above graphics processor embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the graphics processor embodiments of this application for understanding.

[0237] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0238] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0239] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0240] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0241] In addition, each functional unit in the embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0242] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0243] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0244] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A graphics rendering method, characterized in that, Applied in a graphics processing unit (GPU), the GPU includes a geometry processing unit for executing a geometry pipeline and a fragment processing unit for executing a fragment pipeline. The geometry processing unit includes a vertex shader, and the fragment processing unit includes an interpolation unit and a fragment shader. The graphics rendering method includes: The vertex shader is used to determine the projection result of the target 3D Gaussian volume onto the image plane based on the parameters of the target 3D Gaussian volume; The interpolation unit determines the attenuation coefficient corresponding to the target three-dimensional Gaussian body based on the projection result of the target three-dimensional Gaussian body and the position information of at least one pixel covered by the target three-dimensional Gaussian body. The attenuation coefficient corresponding to the target three-dimensional Gaussian body characterizes the degree of attenuation of the deposition intensity of each pixel in the at least one pixel relative to the center point of the target three-dimensional Gaussian body in the image plane. The rendering result of at least one pixel is determined using the fragment shader based on the attenuation coefficient corresponding to the target 3D Gaussian volume.

2. The graphics rendering method according to claim 1, characterized in that, Using the vertex shader based on the parameters of the target 3D Gaussian volume, the projection result of the target 3D Gaussian volume onto the image plane is determined, including: The vertex shader is used to obtain the target primitives corresponding to the target 3D Gaussian volume; The target primitive is projected using the vertex shader to obtain the projection result of the target three-dimensional Gaussian volume.

3. The graphics rendering method according to claim 2, characterized in that, The graphics processor further includes a computing unit, and the graphics rendering method further includes: The calculation unit is used to obtain the parameters of the target three-dimensional Gaussian body; wherein, the parameters of the target three-dimensional Gaussian body include the center point of the target three-dimensional Gaussian body and the scaling matrix of the target three-dimensional Gaussian body, and the scaling matrix of the target three-dimensional Gaussian body is used to indicate the degree of extension of the target three-dimensional Gaussian body in the three principal axis directions; The calculation unit determines the target primitive based on the center point of the target 3D Gaussian body and the scaling matrix of the target 3D Gaussian body; wherein the center point of the target primitive is determined by the center point of the target 3D Gaussian body, and the size of the target primitive is determined by the scaling matrix of the target 3D Gaussian body.

4. The graphics rendering method according to claim 1, characterized in that, The attenuation coefficient corresponding to the target three-dimensional Gaussian volume includes the attenuation coefficient of each of the at least one pixel in the target three-dimensional Gaussian volume; Using the interpolation unit, based on the projection result of the target 3D Gaussian body and the position information of at least one pixel covered by the target 3D Gaussian body, the attenuation coefficient corresponding to the target 3D Gaussian body is determined, including: The interpolation unit uses the projection results of the target three-dimensional Gaussian volume to determine the common information between the at least one pixel; wherein, the common information is information shared by each pixel and decoupled from each pixel. For each pixel, the interpolation unit determines the attenuation coefficient of the pixel in the target three-dimensional Gaussian volume based on the common information and the position information of the pixel.

5. The graphics rendering method according to claim 4, characterized in that, The projection result of the target 3D Gaussian body includes the projection point corresponding to the center point of the target 3D Gaussian body and the projection matrix corresponding to the covariance matrix of the target 3D Gaussian body; the common information includes a first common coefficient and a second common coefficient, wherein the first common coefficient is a constant coefficient shared by all pixels and coupled with the projection result of the target 3D Gaussian body, and the second common coefficient is a constant coefficient shared by all pixels and decoupled from the projection point corresponding to the center point of the target 3D Gaussian body; Using the interpolation unit to determine the common information between the at least one pixel based on the projection result of the target 3D Gaussian volume, including: The first common coefficient is determined using the interpolation unit based on the projection point corresponding to the center point of the target 3D Gaussian body and the projection matrix corresponding to the covariance matrix of the target 3D Gaussian body; The second common coefficient is determined using the interpolation unit based on the projection matrix corresponding to the covariance matrix of the target three-dimensional Gaussian body.

6. The graphics rendering method according to claim 5, characterized in that, The projection matrix corresponding to the covariance matrix of the target 3D Gaussian solid includes a first element, a second element, a third element, and a fourth element; the first common coefficient includes a first coefficient, a second coefficient, and a third coefficient; The first common coefficient is determined using the interpolation unit based on the projection point corresponding to the center point of the target 3D Gaussian solid and the projection matrix corresponding to the covariance matrix of the target 3D Gaussian solid, including: The first coefficient is determined using the interpolation unit based on the projection point corresponding to the center point of the target 3D Gaussian body, the first element, the second element, and the third element; wherein, the first element indicates the degree of diffusion of the projection result of the target 3D Gaussian body in a first direction on the image plane, and the second and third elements indicate the rotation tilt angle of the projection result of the target 3D Gaussian body in the image plane; the first coefficient is used to correct the offset of each pixel in the first direction. The second coefficient is determined using the interpolation unit based on the projection point corresponding to the center point of the target 3D Gaussian body, the second element, the third element, and the fourth element; wherein the fourth element indicates the degree of diffusion of the projection result of the target 3D Gaussian body in a second direction on the image plane, the second direction being perpendicular to the first direction; the second coefficient is used to correct the offset of each pixel in the second direction; The interpolation unit determines the third coefficient based on the projection point corresponding to the center point of the target three-dimensional Gaussian body, the first element, the second element, the third element, and the fourth element; wherein the third coefficient is used to correct the offset of each pixel from the projection point corresponding to the center point of the target three-dimensional Gaussian body.

7. The graphics rendering method according to claim 5, characterized in that, The projection matrix corresponding to the covariance matrix of the target three-dimensional Gaussian solid includes a second element and a third element; The second common coefficient is determined using the interpolation unit based on the projection matrix corresponding to the covariance matrix of the target 3D Gaussian body, including: The second common coefficient is determined using the interpolation unit based on the second element and the third element.

8. The graphics rendering method according to claim 4, characterized in that, The pixel's position information includes the pixel's first coordinate value in a first direction on the image plane and the pixel's second coordinate value in a second direction on the image plane; the common information includes a first common coefficient and a second common coefficient; the first common coefficient includes a first coefficient, a second coefficient, and a third coefficient; the projection matrix corresponding to the covariance matrix of the target 3D Gaussian solid includes a first element and a fourth element; The interpolation unit determines the attenuation coefficient of the pixel in the target 3D Gaussian volume based on the common information and the pixel's position information, including: The interpolation unit determines a first value based on the first element, the first coefficient, and the first coordinate value; wherein the first value represents the attenuation contribution of the pixel in the first direction; The interpolation unit determines a second value based on the fourth element, the second coefficient, and the second coordinate value; wherein the second value characterizes the attenuation contribution of the pixel in the second direction; The interpolation unit determines a third value based on the second common coefficient, the first coordinate value, and the second coordinate value; wherein the third value represents the attenuation contribution of the pixel in a non-orthogonal direction, the non-orthogonal direction being a direction that is not parallel to both the first direction and the second direction; The interpolation unit uses the first value, the second value, the third value, and the third coefficient to determine the attenuation coefficient of the pixel in the target three-dimensional Gaussian volume.

9. The graphics rendering method according to claim 1, characterized in that, The rendering result of at least one pixel is determined using the fragment shader based on the attenuation coefficient corresponding to the target 3D Gaussian volume, including: The density of each pixel in the target three-dimensional Gaussian volume is determined using the fragment shader based on the attenuation coefficient corresponding to the target three-dimensional Gaussian volume; For each pixel, the fragment shader determines the rendering result of the pixel based on the pixel's density in the target 3D Gaussian volume and the color of the target 3D Gaussian volume.

10. The graphics rendering method according to claim 9, characterized in that, The attenuation coefficient corresponding to the target 3D Gaussian volume includes the attenuation coefficient of each pixel in the target 3D Gaussian volume; the rendering result of the pixel includes the color of the pixel; Determining the density of each pixel in the target three-dimensional Gaussian volume using the fragment shader based on the attenuation coefficient corresponding to the target three-dimensional Gaussian volume includes: for each pixel, determining the density of the pixel in the target three-dimensional Gaussian volume using the fragment shader based on the attenuation coefficient of the pixel in the target three-dimensional Gaussian volume and the opacity of the target three-dimensional Gaussian volume; The rendering result of the pixel is determined using the fragment shader based on the pixel density in the target 3D Gaussian volume and the color of the target 3D Gaussian volume. This includes: determining the contribution of the target 3D Gaussian volume based on the pixel density in the target 3D Gaussian volume, the color of the target 3D Gaussian volume, and the transmittance of the target 3D Gaussian volume; and determining the color of the pixel based on the contribution of at least one 3D Gaussian volume, wherein the at least one 3D Gaussian volume includes the target 3D Gaussian volume, and each of the at least one 3D Gaussian volume covers the pixel.

11. The graphics rendering method according to claim 1, characterized in that, The graphics processor further includes a tileization unit, and before the target primitive corresponding to the target 3D Gaussian volume enters the fragment pipeline, the graphics rendering method further includes: The intersection result between the target primitive and at least one block is determined using the blockification unit; When the intersection result indicates that the target primitive intersects with the target tile and the opacity of the target 3D Gaussian body is greater than a preset threshold, the target primitive is fed into the fragment pipeline using the tileization unit.

12. The graphics rendering method according to any one of claims 1 to 11, characterized in that, The graphics processor further includes a computing unit, and the graphics rendering method further includes, prior to rendering: The computing unit uses the depth information of each three-dimensional Gaussian volume in the scene to determine the order of each three-dimensional Gaussian volume, and sends each three-dimensional Gaussian volume into the geometry pipeline according to the order of each three-dimensional Gaussian volume.

13. A graphics processor, characterized in that, It includes a geometry processing unit for performing a geometry pipeline and a fragment processing unit for performing a fragment pipeline. The geometry processing unit includes a vertex shader, and the fragment processing unit includes an interpolation unit and a fragment shader, wherein: The vertex shader is used to determine the projection result of the target 3D Gaussian volume onto the image plane based on the parameters of the target 3D Gaussian volume; The interpolation unit is used to determine the attenuation coefficient corresponding to the target three-dimensional Gaussian body based on the projection result of the target three-dimensional Gaussian body and the position information of at least one pixel covered by the target three-dimensional Gaussian body. The attenuation coefficient corresponding to the target three-dimensional Gaussian body characterizes the degree of attenuation of the deposition intensity of each pixel in the at least one pixel relative to the center point of the target three-dimensional Gaussian body in the image plane. The fragment shader is used to determine the rendering result of at least one pixel based on the attenuation coefficient corresponding to the target three-dimensional Gaussian volume.

14. An electronic device, characterized in that, Includes the graphics processor as described in claim 13.

Images