3D Gaussian Splashing Rendering Method, 3D Rendering Device, Electronic Device, and Storage Medium

Abstract

The present application relates to the field of 3D rendering technology, and discloses a 3D Gaussian splash rendering method, a 3D Gaussian splash rendering device, an electronic device, and a storage medium. The method includes: determining a set of 3D ellipsoids corresponding to a 3D model in a 3D Gaussian splash, where the set of 3D ellipsoids includes a plurality of 3D ellipsoids; for each 3D ellipsoid in the set of 3D ellipsoids, mapping the 3D ellipsoid to the imaging plane of a rendering camera to obtain a 2D ellipse; determining the circumscribed rectangle of the 2D ellipse; using the circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and rendering all pixel points within the coloring area. By using the circumscribed rectangle of the 2D ellipse obtained by mapping each 3D ellipsoid in the imaging plane as the coloring area, the range of the coloring area in the present application is smaller than the 3-sigma range, thereby reducing invalid color calculations, saving fragment shader computing resources, and improving the rendering speed.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of 3D rendering technology, and more particularly to a 3D Gaussian splashing rendering method, a 3D Gaussian splashing rendering device, an electronic device, and a storage medium. Background Art

[0002] Recently, in the field of 3D scene reconstruction, the 3D Gaussian splashing (hereinafter referred to as 3DGS) technology has made significant progress based on Neural Radiance Fields (NeRF). It achieves high-fidelity details while maintaining low scene complexity and also has high real-time rendering performance. 3DGS uses rasterization instead of the computationally expensive ray tracing technology for rendering, greatly improving the real-time rendering performance. During the 3DGS rendering process, the fragment shader part consumes approximately 70% of the performance, and a large part of the performance of the Graphics Processing Unit (GPU) is consumed in fragment shader calculations.

[0003] However, since each 3D ellipsoid in 3DGS rendering uses a 3-sigma boundary during projection, there are many invalid shading areas within the projection range, increasing the computational amount in the invalid shading areas, resulting in a large amount of invalid calculations, wasting fragment shader computing resources, increasing the requirements for the GPU, and affecting the rendering speed. Summary of the Invention

[0004] In view of the above problems, embodiments of the present application provide a 3D Gaussian splashing rendering method, a 3D Gaussian splashing rendering device, an electronic device, and a storage medium, which are used to solve the problem in the prior art that the shading area corresponding to each 3D ellipsoid in 3DGS is too large, resulting in excessive invalid calculations and wasting GPU computing resources.

[0005] According to one aspect of the embodiments of the present application, a 3D Gaussian splashing method is provided. The method includes: determining a set of 3D ellipsoids corresponding to a 3D model in 3D Gaussian splashing, where the set of 3D ellipsoids includes multiple 3D ellipsoids; for each 3D ellipsoid in the set of 3D ellipsoids, mapping the 3D ellipsoid to the imaging plane of a rendering camera to obtain a 2D ellipse; determining the circumscribed rectangle of the 2D ellipse; using the circumscribed rectangle of the 2D ellipse as the shading area corresponding to the 3D ellipsoid, and rendering all pixel points within the shading area.

[0006] In an alternative manner, determining the circumscribed rectangle of the 2D ellipse; using the circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and rendering all pixel points within the coloring area further includes: using the major axis of the 2D ellipse as the long side of the rectangle and the minor axis of the 2D ellipse as the width of the rectangle to construct the minimum circumscribed rectangle of the 2D ellipse, wherein the long side of the minimum circumscribed rectangle is parallel to the major axis of the 2D ellipse; using the minimum circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and determining the colors of all the pixel points within the coloring area.

[0007] In an alternative manner, rendering all pixel points within the coloring area further includes: traversing each pixel point within the coloring area to determine whether each pixel point is located within any one of the 2D ellipses; when the pixel point is located within any one of the 2D ellipses, using the pixel point located within any one of the 2D ellipses as a target pixel point, and using all the 3D ellipsoids corresponding to all the 2D ellipses within which the target pixel point is located as target 3D ellipsoids; obtaining the total number of the target 3D ellipsoids, the color of each target 3D ellipsoid, the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point, and the distance from each target 3D ellipsoid to the rendering camera; determining the color contribution weight of each target 3D ellipsoid to the target pixel point according to the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point and the distance from each target 3D ellipsoid to the rendering camera; determining the color of the target pixel point according to the total number of the target 3D ellipsoids, the color of each target 3D ellipsoid, the color contribution weight of each target 3D ellipsoid to the target pixel point, and the preset background color at the target pixel point.

[0008] In an alternative manner, determining the color of the target pixel point according to the total number of the target 3D ellipsoids, the color of each target 3D ellipsoid, the color contribution weight of each target 3D ellipsoid to the target pixel point, and the preset background color at the target pixel point further includes: determining the color of the target pixel point according to the following formula: ; where C represents the color of the target pixel point, N represents the total number of the target 3D ellipsoids, c i represents the color of the i-th target 3D ellipsoid among all the target 3D ellipsoids, represents the transparency at the position on the i-th target 3D ellipsoid among all the target 3D ellipsoids that has a mapping relationship with the target pixel point, d iRepresents the distance from the i-th target 3D ellipsoid among all the target 3D ellipsoids to the rendering camera, w(d i , ) represents the color contribution weight of the i-th target 3D ellipsoid among all the target 3D ellipsoids to the target pixel point, and c0 represents the preset background color at the target pixel point.

[0009] In an optional manner, rendering all the pixel points in the coloring area further includes: when the pixel point is not located within any of the 2D ellipses, taking the pixel point not located within any of the 2D ellipses as a non-target pixel point; when it is determined that there is a preset background color at the non-target pixel point, taking the preset background color at the non-target pixel point as the color of the non-target pixel point.

[0010] In an optional manner, after taking the pixel point not located within any of the 2D ellipses as a non-target pixel point when the pixel point is not located within any of the 2D ellipses, rendering all the pixel points in the coloring area further includes: when it is determined that there is no preset background color at the non-target pixel point, determining the rendering camera background color at the non-target pixel point and taking the rendering camera background color at the non-target pixel point as the color of the non-target pixel point.

[0011] In an optional manner, determining the color contribution weight of each target 3D ellipsoid at the target pixel point according to the transparency of each target 3D ellipsoid at the target pixel point and the distance from each target 3D ellipsoid at the target pixel point to the rendering camera further includes: determining the color contribution weight of each target 3D ellipsoid at the target pixel point by using a preset method according to the transparency of each target 3D ellipsoid at the target pixel point and the distance from each target 3D ellipsoid at the target pixel point to the rendering camera, where the preset method is the direct weight accumulation method, the exponential weight accumulation method or the linear correction weight accumulation method.

[0012] According to another aspect of the embodiments of the present application, a 3D Gaussian splash rendering device is provided. The device includes: an ellipsoid determination module for determining a set of 3D ellipsoids corresponding to a 3D model in 3D Gaussian splash, where the set of 3D ellipsoids includes a plurality of 3D ellipsoids; an ellipsoid projection module for mapping each 3D ellipsoid in the set of 3D ellipsoids to the imaging plane of a rendering camera to obtain a 2D ellipse; a rectangle determination module for determining the circumscribed rectangle of the 2D ellipse; and a pixel rendering module for using the circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid and rendering all pixel points within the coloring area.

[0013] According to another aspect of the embodiments of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the foregoing 3D Gaussian splash rendering method.

[0014] According to still another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the foregoing 3D Gaussian splash rendering method is implemented.

[0015] In the embodiments of the present application, by using the circumscribed rectangle of the 2D ellipse obtained by mapping each 3D ellipsoid on the imaging plane as the coloring area, the range of the coloring area in the present application is smaller than the 3-sigma range, thereby reducing invalid color calculations, saving fragment shader computing resources, and improving the rendering speed.

[0016] The above description is only an overview of the technical solutions of the embodiments of the present application. In order to be able to understand the technical means of the embodiments of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features, and advantages of the embodiments of the present application more obvious and understandable, the following specifically illustrates the embodiments of the present application. Description of the Drawings

[0017] The drawings are only used to illustrate the embodiments and are not considered as a limitation to the present application. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0018] Figure 1 A flowchart showing the 3D Gaussian splash rendering method provided by the embodiments of the present application is shown;

[0019] Figure 2 A sub-step flowchart showing the 3D Gaussian splash rendering method provided by the embodiments of the present application is shown;

[0020] Figure 3 A schematic diagram of the coloring area of the 3D Gaussian splash rendering method provided by the embodiments of the present application is shown;

[0021] Figure 4 Shows a flowchart of a sub-step of the 3D Gaussian splash rendering solution provided by an embodiment of the present application;

[0022] Figure 5 Shows a schematic diagram of the projection of a 3D ellipsoid of the 3D Gaussian splash rendering method provided by an embodiment of the present application;

[0023] Figure 6 Shows a flowchart of a sub-step of the 3D Gaussian splash rendering method provided by an embodiment of the present application;

[0024] Figure 7 Shows a schematic diagram of the structure of the 3D Gaussian splash rendering device provided by an embodiment of the present application;

[0025] Figure 8 Shows a schematic diagram of the structure of the electronic device provided by an embodiment of the present application. Detailed implementation manners

[0026] Hereinafter, exemplary embodiments of the present application will be described in more detail with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0027] Recently, in the field of 3D scene reconstruction, the 3D Gaussian splash (hereinafter referred to as 3DGS) technology has made significant progress based on Neural Radiance Fields (NeRF), achieving high-fidelity details while maintaining low scene complexity, and also having high real-time rendering performance. 3DGS uses rasterization instead of the computationally expensive ray tracing technology for rendering, greatly improving the real-time rendering performance. During the 3DGS rendering process, the Fragment Shader part consumes about 70% of the performance, and a large part of the performance of the Graphics Processing Unit (GPU) is consumed in the Fragment Shader calculation.

[0028] However, since each 3D ellipsoid in 3DGS rendering uses a 3-sigma boundary during projection, there are many invalid shading areas within the projection range, increasing the amount of calculation in the invalid shading areas, resulting in a large number of invalid calculations and wasting the calculation resources of the Fragment Shader. This situation increases the performance requirements for the GPU and will affect the rendering speed when the GPU performance is fixed.

[0029] Based on this, the present application proposes a 3D Gaussian splash rendering method. By using the circumscribed rectangle of the 2D ellipse obtained by mapping each 3D ellipsoid onto the imaging plane as the coloring area, the range of the coloring area in the present application is smaller than the 3-sigma range, thereby reducing invalid color calculations, saving fragment shader computing resources, and improving the rendering speed.

[0030] Figure 1 The flowchart of the 3D Gaussian splash rendering method provided by an embodiment of the present application is shown. This method is executed by an electronic device. The electronic device includes a GPU or a high-performance graphics card that supports HDR (High Dynamic Range Imaging) rendering, an AR (Augmented Reality) / VR (Virtual Reality) headset device, a high-performance GPU workstation, a robot, or an in-vehicle computing unit for autonomous driving, etc.

[0031] As Figure 1 shown, the method includes the following steps.

[0032] Step S110: Determine the set of 3D ellipsoids corresponding to the 3D model in the 3D Gaussian splash. The set of 3D ellipsoids includes multiple 3D ellipsoids.

[0033] 3D Gaussian splash is a rendering method based on Gaussian ellipsoids, used to efficiently represent and render 3D scenes. In 3D space, the Gaussian distribution can be represented as a three-dimensional Gaussian ellipsoid. The Gaussian ellipsoid is the basic unit of the 3D Gaussian splash technology, used to represent a local 3D area in the scene. Each ellipsoid is defined by the following parameters. The position of the ellipsoid (the center point of the ellipsoid) is represented by the coordinates μ=(x,y,z) in three-dimensional space. The covariance matrix (∑) is used to describe the shape (anisotropy), size, and direction of the ellipsoid. The color (usually RGB or spherical harmonics) represents the color attribute of the ellipsoid. The transparency (α) represents the degree of transparency at a certain position of the ellipsoid.

[0034] In some embodiments, record a video or take a set of photos from different angles, estimate the point cloud of the photos using related technologies (such as Structure from Motion, SFM), or directly randomly initialize a set of point clouds. Each point in the point cloud represents a three-dimensional Gaussian distribution. In addition to the position of the point (which can be represented by the mean μ), the parameters at each point also include covariance, transparency, and color (spherical harmonic coefficients). In 3DGS, each point in the point cloud can be intuitively understood as a 3D Gaussian ellipsoid (hereinafter referred to as a 3D ellipsoid).

[0035] The shape of a 3D ellipsoid is determined by the mean (μ) and the covariance matrix (Σ). The mean represents the center position of the 3D ellipsoid, while the covariance matrix describes the shape, size, and orientation of the 3D ellipsoid. The covariance matrix is a symmetric and positive-definite matrix that can be decomposed into a scaling matrix (S) and a rotation matrix (R). The scaling matrix determines the degree of diffusion of the 3D ellipsoid in different directions, and the rotation matrix determines the orientation of the ellipsoid. The 3D ellipsoid can be anisotropic (with different degrees of diffusion in different directions) or isotropic (with the same degree of diffusion in all directions). The 3D Gaussian ellipsoid is the core of the 3D Gaussian splatting technique. Through flexible parameterization (position, covariance, color, transparency) and differentiable optimization, it can achieve efficient 3D scene representation and rendering. The anisotropic and unstructured characteristics of 3DGS make it show significant advantages in complex geometry reconstruction and real-time rendering.

[0036] Step S120: For each 3D ellipsoid in the 3D ellipsoid set, map the 3D ellipsoid to the imaging plane of the rendering camera to obtain a 2D ellipse.

[0037] Each 3D ellipsoid in the 3D ellipsoid set obtained in step S110 is projected onto the imaging plane of the rendering camera along a specific angle. This process is called "splatting". The projection process utilizes the mathematical closure of the Gaussian distribution to ensure that the projection result is still a Gaussian distribution. The projection process involves converting the covariance matrix of the 3D ellipsoid into the covariance matrix of the 2D ellipse. Through Taylor expansion and transformation of the covariance matrix, the shape and position of the projected 2D ellipse can be calculated.

[0038] The rendering of 3D Gaussian splatting is not based on ray tracing. Instead, after projecting the 3D ellipsoids onto a 2D plane, the pixel colors on the 2D plane are calculated through rasterization and blending. The color of each pixel is determined by the weighted superposition of the projections of multiple 3D ellipsoids at that position, rather than through geometric intersection calculations. This method significantly optimizes the computational efficiency while ensuring the visual effect.

[0039] Step S130: Determine the bounding rectangle of the 2D ellipse.

[0040] Step S140: Use the bounding rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and render all the pixel points within the coloring area.

[0041] As Figure 2 shown, Figure 2 shows a sub-step flowchart of the 3D Gaussian splatting rendering method provided by an embodiment of the present application. Steps S130 - S140 further include the following steps.

[0042] Step S131: Construct the minimum circumscribed rectangle of the 2D ellipse with the major axis of the 2D ellipse as the long side of the rectangle and the minor axis of the 2D ellipse as the width of the rectangle, where the long side of the minimum circumscribed rectangle is parallel to the major axis of the 2D ellipse.

[0043] Step S141: Use the minimum circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and determine the colors of all pixel points within the coloring area.

[0044] As Figure 3 shown, Figure 3 it shows a schematic diagram of the coloring area of the 3D Gaussian splash rendering method provided by the embodiment of the present application.

[0045] In the related art, after the 3D ellipsoid is projected onto the imaging plane to obtain the corresponding 2D ellipse, usually the 3-sigma area in this imaging plane is used as a color mixing area (i.e., the coloring area, such as Figure 3 S in). In probability distribution, the 3-sigma interval covers approximately 99.7% of the probability mass. For a 2D Gaussian distribution, the 3-sigma equiprobability line is a circle (or an ellipse), and this circle is the visual boundary of the projection. During rendering, a surrounding area (such as an axis-aligned bounding box or a pixel-level coverage test) is generated according to the parameters of this visual boundary (such as the diameter of the circle or the major and minor axes of the ellipse), and this surrounding area is the 3-sigma area. The 3-sigma area can cover the 2D ellipse corresponding to the 3-sigma area. For each pixel point within the 3-sigma area, the color and transparency contributions of the 3D ellipsoid corresponding to the 2D ellipse at a certain pixel point within the 3-sigma area can be calculated.

[0046] However, using the 3-sigma area as the coloring area will cause the pixel colors in a large number of invalid areas (such as the areas within the 3-sigma area S and outside the 2D ellipse) to be calculated invalidly, because there may be no color contribution from the 3D ellipsoid to the pixel points in this invalid area, resulting in a large number of invalid calculations, wasting the computing resources of the fragment shader and affecting the rendering speed. To solve the above problems, in the 3D Gaussian splash rendering method provided by the embodiment of the present application, for each 2D ellipse, the circumscribed rectangle of the 2D ellipse is used as the coloring area (such as Figure 3In S’), the colored area of the circumscribed rectangle can exactly cover the area where the 2D ellipse is located, so that by calculating the color of each pixel point in the colored area of the circumscribed rectangle, the color of each pixel point in the projection of the 3D ellipsoid corresponding to the 2D ellipse in the colored area onto the imaging plane can be obtained. And the colored area of the circumscribed rectangle corresponding to the 2D ellipse is usually smaller than the 3-sigma area corresponding to the 2D ellipse, so the calculation of invalid colors in the invalid area can be greatly reduced.

[0047] In some embodiments, in order to further reduce the range of the colored area to save computational resources on the basis of covering the projection range of the 2D ellipse of a 3D ellipsoid, the minimum circumscribed rectangle of the 2D ellipse can also be used as the colored area. It is easy to understand that for an ellipse, there can be countless circumscribed rectangles. When the long side of the circumscribed rectangle of the 2D ellipse is equal to the major axis of the 2D ellipse, the short side of the circumscribed rectangle is equal to the minor axis of the 2D ellipse, and the long side of the circumscribed rectangle is parallel to the major axis of the 2D ellipse, the circumscribed rectangle is the minimum circumscribed rectangle of the 2D ellipse. At this time, the area of the minimum circumscribed rectangle is the smallest among the areas of all circumscribed rectangles of the 2D ellipse. Therefore, using the minimum circumscribed rectangle as the colored area of the 2D ellipse projection of the 3D ellipsoid on the imaging plane can minimize the area of the invalid area, thereby reducing the calculation of invalid colors in the invalid area, further saving the computational resources of the fragment shader, and improving the rendering speed.

[0048] As Figure 4 shown, Figure 4 shows a sub-step flowchart of the 3D Gaussian splash rendering scheme provided by the embodiment of the present application. Rendering all pixel points in the colored area in step S140 further includes the following steps.

[0049] Step S142: Traverse each pixel point in the colored area to determine whether each pixel point is located within any 2D ellipse.

[0050] Step S143: When a pixel point is located within any 2D ellipse, the pixel point located within any 2D ellipse is used as the target pixel point, and all 3D ellipsoids corresponding to all 2D ellipses where the target pixel point is located are used as the target 3D ellipsoids.

[0051] In step S110, the 2D ellipse obtained by mapping the 3D ellipsoid to the imaging plane of the rendering camera is the 2D ellipse corresponding to the 3D ellipsoid. Each 3D ellipsoid is mapped to the imaging plane of the rendering camera, and the same number of 2D ellipses as the number of 3D ellipsoids are obtained, that is, multiple 2D ellipses are finally formed on the imaging plane. For each pixel point in each coloring area, the pixel point may be located within one or more 2D ellipses, or may not be located within any 2D ellipse. In the embodiment of the present application, the pixel points located within any 2D ellipse are determined as target pixel points, and then all the target 3D ellipsoids required for rendering the target pixel points are determined.

[0052] As Figure 5 shown, Figure 5 Fig. shows a schematic projection diagram of the 3D ellipsoids of the 3D Gaussian splash rendering method provided by the embodiment of the present application.

[0053] In 3DGS, the centers of the 3D ellipsoids are not always arranged in sequence along the same straight line, but may exist at any position in the viewing space (3D three-dimensional space). In a specific implementation, as Figure 5 , the projection of the 3D ellipsoid A on the imaging plane Z of the rendering camera is the 2D ellipse a, and the minimum circumscribed rectangle of the 2D ellipse a is A'. The projection of the 3D ellipsoid B on the imaging plane Z of the rendering camera is the 2D ellipse b, and the minimum circumscribed rectangle of the 2D ellipse b is B'. The projection of the 3D ellipsoid C on the imaging plane Z of the rendering camera is the 2D ellipse c, and the minimum circumscribed rectangle of the 2D ellipse c is C'. In the imaging plane Z, for the pixel point p within the coloring area defined by the minimum circumscribed rectangle C', it is located within both the 2D ellipse c and the 2D ellipse b. Therefore, the pixel point p is used as the target pixel point, and the 3D ellipsoid B and the 3D ellipsoid C corresponding to the 2D ellipse b and the 2D ellipse c where the target pixel point p is located are used as the target 3D ellipsoids.

[0054] In actual rendering, usually only the pixel points within the area of the 2D ellipse projection need to perform color calculation. Determining the target pixel points that need to perform color calculation in advance can avoid invalid calculation of the pixel points not within the area of the 2D ellipse projection, further saving computing resources.

[0055] Step S144: Obtain the total number of target 3D ellipsoids, the color of each target 3D ellipsoid, the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point, and the distance from each target 3D ellipsoid to the rendering camera.

[0056] In some embodiments, the distance from the target 3D ellipsoid to the rendering camera is the straight-line distance from the center of the target 3D ellipsoid to the rendering camera. For each target 3D ellipsoid, after obtaining the distance d from the target 3D ellipsoid to the rendering camera, it is also necessary to obtain the transparency at the position on the target 3D ellipsoid that has a mapping relationship with the target pixel point.

[0057] In some embodiments, the transparency can generally be determined according to the distance between the target pixel point and the center of the 2D elliptical projection of the target 3D ellipsoid on the imaging plane. The closer the target pixel point is to the center of the 2D ellipse, the greater the transparency (i.e., the less transparent) of the 3D ellipsoid corresponding to the 2D ellipse at the position having a mapping relationship with the target pixel point. The farther the target pixel point is from the center of the 2D ellipse (i.e., the closer it is to the edge of the 2D ellipse), the smaller the transparency (i.e., the more transparent) of the 3D ellipsoid corresponding to the 2D ellipse at the position having a mapping relationship with the target pixel point.

[0058] Step S145: Determine the color contribution weight of each target 3D ellipsoid to the target pixel point according to the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point and the distance from each target 3D ellipsoid to the rendering camera.

[0059] Step S146: Determine the color of the target pixel point according to the total number of target 3D ellipsoids, the color of each target 3D ellipsoid, the color contribution weight of each target 3D ellipsoid to the target pixel point, and the preset background color at the target pixel point.

[0060] Step S146 further includes: determining the color of the target pixel point according to the following formula:

[0061] .

[0062] Where C represents the color of the target pixel point, N represents the total number of target 3D ellipsoids, c i represents the color of the i-th target 3D ellipsoid among all target 3D ellipsoids, represents the transparency at the position on the i-th target 3D ellipsoid among all target 3D ellipsoids that has a mapping relationship with the target pixel point, d i represents the distance from the i-th target 3D ellipsoid among all target 3D ellipsoids to the rendering camera, w(d i , ) represents the color contribution weight of the i-th target 3D ellipsoid among all target 3D ellipsoids to the target pixel point, and c0 represents the preset background color at the target pixel point.

[0063] For semi-transparent scenes or models, all current 3DGS renderings use anisotropic 3D ellipsoids to represent the scene, and utilize their differentiable characteristics to project the 3D ellipsoids onto a two-dimensional plane for rasterization, and then calculate the colors of the pixels within the shaded area through a semi-transparent blending formula (such as Formula 1). In related technologies, 3DGS rendering methods all require sorting the 3D ellipsoids relative to the rendering camera from far to near according to the distance from the 3D ellipsoids to the rendering camera. The 3D ellipsoids are usually arranged by means of line-of-sight tracing. For example, the centers of each 3D ellipsoid are arranged along the rays emitted from the rendering camera and pointing to the two-dimensional plane (such as the imaging plane of the rendering camera).

[0064] (Formula 1)

[0065] Formula 1 indicates that the final color C at a certain pixel point is the weighted sum of the colors of all 3D ellipsoids that have a mapping relationship with this pixel point. The weight of each color is determined by its transparency α and the cumulative product of the transparencies of all previous (farther from the rendering) 3D ellipsoids. This blending method needs to ensure the correct occlusion relationship between the 3D ellipsoids, that is, the 3D ellipsoids closer to the rendering camera will be partially or completely occluded by the 3D ellipsoids farther from the rendering camera.

[0066] However, since Formula 1 does not satisfy the commutative law, that is, it is necessary to calculate the color / transparency contribution of each of all 3D ellipsoids that have a mapping relationship with a certain pixel point in a certain order in sequence to ensure the correct result. In order to perform semi-transparent blending, it is necessary to render from back to front according to the distance between the 3D ellipsoids and the rendering camera. Therefore, it is necessary to sort the 3D ellipsoids first. Because the blending of the transparencies of multiple 3D ellipsoids needs to consider the occlusion relationship between these 3D ellipsoids, only when the 3D ellipsoids are sorted according to the distance to the rendering camera can the contribution of each 3D ellipsoid to the final color of a certain pixel point on the two-dimensional plane be correctly calculated.

[0067] However, for large scenes, where the number of 3D ellipsoids is very large (the order of magnitude may be from millions to hundreds of millions), sorting all the 3D ellipsoids will bring a large performance overhead and affect the rendering speed. To solve this problem, in the 3D Gaussian splash rendering method of this application, the final color of the target pixel point is determined according to the color of the target 3D ellipsoid and its weight of the color contribution to the target pixel point.

[0068] The weight w(d i , ) of each target 3D ellipsoid to the color contribution of the target pixel point can be understood as a function of the distance d i and the transparency The weight function. In some embodiments, the distance d from the i-th target 3D ellipsoid in the target 3D ellipsoids to the rendering camera i The smaller it is, the closer the target 3D ellipsoid is to the rendering camera. At the position on the i-th target 3D ellipsoid that has a mapping relationship with the target pixel point, the transparency Under the same circumstances, the weight w(d i , ) should be larger. The transparency at the position on the i-th target 3D ellipsoid that has a mapping relationship with the target pixel point The smaller it is, the more transparent the target 3D ellipsoid is. Under the same distance d from the i-th target 3D ellipsoid to the rendering camera i , the weight w(d i , ) should be smaller.

[0069] In some embodiments, w(d i , ) can be determined in various ways, such as by direct weight accumulation, exponential weight accumulation, and linear correction weight accumulation to determine the weight w(d i , ). In other embodiments, w(d i , ) can also be determined by other related technologies, and this application does not limit this.

[0070] Then, the color of the target pixel point is determined according to the following formula 2.

[0071] (Formula 2)

[0072] Indicates that the color contribution of the background color c0 to the final target pixel color is related to the transparency of all target 3D ellipsoids. The background color c0 can be preset. The value range of is [, 1], where 0 means completely transparent and 1 means completely opaque. If the transparency at the positions on all target 3D ellipsoids that have a mapping relationship with the target pixel point are all 0, then the color contribution of the background color to the final color of the target pixel point is the largest. If the transparency at the position on one target 3D ellipsoid that has a mapping relationship with the target pixel point is 1, then the color contribution of the background color to the final color of the target pixel point is 0.

[0073] For example, referring to Figure 5 , Figure 5The target pixel point o in it is inside the 2D ellipse b, and the target pixel point o is at the boundary of the 2D ellipse b. The transparency at the position on the target 3D ellipsoid B that has a mapping relationship with the target pixel point o is 0, that is, N = 1 in Formula 2, = 0, then according to Formula 2, it can be calculated that the calculation result of the color C at the target pixel point o at this time is the background color c0.

[0074] Formula 2 is in the form of the sum of polynomials and satisfies the commutative law, that is, the color calculation process of the target pixel point satisfies the commutative law. Therefore, the result of the color C of the target pixel point calculated according to the foregoing method and Formula 2 is independent of the arrangement order of the target 3D ellipsoids, thereby eliminating the requirement for sorting the target 3D ellipsoids during 3DGS rendering, saving the calculation amount and data processing amount, and improving the rendering speed.

[0075] As Figure 6 shown, Figure 6 shows a sub-step flowchart of the 3D Gaussian splash rendering method provided by an embodiment of the present application. Rendering all pixel points in the coloring area in step S140 further includes the following steps.

[0076] Step S147: When the pixel point is not inside any 2D ellipse, the pixel point that is not inside any 2D ellipse is used as a non-target pixel point.

[0077] Step S148: When it is determined that there is a preset background color at the non-target pixel point, the preset background color at the non-target pixel point is used as the color of the non-target pixel point.

[0078] As described above, for each pixel point in each coloring area, this pixel point may also not be inside any 2D ellipse. For such a pixel point, the embodiment of the present application determines it as a non-target pixel point, and then uses the preset background color at this pixel point for rendering when rendering this pixel point.

[0079] In a specific implementation, referring to Figure 5 , the pixel point q is not inside any 2D ellipse. The area where the pixel point q is located is usually an invalid area where no 3D ellipsoid has a color contribution to this pixel point. Therefore, the pixel point q is used as a non-target pixel point. For the non-target pixel point q, it is also necessary to determine whether there is a preset background color at this non-target pixel point q. If there is a preset background color at the non-target pixel point q, then directly use the preset background color at this non-target pixel point q as the color of the non-target pixel point q, thereby reducing the color calculation in the invalid area, greatly saving the data processing amount, saving computing resources, and improving the rendering speed.

[0080] After step S147, the method further includes: when it is determined that there is no preset background color at a non-target pixel point, determining the rendering camera background color at the non-target pixel point, and using the rendering camera background color at the non-target pixel point as the color of the non-target pixel point.

[0081] In the foregoing specific embodiment, for Figure 5 the non-target pixel point q, if there is no preset background color at the non-target pixel point q, directly using the rendering camera background color at the non-target pixel point q as the color of the non-target pixel point q, where the rendering camera background color is pre-set, thereby reducing color calculation in the invalid area, greatly saving the amount of data processing, saving computing resources, and improving the rendering speed.

[0082] As Figure 7 shown, an embodiment of the present application further provides a 3D Gaussian splash device. The device 200 includes: an ellipsoid determination module 210, an ellipsoid projection module 220, a rectangle determination module 230, and a pixel rendering module 240.

[0083] The ellipsoid determination module 210 is configured to determine a set of 3D ellipsoids corresponding to a 3D model in 3D Gaussian splash. The set of 3D ellipsoids includes a plurality of 3D ellipsoids.

[0084] The ellipsoid projection module 220 is configured to, for each 3D ellipsoid in the set of 3D ellipsoids, map the 3D ellipsoid to the imaging plane of the rendering camera to obtain a 2D ellipse.

[0085] The rectangle determination module 230 is configured to determine the circumscribed rectangle of the 2D ellipse.

[0086] The pixel rendering module 240 is configured to use the circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and render all pixel points in the coloring area.

[0087] In an alternative manner, the rectangle determination module 230 and the pixel rendering module 240 are further configured to use the major axis of the 2D ellipse as the long side of the rectangle and the minor axis of the 2D ellipse as the width of the rectangle to construct the minimum circumscribed rectangle of the 2D ellipse, where the long side of the minimum circumscribed rectangle is parallel to the major axis of the 2D ellipse; using the minimum circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and determining the colors of all pixel points in the coloring area.

[0088] In an alternative manner, the pixel rendering module 240 is further configured to traverse each pixel point within the shaded area to determine whether each pixel point lies within any 2D ellipse; when a pixel point lies within any 2D ellipse, the pixel point that lies within any 2D ellipse is used as a target pixel point, and all 3D ellipsoids corresponding to all 2D ellipses in which the target pixel point lies are used as target 3D ellipsoids; obtain the total number of target 3D ellipsoids, the color of each target 3D ellipsoid, the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point, and the distance from each target 3D ellipsoid to the rendering camera; determine the color contribution weight of each target 3D ellipsoid to the target pixel point according to the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point and the distance from each target 3D ellipsoid to the rendering camera; and determine the color of the target pixel point according to the total number of target 3D ellipsoids, the color of each target 3D ellipsoid, the color contribution weight of each target 3D ellipsoid to the target pixel point, and the preset background color at the target pixel point.

[0089] In an alternative manner, the pixel rendering module 240 is further configured to determine the color of the target pixel point according to the following formula:

[0090] ;

[0091] where C represents the color of the target pixel point, N represents the total number of target 3D ellipsoids, c i represents the color of the i-th target 3D ellipsoid among all target 3D ellipsoids, represents the transparency at the position on the i-th target 3D ellipsoid among all target 3D ellipsoids that has a mapping relationship with the target pixel point, d i represents the distance from the i-th target 3D ellipsoid among all target 3D ellipsoids to the rendering camera, w(d i , ) represents the color contribution weight of the i-th target 3D ellipsoid among all target 3D ellipsoids to the target pixel point, and c0 represents the preset background color at the target pixel point.

[0092] In an alternative manner, when a pixel point does not lie within any 2D ellipse, the pixel rendering module 240 is further configured to use the pixel point that does not lie within any 2D ellipse as a non-target pixel point; when it is determined that there is a preset background color at the non-target pixel point, the preset background color at the non-target pixel point is used as the color of the non-target pixel point.

[0093] In an alternative manner, the pixel rendering module 240 is further configured to determine the rendering camera background color at a non-target pixel point when it is determined that there is no preset background color at the non-target pixel point, and use the rendering camera background color at the non-target pixel point as the color of the non-target pixel point.

[0094] In an alternative manner, the pixel rendering module 240 is further configured to determine the color contribution weight of each target 3D ellipsoid at the target pixel point according to the transparency of each target 3D ellipsoid at the target pixel point and the distance from each target 3D ellipsoid at the target pixel point to the rendering camera, where the preset method is the direct weight accumulation method, the exponential weight accumulation method, or the linear correction weight accumulation method.

[0095] In the embodiment of the present application, the rectangular determination module 230 uses the circumscribed rectangle of the 2D ellipse in which each 3D ellipsoid is mapped on the imaging plane as the coloring area, so that the range of the coloring area of the present application is smaller than the 3-sigma range, thereby reducing invalid color calculations, saving fragment shader calculation resources, and improving the rendering speed.

[0096] Figure 8 The structural schematic diagram of the electronic device provided by the embodiment of the present application is shown. The specific implementation of the electronic device is not limited in the specific embodiment of the present application.

[0097] [[ID=1 = 13]]As Figure 8 shown, the electronic device may include: a processor 301 and a memory 302.

[0098] Among them, the memory 302 is used to store the computer program 303. The memory 302 may include a high-speed RAM memory, and may also include non-volatile memory, such as at least one disk memory in the example. The computer program 303 may include computer executable instructions.

[0099] The processor 301 is configured to execute the computer program 303 to implement the above-mentioned 3D Gaussian splash rendering method embodiment.

[0100] The processor 301 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiment of the present application. One or more processors included in the electronic device may be of the same type of processor, such as one or more CPUs; or may be of different types of processors, such as one or more CPUs and one or more ASICs.

[0101] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned embodiment of the 3D Gaussian splash rendering method.

[0102] In several embodiments provided by the present application, if any function is implemented in the form of a software functional module / unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, all or part of the technical solution of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be an electronic device such as a personal computer or a server) to execute all or part of the steps of the methods in the various embodiments of the present application. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc that can store computer program code.

[0103] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings provided herein. The structure required to construct such a system will be apparent from the above description. In addition, the embodiments of the present application are not directed to any specific programming language. It should be understood that the content of the present application described herein can be implemented using various programming languages, and the description of a specific language above is for disclosing the best mode of the present application.

[0104] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In the claims listing several devices, several units or modules of these devices can be embodied by the same hardware item. The use of the words first, second, and third, etc. does not denote any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

[0105] The above embodiments only illustrate several implementation manners of the present application. The description thereof is relatively specific and detailed, but it should not be construed as a limitation to the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the appended claims.

Claims

1. A 3D Gaussian splash rendering method, characterized in that, The method includes: Determine a set of 3D ellipsoids corresponding to the 3D model in the 3D Gaussian splash, where the set of 3D ellipsoids includes multiple 3D ellipsoids; For each 3D ellipsoid in the set of 3D ellipsoids, map the 3D ellipsoid to the imaging plane of the rendering camera to obtain a 2D ellipse; Determine the circumscribed rectangle of the 2D ellipse; Use the circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and render all pixel points within the coloring area; The rendering of all pixel points within the coloring area further includes: Traverse each pixel point within the coloring area to determine whether each pixel point is within any of the 2D ellipses; When the pixel point is within any of the 2D ellipses, use the pixel point within any of the 2D ellipses as a target pixel point, and use all the 3D ellipsoids corresponding to all the 2D ellipses in which the target pixel point is located as target 3D ellipsoids; Obtain the total number of the target 3D ellipsoids, the color of each target 3D ellipsoid, the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point, and the distance from each target 3D ellipsoid to the rendering camera; Determine the color contribution weight of each target 3D ellipsoid to the target pixel point according to the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point and the distance from each target 3D ellipsoid to the rendering camera; Determine the color of the target pixel point according to the total number of the target 3D ellipsoids, the color of each target 3D ellipsoid, the color contribution weight of each target 3D ellipsoid to the target pixel point, and the preset background color at the target pixel point.

2. The method according to claim 1, characterized in that, The determination of the circumscribed rectangle of the 2D ellipse; using the circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and rendering all pixel points within the coloring area further includes: Construct the minimum circumscribed rectangle of the 2D ellipse with the major axis of the 2D ellipse as the long side of the rectangle and the minor axis of the 2D ellipse as the width of the rectangle, where the long side of the minimum circumscribed rectangle is parallel to the major axis of the 2D ellipse; Use the minimum circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and determine the colors of all pixel points within the coloring area.

3. The method according to claim 1, characterized in that, The determination of the color of the target pixel point according to the total number of the target 3D ellipsoids, the color of each target 3D ellipsoid, the color contribution weight of each target 3D ellipsoid to the target pixel point, and the preset background color at the target pixel point further includes: Determine the color of the target pixel point according to the following formula: ； Among them, C represents the color of the target pixel point, N represents the total number of the target 3D ellipsoids, and c i represents the color of the i-th target 3D ellipsoid among all the target 3D ellipsoids, represents the transparency at the position where the upper part of the i-th target 3D ellipsoid among all the target 3D ellipsoids has a mapping relationship with the target pixel point, and d i represents the distance from the i-th target 3D ellipsoid among all the target 3D ellipsoids to the rendering camera, w(d i , ) represents the color contribution weight of the i-th target 3D ellipsoid among all the target 3D ellipsoids to the target pixel point, and c0 represents the preset background color at the target pixel point.

4. The method according to claim 1, characterized in that, The rendering of all pixel points within the coloring area further includes: When the pixel point is not within any of the 2D ellipses, use the pixel point not within any of the 2D ellipses as a non-target pixel point; When it is determined that there is a preset background color at the non-target pixel point, the preset background color at the non-target pixel point is used as the color of the non-target pixel point.

5. The method according to claim 4, wherein After, when the pixel point is not located within any of the 2D ellipses, the pixel point not located within any of the 2D ellipses is used as a non-target pixel point, the method further includes: When it is determined that there is no preset background color at the non-target pixel point, determine the rendering camera background color at the non-target pixel point, and use the rendering camera background color at the non-target pixel point as the color of the non-target pixel point.

6. The method according to claim 1, wherein The determining, according to the transparency of each target 3D ellipsoid at the target pixel point and the distance from each target 3D ellipsoid at the target pixel point to the rendering camera, the color contribution weight of each target 3D ellipsoid at the target pixel point, further includes: According to the transparency of each target 3D ellipsoid at the target pixel point and the distance from each target 3D ellipsoid at the target pixel point to the rendering camera, use a preset method to determine the color contribution weight of each target 3D ellipsoid at the target pixel point, where the preset method is a direct weight accumulation method, an exponential weight accumulation method, or a linear correction weight accumulation method.

7. A 3D Gaussian splash rendering device, characterized in that, The apparatus includes: An ellipsoid determination module, configured to determine a set of 3D ellipsoids corresponding to a 3D model in a 3D Gaussian splash, where the set of 3D ellipsoids includes a plurality of 3D ellipsoids; An ellipsoid projection module, configured to, for each 3D ellipsoid in the set of 3D ellipsoids, map the 3D ellipsoid to the imaging plane of the rendering camera to obtain a 2D ellipse; A rectangle determination module, configured to determine the circumscribed rectangle of the 2D ellipse; A pixel rendering module, configured to use the circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and render all pixel points within the coloring area; The rendering all pixel points within the coloring area further includes: Traverse each pixel point within the coloring area, and determine whether each pixel point is located within any of the 2D ellipses; When the pixel point is located within any of the 2D ellipses, use the pixel point located within any of the 2D ellipses as a target pixel point, and use all the 3D ellipsoids corresponding to all the 2D ellipses where the target pixel point is located as target 3D ellipsoids; Obtain the total number of target 3D ellipsoids, the color of each target 3D ellipsoid, the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point, and the distance from each target 3D ellipsoid to the rendering camera; According to the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point and the distance from each target 3D ellipsoid to the rendering camera, determine the color contribution weight of each target 3D ellipsoid to the target pixel point; Determine the color of the target pixel point according to the total number of the target 3D ellipsoids, the color of each of the target 3D ellipsoids, the color contribution weight of each of the target 3D ellipsoids to the target pixel point, and the preset background color at the target pixel point.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory, characterized in that, The processor executes the computer program to implement the 3D Gaussian splash rendering method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the 3D Gaussian splash rendering method according to any one of claims 1 to 6.

Images