3D Gaussian splash rendering method, 3D rendering device, electronic equipment and storage medium

Abstract

The present application relates to the technical field of 3D rendering, and discloses a 3D Gaussian splash rendering method, a 3D Gaussian splash rendering device, an electronic device and a storage medium, the method comprising: determining a 3D ellipsoid set corresponding to a 3D model in 3D Gaussian splash, the 3D ellipsoid set comprising a plurality of 3D ellipsoids; for each of the 3D ellipsoids in the set of 3D ellipsoids, mapping the 3D ellipsoids to an imaging plane of a rendering camera to obtain a 2D ellipse; determining a circumscribed rectangle of the 2D ellipse; and taking a circumscribed rectangle of the 2D ellipse as a coloring area corresponding to the 3D ellipsoid, and rendering all pixel points in the coloring area. According to the invention, each 3D ellipsoid is mapped in the external rectangle of the 2D ellipse in the imaging plane to serve as the coloring area, so that the range of the coloring area is smaller than the 3-sigma range, invalid color calculation is reduced, computing resources of a fragment shader are saved, and the rendering speed is increased.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of 3D rendering technology, and specifically to a 3D Gaussian splatter rendering method, a 3D Gaussian splatter rendering device, an electronic device, and a storage medium. Background Art

[0002] Recently, in the field of 3D scene reconstruction, the 3D Gaussian splatter (hereinafter referred to as 3DGS) technology has made significant progress based on Neural Radiance Fields (NeRF). It achieves high-fidelity details while maintaining low scene complexity and also has high real-time rendering performance. 3DGS uses rasterization instead of the computationally expensive ray tracing technology for rendering, which greatly improves the real-time rendering performance. During the 3DGS rendering process, the fragment shader part consumes about 70% of the performance, and a large part of the performance of the Graphics Processing Unit (GPU) is consumed in the fragment shader calculation.

[0003] However, since each 3D ellipsoid in 3DGS rendering uses a 3-sigma boundary during projection, there are many invalid shading areas within the projection range, increasing the computational amount within the invalid shading areas, resulting in a large amount of invalid calculations, wasting fragment shader computing resources, increasing the requirements for the GPU, and affecting the rendering speed. Summary of the Invention

[0004] In view of the above problems, the embodiments of the present application provide a 3D Gaussian splatter rendering method, a 3D Gaussian splatter rendering device, an electronic device, and a storage medium, which are used to solve the problem in the prior art that the shading area corresponding to each 3D ellipsoid in 3DGS is too large, resulting in excessive invalid calculations and wasting GPU computing resources.

[0005] According to one aspect of the embodiments of the present application, a 3D Gaussian splatter method is provided. The method includes: determining a set of 3D ellipsoids corresponding to a 3D model in 3D Gaussian splatter, where the set of 3D ellipsoids includes multiple 3D ellipsoids; for each 3D ellipsoid in the set of 3D ellipsoids, mapping the 3D ellipsoid to the imaging plane of a rendering camera to obtain a 2D ellipse; determining the circumscribed rectangle of the 2D ellipse; using the circumscribed rectangle of the 2D ellipse as the shading area corresponding to the 3D ellipsoid, and rendering all pixel points within the shading area.

[0006] In an alternative manner, determining the circumscribed rectangle of the 2D ellipse; using the circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and rendering all pixel points within the coloring area further includes: using the major axis of the 2D ellipse as the long side of the rectangle and the minor axis of the 2D ellipse as the width of the rectangle to construct the minimum circumscribed rectangle of the 2D ellipse, wherein the long side of the minimum circumscribed rectangle is parallel to the major axis of the 2D ellipse; using the minimum circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and determining the colors of all the pixel points within the coloring area.

[0007] In an alternative manner, rendering all pixel points within the coloring area further includes: traversing each pixel point within the coloring area to determine whether each pixel point is located within any of the 2D ellipses; when the pixel point is located within any of the 2D ellipses, using the pixel point located within any of the 2D ellipses as the target pixel point, and using all the 3D ellipsoids corresponding to all the 2D ellipses in which the target pixel point is located as the target 3D ellipsoids; obtaining the total number of the target 3D ellipsoids, the color of each target 3D ellipsoid, the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point, and the distance from each target 3D ellipsoid to the rendering camera; determining the color contribution weight of each target 3D ellipsoid to the target pixel point according to the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point and the distance from each target 3D ellipsoid to the rendering camera; determining the color of the target pixel point according to the total number of the target 3D ellipsoids, the color of each target 3D ellipsoid, the color contribution weight of each target 3D ellipsoid to the target pixel point, and the preset background color at the target pixel point.

[0008] In an alternative manner, determining the color of the target pixel point according to the total number of the target 3D ellipsoids, the color of each target 3D ellipsoid, the color contribution weight of each target 3D ellipsoid to the target pixel point, and the preset background color at the target pixel point further includes: determining the color of the target pixel point according to the following formula: ; where C represents the color of the target pixel point, N represents the total number of the target 3D ellipsoids, c i represents the color of the i-th target 3D ellipsoid among all the target 3D ellipsoids, represents the transparency at the position on the i-th target 3D ellipsoid among all the target 3D ellipsoids that has a mapping relationship with the target pixel point, d iRepresents the distance from the i-th target 3D ellipsoid among all the target 3D ellipsoids to the rendering camera, w(d i , ) represents the color contribution weight of the i-th target 3D ellipsoid among all the target 3D ellipsoids to the target pixel point, c 0 Represents the preset background color at the target pixel point.

[0009] In an alternative manner, rendering all the pixel points in the coloring area further includes: when the pixel point is not located within any of the 2D ellipses, taking the pixel point not located within any of the 2D ellipses as a non-target pixel point; when it is determined that there is a preset background color at the non-target pixel point, taking the preset background color at the non-target pixel point as the color of the non-target pixel point.

[0010] In an alternative manner, after taking the pixel point not located within any of the 2D ellipses as a non-target pixel point when the pixel point is not located within any of the 2D ellipses, rendering all the pixel points in the coloring area further includes: when it is determined that there is no preset background color at the non-target pixel point, determining the rendering camera background color at the non-target pixel point and taking the rendering camera background color at the non-target pixel point as the color of the non-target pixel point.

[0011] In an alternative manner, determining the color contribution weight of each target 3D ellipsoid at the target pixel point according to the transparency of each target 3D ellipsoid at the target pixel point and the distance from each target 3D ellipsoid at the target pixel point to the rendering camera further includes: determining the color contribution weight of each target 3D ellipsoid at the target pixel point by using a preset method according to the transparency of each target 3D ellipsoid at the target pixel point and the distance from each target 3D ellipsoid at the target pixel point to the rendering camera, where the preset method is the direct weight accumulation method, the exponential weight accumulation method, or the linear correction weight accumulation method.

[0012] According to another aspect of the embodiments of the present application, a 3D Gaussian splash rendering device is provided. The device includes: an ellipsoid determination module for determining a set of 3D ellipsoids corresponding to a 3D model in 3D Gaussian splash, where the set of 3D ellipsoids includes a plurality of 3D ellipsoids; an ellipsoid projection module for mapping each 3D ellipsoid in the set of 3D ellipsoids to the imaging plane of a rendering camera to obtain a 2D ellipse; a rectangle determination module for determining the circumscribed rectangle of the 2D ellipse; and a pixel rendering module for using the circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid and rendering all pixel points within the coloring area.

[0013] According to another aspect of the embodiments of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the foregoing 3D Gaussian splash rendering method.

[0014] According to yet another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the foregoing 3D Gaussian splash rendering method is implemented.

[0015] In the embodiments of the present application, by using the circumscribed rectangle of the 2D ellipse obtained by mapping each 3D ellipsoid in the imaging plane as the coloring area, the range of the coloring area in the present application is smaller than the 3-sigma range, thereby reducing invalid color calculations, saving fragment shader computing resources, and improving the rendering speed.

[0016] The above description is only an overview of the technical solutions of the embodiments of the present application. In order to be able to understand the technical means of the embodiments of the present application more clearly, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features, and advantages of the embodiments of the present application more obvious and understandable, the following specifically describes the embodiments of the present application. Description of the Drawings

[0017] The drawings are only used to illustrate the embodiments and are not considered to be a limitation of the present application. And throughout the drawings, the same reference numerals are used to represent the same components. In the drawings: Figure 1 A flowchart of the 3D Gaussian splash rendering method provided by the embodiments of the present application is shown; Figure 2 A sub-step flowchart of the 3D Gaussian splash rendering method provided by the embodiments of the present application is shown; Figure 3 A schematic diagram of the coloring area of the 3D Gaussian splash rendering method provided by the embodiments of the present application is shown; Figure 4Shows a sub-step flowchart of the 3D Gaussian splash rendering solution provided by the embodiments of the present application; Figure 5 Shows a schematic diagram of the projection of a 3D ellipsoid of the 3D Gaussian splash rendering method provided by the embodiments of the present application; Figure 6 Shows a sub-step flowchart of the 3D Gaussian splash rendering method provided by the embodiments of the present application; Figure 7 Shows a schematic diagram of the structure of the 3D Gaussian splash rendering device provided by the embodiments of the present application; Figure 8 Shows a schematic diagram of the structure of the electronic device provided by the embodiments of the present application. Detailed implementation manners

[0018] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0019] Recently, in the field of 3D scene reconstruction, the 3D Gaussian splash (hereinafter referred to as 3DGS) technology has made significant progress based on Neural Radiance Fields (NeRF). It achieves high-fidelity details while maintaining low scene complexity and also has high real-time rendering performance. 3DGS uses rasterization instead of the computationally expensive ray tracing technology for rendering, greatly improving the real-time rendering performance. During the 3DGS rendering process, the fragment shader part consumes about 70% of the performance, and a large part of the performance of the Graphics Processing Unit (GPU) is consumed in the fragment shader calculations.

[0020] However, since each 3D ellipsoid in the 3DGS rendering uses a 3-sigma boundary during projection, there are many invalid shading areas within the projection range, increasing the amount of computation in the invalid shading areas, resulting in a large amount of invalid computation and wasting fragment shader computing resources. This situation increases the performance requirements for the GPU and will affect the rendering speed when the GPU performance is fixed.

[0021] Based on this, the present application proposes a 3D Gaussian splash rendering method. By using the circumscribed rectangle of the 2D ellipse obtained by mapping each 3D ellipsoid on the imaging plane as the shading area, the range of the shading area in the present application is smaller than the 3-sigma range, thereby reducing the invalid color calculation, saving the fragment shader computing resources, and improving the rendering speed.

[0022] Figure 1The flowchart of the 3D Gaussian splash rendering method provided by an embodiment of the present application is shown. This method is executed by an electronic device. The electronic device includes a GPU or a high-performance graphics card that supports HDR (High Dynamic Range Imaging) rendering, an AR (Augmented Reality) / VR (Virtual Reality) headset device, a high-performance GPU workstation, a robot, or an in-vehicle computing unit for autonomous driving, etc.

[0023] As Figure 1 shown, the method includes the following steps.

[0024] Step S110: Determine the set of 3D ellipsoids corresponding to the 3D model in the 3D Gaussian splash. The set of 3D ellipsoids includes multiple 3D ellipsoids.

[0025] 3D Gaussian splash is a rendering method based on Gaussian ellipsoids, used to efficiently represent and render 3D scenes. In 3D space, the Gaussian distribution can be represented as a three-dimensional Gaussian ellipsoid. The Gaussian ellipsoid is the basic unit of the 3D Gaussian splash technology, used to represent a local 3D area in the scene. Each ellipsoid is defined by the following parameters. The position of the ellipsoid (the center point of the ellipsoid) is represented by the coordinates μ=(x,y,z) in three-dimensional space. The covariance matrix (∑) is used to describe the shape (anisotropy), size, and direction of the ellipsoid. The color (usually RGB or spherical harmonics) represents the color attribute of the ellipsoid. The transparency (α) represents the degree of transparency at a certain position of the ellipsoid.

[0026] In some embodiments, record a video or take a set of photos from different angles, estimate the point cloud of the photos using related technologies (such as Structure from Motion, SFM), or directly randomly initialize a set of point clouds. Each point in the point cloud represents a three-dimensional Gaussian distribution. In addition to the position of the point (which can be represented by the mean μ), the parameters at each point also include covariance, transparency, and color (spherical harmonic coefficients). In 3DGS, each point in the point cloud can be intuitively understood as a 3D Gaussian ellipsoid (hereinafter referred to as a 3D ellipsoid).

[0027] The shape of a 3D ellipsoid is determined by the mean (μ) and the covariance matrix (Σ). The mean represents the center position of the 3D ellipsoid, while the covariance matrix describes the shape, size, and orientation of the 3D ellipsoid. The covariance matrix is a symmetric and positive definite matrix, which can be decomposed into a scaling matrix (S) and a rotation matrix (R). The scaling matrix determines the degree of diffusion of the 3D ellipsoid in different directions, and the rotation matrix determines the orientation of the ellipsoid. The 3D ellipsoid can be anisotropic (with different degrees of diffusion in different directions) or isotropic (with the same degree of diffusion in all directions). The 3D Gaussian ellipsoid is the core of the 3D Gaussian splatting technique. Through flexible parameterization (position, covariance, color, transparency) and differentiable optimization, it can achieve efficient 3D scene representation and rendering. The anisotropic and unstructured characteristics of 3DGS make it show significant advantages in complex geometry reconstruction and real-time rendering.

[0028] Step S120: For each 3D ellipsoid in the 3D ellipsoid set, map the 3D ellipsoid to the imaging plane of the rendering camera to obtain a 2D ellipse.

[0029] Each 3D ellipsoid in the 3D ellipsoid set obtained in step S110 is projected onto the imaging plane of the rendering camera along a specific angle. This process is called "splatting". The projection process utilizes the mathematical closure of the Gaussian distribution to ensure that the projection result is still a Gaussian distribution. The projection process involves converting the covariance matrix of the 3D ellipsoid into the covariance matrix of the 2D ellipse. Through Taylor expansion and the transformation of the covariance matrix, the shape and position of the projected 2D ellipse can be calculated.

[0030] The rendering of 3D Gaussian splatting is not based on ray tracing. Instead, after projecting the 3D ellipsoids onto a 2D plane, the pixel colors on the 2D plane are calculated through rasterization and blending. The color of each pixel is determined by the weighted superposition of the projections of multiple 3D ellipsoids at that position, rather than through geometric intersection calculations. This method significantly optimizes the computational efficiency while ensuring the visual effect.

[0031] Step S130: Determine the bounding rectangle of the 2D ellipse.

[0032] Step S140: Use the bounding rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and render all the pixel points within the coloring area.

[0033] As Figure 2 shown, Figure 2 shows a sub-step flowchart of the 3D Gaussian splatting rendering method provided by the embodiment of the present application. Steps S130 - S140 further include the following steps.

[0034] Step S131: Construct a minimum circumscribed rectangle of the 2D ellipse with the major axis of the 2D ellipse as the long side of the rectangle and the minor axis of the 2D ellipse as the width of the rectangle, wherein the long side of the minimum circumscribed rectangle is parallel to the major axis of the 2D ellipse.

[0035] Step S141: Use the minimum circumscribed rectangle of the 2D ellipse as the corresponding coloring area of the 3D ellipsoid, and determine the colors of all pixel points within the coloring area.

[0036] As Figure 3 shown, Figure 3 it shows a schematic diagram of the coloring area of the 3D Gaussian splash rendering method provided by the embodiment of the present application.

[0037] In the related art, after the 3D ellipsoid is projected onto the imaging plane to obtain the corresponding 2D ellipse, usually the 3-sigma area within the imaging plane is used as a color mixing area (i.e., the coloring area, such as Figure 3 S in it). In probability distribution, the 3-sigma interval covers approximately 99.7% of the probability mass. For a 2D Gaussian distribution, the 3-sigma equiprobability line is a circle (or an ellipse), and this circle is the visual boundary of the projection. During rendering, a surrounding area (such as an axis-aligned bounding box or a pixel-level coverage test) is generated according to the parameters of this visual boundary (such as the diameter of the circle or the major axis and minor axis of the ellipse), and this surrounding area is the 3-sigma area. The 3-sigma area can cover the 2D ellipse corresponding to the 3-sigma area. For each pixel point within the 3-sigma area, the color and transparency contributions of the 3D ellipsoid corresponding to the 2D ellipse at a certain pixel point within the 3-sigma area can be calculated.

[0038] However, using the 3-sigma area as the coloring area will cause the pixel colors in a large number of invalid areas (such as the areas within the 3-sigma area S and outside the 2D ellipse) to be calculated invalidly, because there may be no color contribution from the 3D ellipsoid to the pixel points in this invalid area, resulting in a large number of invalid calculations, wasting the computing resources of the fragment shader and affecting the rendering speed. To solve the above problems, in the 3D Gaussian splash rendering method provided by the embodiment of the present application, for each 2D ellipse, the circumscribed rectangle of the 2D ellipse is used as the coloring area (such as Figure 3In S’), the colored area of the circumscribed rectangle can exactly cover the area where the 2D ellipse is located, so that by calculating the color of each pixel point in the colored area of the circumscribed rectangle, the color of each pixel point in the projection of the 3D ellipsoid corresponding to the 2D ellipse in the colored area onto the imaging plane can be obtained. Usually, the colored area of the circumscribed rectangle corresponding to the 2D ellipse is smaller than the 3-sigma area corresponding to the 2D ellipse. Therefore, the calculation of invalid colors in the invalid area can be greatly reduced.

[0039] In some embodiments, in order to further reduce the range of the colored area on the basis of covering the projection range of the 2D ellipse of a 3D ellipsoid to save the calculation amount, the minimum circumscribed rectangle of the 2D ellipse can also be used as the colored area. It is easy to understand that for an ellipse, there can be countless circumscribed rectangles. When the long side of the circumscribed rectangle of the 2D ellipse is equal to the major axis of the 2D ellipse, the short side of the circumscribed rectangle is equal to the minor axis of the 2D ellipse, and the long side of the circumscribed rectangle is parallel to the major axis of the 2D ellipse, the circumscribed rectangle is the minimum circumscribed rectangle of the 2D ellipse. At this time, the area of the minimum circumscribed rectangle is the smallest among the areas of all circumscribed rectangles of the 2D ellipse. Therefore, using the minimum circumscribed rectangle as the colored area of the 2D ellipse projection of the 3D ellipsoid on the imaging plane can minimize the area of the invalid area, thereby reducing the calculation of invalid colors in the invalid area, further saving the computing resources of the fragment shader, and improving the rendering speed.

[0040] As Figure 4 shown, Figure 4 shows a flowchart of a sub-step of the 3D Gaussian splash rendering scheme provided by the embodiment of the present application. Rendering all pixel points in the colored area in step S140 further includes the following steps.

[0041] Step S142: Traverse each pixel point in the colored area to determine whether each pixel point is located within any 2D ellipse.

[0042] Step S143: When a pixel point is located within any 2D ellipse, the pixel point located within any 2D ellipse is used as the target pixel point, and all 3D ellipsoids corresponding to all 2D ellipses where the target pixel point is located are used as the target 3D ellipsoids.

[0043] In step S110, the 2D ellipse obtained by mapping the 3D ellipsoid to the imaging plane of the rendering camera is the 2D ellipse corresponding to the 3D ellipsoid. Mapping each 3D ellipsoid to the imaging plane of the rendering camera yields the same number of 2D ellipses as the number of 3D ellipsoids, that is, multiple 2D ellipses are finally formed on the imaging plane. For each pixel point within each coloring region, the pixel point may be located within one or more 2D ellipses, or it may not be located within any 2D ellipse. In the embodiment of the present application, the pixel points located within any 2D ellipse are determined as target pixel points, and then all the target 3D ellipsoids required for rendering the target pixel points are determined.

[0044] As Figure 5 shown, Figure 5 FIG. shows a schematic projection diagram of the 3D ellipsoids of the 3D Gaussian splash rendering method provided by the embodiment of the present application.

[0045] In 3DGS, the centers of the individual 3D ellipsoids are not always arranged in sequence along the same straight line, but may be located at any position in the viewing space (3D three-dimensional space). In a specific embodiment, as Figure 5 , the projection of the 3D ellipsoid A on the imaging plane Z of the rendering camera is the 2D ellipse a, and the minimum circumscribed rectangle of the 2D ellipse a is A'. The projection of the 3D ellipsoid B on the imaging plane Z of the rendering camera is the 2D ellipse b, and the minimum circumscribed rectangle of the 2D ellipse b is B'. The projection of the 3D ellipsoid C on the imaging plane Z of the rendering camera is the 2D ellipse c, and the minimum circumscribed rectangle of the 2D ellipse c is C'. In the imaging plane Z, for the pixel point p within the coloring region defined by the minimum circumscribed rectangle C', it is located within both the 2D ellipse c and the 2D ellipse b. Therefore, the pixel point p is taken as the target pixel point, and the 3D ellipsoids B and C corresponding to the 2D ellipse b and the 2D ellipse c where the target pixel point p is located are taken as the target 3D ellipsoids.

[0046] In actual rendering, usually only the pixel points within the region of the 2D ellipse projection need to perform color calculation. Determining the target pixel points that need to perform color calculation in advance can prevent the pixel points not within the region of the 2D ellipse projection from being subjected to invalid calculations, further saving computing resources.

[0047] Step S144: Obtain the total number of target 3D ellipsoids, the color of each target 3D ellipsoid, the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point, and the distance from each target 3D ellipsoid to the rendering camera.

[0048] In some embodiments, the distance from the target 3D ellipsoid to the rendering camera is the straight-line distance from the center of the target 3D ellipsoid to the rendering camera. For each target 3D ellipsoid, after obtaining the distance d from the target 3D ellipsoid to the rendering camera, it is also necessary to obtain the transparency at the position on the target 3D ellipsoid that has a mapping relationship with the target pixel point.

[0049] In some embodiments, the transparency can generally be determined according to the distance between the target pixel point and the center of the 2D elliptical projection of the target 3D ellipsoid on the imaging plane. The closer the target pixel point is to the center of the 2D ellipse, the greater the transparency (i.e., the less transparent) of the 3D ellipsoid corresponding to the 2D ellipse at the position that has a mapping relationship with the target pixel point. The farther the target pixel point is from the center of the 2D ellipse (i.e., the closer it is to the edge of the 2D ellipse), the smaller the transparency (i.e., the more transparent) of the 3D ellipsoid corresponding to the 2D ellipse at the position that has a mapping relationship with the target pixel point.

[0050] Step S145: Determine the color contribution weight of each target 3D ellipsoid to the target pixel point according to the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point and the distance from each target 3D ellipsoid to the rendering camera.

[0051] Step S146: Determine the color of the target pixel point according to the total number of target 3D ellipsoids, the color of each target 3D ellipsoid, the color contribution weight of each target 3D ellipsoid to the target pixel point, and the preset background color at the target pixel point.

[0052] Step S146 further includes: determining the color of the target pixel point according to the following formula: .

[0053] Where C represents the color of the target pixel point, N represents the total number of target 3D ellipsoids, c i represents the color of the i-th target 3D ellipsoid among all target 3D ellipsoids, represents the transparency at the position on the i-th target 3D ellipsoid among all target 3D ellipsoids that has a mapping relationship with the target pixel point, d i represents the distance from the i-th target 3D ellipsoid among all target 3D ellipsoids to the rendering camera, w(d i , ) represents the color contribution weight of the i-th target 3D ellipsoid among all target 3D ellipsoids to the target pixel point, c 0 represents the preset background color at the target pixel point.

[0054] For semi-transparent scenes or models, all current 3DGS renderings use anisotropic 3D ellipsoids to represent the scene, and utilize their differentiable characteristics to project the 3D ellipsoids onto a two-dimensional plane for rasterization, and then calculate the color of the pixels within the shaded area through a semi-transparent blending formula (such as Formula 1). In related technologies, 3DGS rendering methods all require sorting the 3D ellipsoids relative to the rendering camera from far to near according to the distance from the 3D ellipsoids to the rendering camera. The 3D ellipsoids are usually arranged by means of line-of-sight tracing. For example, the centers of each 3D ellipsoid are arranged along the rays emitted from the rendering camera and pointing to the two-dimensional plane (such as the imaging plane of the rendering camera).

[0055] (Formula 1) Formula 1 indicates that the final color C at a certain pixel point is the weighted sum of the colors of all 3D ellipsoids that have a mapping relationship with this pixel point. The weight of each color is determined by its transparency α and the cumulative product of the transparencies of all previous (farther from the rendering) 3D ellipsoids. This blending method needs to ensure the correct occlusion relationship between the 3D ellipsoids, that is, the 3D ellipsoid closer to the rendering camera will be partially or completely occluded by the 3D ellipsoid farther from the rendering camera.

[0056] However, since Formula 1 does not satisfy the commutative law, that is, it is necessary to calculate the color / transparency contribution of each of all 3D ellipsoids that have a mapping relationship with a certain pixel point in a certain order in turn to ensure the correct result. In order to perform semi-transparent blending, it is necessary to render from back to front according to the distance between the 3D ellipsoids and the rendering camera. Therefore, it is necessary to sort the 3D ellipsoids first. Because the blending of the transparencies of multiple 3D ellipsoids needs to consider the occlusion relationship between these 3D ellipsoids, only when the 3D ellipsoids are sorted according to the distance to the rendering camera can the contribution of each 3D ellipsoid to the final color of a certain pixel point on the two-dimensional plane be correctly calculated.

[0057] However, for large scenes where the number of 3D ellipsoids is very large (the order of magnitude may be from millions to hundreds of millions), sorting all the 3D ellipsoids will bring a large performance overhead and affect the rendering speed. To solve this problem, in the 3D Gaussian splash rendering method of this application, the final color of the target pixel point is determined according to the color of the target 3D ellipsoid and its weight of the color contribution to the target pixel point.

[0058] The weight w(d i , ) of each target 3D ellipsoid to the color contribution of the target pixel point can be understood as a function of the distance d i and transparency The weight function. In some embodiments, the distance d from the i-th target 3D ellipsoid in the target 3D ellipsoid to the rendering camera i The smaller it is, the closer the target 3D ellipsoid is to the rendering camera. At the position on the i-th target 3D ellipsoid that has a mapping relationship with the target pixel point, the transparency Under the same condition, the weight w(d i , ) should be larger. The transparency at the position on the i-th target 3D ellipsoid that has a mapping relationship with the target pixel point The smaller it is, the more transparent the target 3D ellipsoid is. Under the condition that the distance d from the i-th target 3D ellipsoid to the rendering camera i is the same, the weight w(d i , ) should be smaller.

[0059] In some embodiments, w(d i , ) can be determined in various ways, such as determining the weight w(d i , ) by direct weight accumulation, exponential weight accumulation, or linear correction weight accumulation. In some other embodiments, w(d i , ) can also be determined by other related technologies, and this application does not limit this.

[0060] Then, the color of the target pixel point is determined according to the following formula 2.

[0061] (Formula 2) represents the background color c 0 The color contribution of the background color c to the final target pixel color is related to the transparency of all target 3D ellipsoids. The background color c 0 can be set in advance. The value range of is [0, 1], where 0 represents completely transparent and 1 represents completely opaque. If the transparency at the positions on all target 3D ellipsoids that have a mapping relationship with the target pixel point is all 0, then the color contribution of the background color to the final color of the target pixel point is the largest. If the transparency at the position on one target 3D ellipsoid that has a mapping relationship with the target pixel point is 1, then the color contribution of the background color to the final color of the target pixel point is 0.

[0062] For example, referring to Figure 5 , Figure 5The target pixel point o in it is inside the 2D ellipse b, and the target pixel point o is at the boundary of the 2D ellipse b. The transparency at the position on the target 3D ellipsoid B that has a mapping relationship with the target pixel point o is 0, that is, N = 1 in Formula 2, = 0, then according to Formula 2, it can be calculated that the calculation result of the color C at the target pixel point o at this time is the background color c 0 .

[0063] Formula 2 is in the form of the sum of polynomials and satisfies the commutative law, that is, the color calculation process of the target pixel point satisfies the commutative law. Therefore, the result of the color C of the target pixel point calculated according to the foregoing method and Formula 2 is independent of the arrangement order of the target 3D ellipsoids, thereby eliminating the requirement for sorting the target 3D ellipsoids during 3DGS rendering, saving the calculation amount and data processing amount, and improving the rendering speed.

[0064] As Figure 6 shown, Figure 6 shows a sub-step flowchart of the 3D Gaussian splash rendering method provided by the embodiment of the present application. Rendering all pixel points in the coloring area in step S140 further includes the following steps.

[0065] Step S147: When the pixel point is not inside any 2D ellipse, the pixel point that is not inside any 2D ellipse is used as a non-target pixel point.

[0066] Step S148: When it is determined that there is a preset background color at the non-target pixel point, the preset background color at the non-target pixel point is used as the color of the non-target pixel point.

[0067] As described above, for each pixel point in each coloring area, this pixel point may also not be inside any 2D ellipse. For such a pixel point, the embodiment of the present application determines it as a non-target pixel point, and then uses the preset background color at this pixel point for rendering when rendering this pixel point.

[0068] In a specific implementation, referring to Figure 5 , the pixel point q is not inside any 2D ellipse. The area where the pixel point q is located is usually an invalid area where no 3D ellipsoid has a color contribution to this pixel point. Therefore, the pixel point q is used as a non-target pixel point. For the non-target pixel point q, it is also necessary to determine whether there is a preset background color at this non-target pixel point q. If there is a preset background color at the non-target pixel point q, then directly use the preset background color at this non-target pixel point q as the color of the non-target pixel point q, thereby reducing the color calculation in the invalid area, greatly saving the data processing amount, saving computing resources, and improving the rendering speed.

[0069] After step S147, the method further includes: when it is determined that there is no preset background color at a non-target pixel point, determining the rendering camera background color at the non-target pixel point, and using the rendering camera background color at the non-target pixel point as the color of the non-target pixel point.

[0070] In the foregoing specific embodiment, for Figure 5 the non-target pixel point q, if there is no preset background color at the non-target pixel point q, directly using the rendering camera background color at the non-target pixel point q as the color of the non-target pixel point q, where the rendering camera background color is pre-set, thereby reducing color calculation in the invalid area, greatly saving the amount of data processing, saving computing resources, and improving the rendering speed.

[0071] As Figure 7 shown, the embodiment of the present application further provides a 3D Gaussian splash device, and the device 200 includes: an ellipsoid determination module 210, an ellipsoid projection module 220, a rectangle determination module 230, and a pixel rendering module 240.

[0072] The ellipsoid determination module 210 is configured to determine a set of 3D ellipsoids corresponding to the 3D model in the 3D Gaussian splash, and the set of 3D ellipsoids includes a plurality of 3D ellipsoids.

[0073] The ellipsoid projection module 220 is configured to, for each 3D ellipsoid in the set of 3D ellipsoids, map the 3D ellipsoid to the imaging plane of the rendering camera to obtain a 2D ellipse.

[0074] The rectangle determination module 230 is configured to determine the circumscribed rectangle of the 2D ellipse.

[0075] The pixel rendering module 240 is configured to use the circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and render all pixel points in the coloring area.

[0076] In an alternative manner, the rectangle determination module 230 and the pixel rendering module 240 are further configured to use the major axis of the 2D ellipse as the long side of the rectangle and the minor axis of the 2D ellipse as the width of the rectangle to construct the minimum circumscribed rectangle of the 2D ellipse, where the long side of the minimum circumscribed rectangle is parallel to the major axis of the 2D ellipse; using the minimum circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid, and determining the colors of all pixel points in the coloring area.

[0077] In an alternative manner, the pixel rendering module 240 is further configured to traverse each pixel point within the shaded area to determine whether each pixel point is located within any 2D ellipse; when a pixel point is located within any 2D ellipse, the pixel point located within any 2D ellipse is used as a target pixel point, and all 3D ellipsoids corresponding to all the 2D ellipses in which the target pixel point is located are used as target 3D ellipsoids; obtain the total number of target 3D ellipsoids, the color of each target 3D ellipsoid, the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point, and the distance from each target 3D ellipsoid to the rendering camera; determine the color contribution weight of each target 3D ellipsoid to the target pixel point according to the transparency at the position on each target 3D ellipsoid that has a mapping relationship with the target pixel point and the distance from each target 3D ellipsoid to the rendering camera; and determine the color of the target pixel point according to the total number of target 3D ellipsoids, the color of each target 3D ellipsoid, the color contribution weight of each target 3D ellipsoid to the target pixel point, and the preset background color at the target pixel point.

[0078] In an alternative manner, the pixel rendering module 240 is further configured to determine the color of the target pixel point according to the following formula: ; where C represents the color of the target pixel point, N represents the total number of target 3D ellipsoids, c i represents the color of the i-th target 3D ellipsoid among all target 3D ellipsoids, represents the transparency at the position on the i-th target 3D ellipsoid among all target 3D ellipsoids that has a mapping relationship with the target pixel point, d i represents the distance from the i-th target 3D ellipsoid among all target 3D ellipsoids to the rendering camera, w(d i , ) represents the color contribution weight of the i-th target 3D ellipsoid among all target 3D ellipsoids to the target pixel point, and c 0 represents the preset background color at the target pixel point.

[0079] In an alternative manner, when a pixel point is not located within any 2D ellipse, the pixel rendering module 240 is further configured to use the pixel point not located within any 2D ellipse as a non-target pixel point; when it is determined that there is a preset background color at the non-target pixel point, the preset background color at the non-target pixel point is used as the color of the non-target pixel point.

[0080] In an alternative manner, when it is determined that there is no preset background color at the non-target pixel point, the pixel rendering module 240 is further configured to determine the background color of the rendering camera at the non-target pixel point and use the background color of the rendering camera at the non-target pixel point as the color of the non-target pixel point.

[0081] In an alternative manner, the pixel rendering module 240 is further configured to determine, by using a preset method, the color contribution weight of each target 3D ellipsoid at the target pixel point according to the transparency of each target 3D ellipsoid at the target pixel point and the distance from each target 3D ellipsoid at the target pixel point to the rendering camera, where the preset method is a direct weight accumulation method, an exponential weight accumulation method, or a linear correction weight accumulation method.

[0082] In the embodiment of the present application, the rectangular determination module 230 uses the circumscribed rectangle of the 2D ellipse in which each 3D ellipsoid is mapped in the imaging plane as the coloring area, so that the range of the coloring area of the present application is smaller than the 3-sigma range, thereby reducing invalid color calculations, saving fragment shader computing resources, and improving the rendering speed.

[0083] Figure 8 FIG. shows a schematic structural diagram of an electronic device provided by an embodiment of the present application. The specific implementation of the electronic device is not limited in the specific embodiments of the present application.

[0084] As Figure 8 shown, the electronic device may include: a processor 301 and a memory 302.

[0085] Among them, the memory 302 is used to store a computer program 303. The memory 302 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory. The computer program 303 may include computer executable instructions.

[0086] The processor 301 is configured to execute the computer program 303 to implement the above-mentioned 3D Gaussian splash rendering method embodiment.

[0087] The processor 301 may be a central processing unit CPU, or a specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. One or more processors included in the electronic device may be of the same type of processor, such as one or more CPUs; or may be of different types of processors, such as one or more CPUs and one or more ASICs.

[0088] The embodiment of the present application provides a computer-readable storage medium, and the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned 3D Gaussian splash rendering method embodiment is implemented.

[0089] In several embodiments provided in this application, if any function is implemented in the form of a software functional module / unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, all or part of the technical solution of this application can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be an electronic device such as a personal computer or a server) to execute all or part of the steps of the methods in various embodiments of this application. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs that can store computer program code.

[0090] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings provided herein. Based on the above description, the structure required to construct such systems will be apparent. In addition, the embodiments of this application are not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the descriptions made above regarding specific languages are for the purpose of disclosing the best mode of this application.

[0091] It should be noted that the above embodiments illustrate this application rather than limit this application, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In the claims listing several devices, several of the units or modules in these devices can be embodied by the same hardware item. The use of the words first, second, and third, etc. does not denote any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

[0092] The above embodiments only represent several implementation manners of this application, and their descriptions are relatively specific and detailed, but should not be construed as limiting the patent scope of this application. It should be pointed out that for those of ordinary skill in the art, without departing from the concept of this application, several modifications and improvements can still be made, and these all belong to the protection scope of this application. Therefore, the protection scope of this application shall be subject to the appended claims.

Claims

1. A 3D Gaussian splash rendering method, characterized in that: The method comprises: Determine a 3D ellipsoid set corresponding to the 3D model in the 3D Gaussian splash, wherein the 3D ellipsoid set includes a plurality of 3D ellipsoids; For each of the 3D ellipsoids in the set of 3D ellipsoids, mapping the 3D ellipsoid to an imaging plane of a rendering camera to obtain a 2D ellipse; Determine the circumscribed rectangle of the 2D ellipse; The circumscribed rectangle of the 2D ellipse is used as the colored area corresponding to the 3D ellipsoid, and all the pixels in the colored area are rendered.

2. The method according to claim 1, characterized in that The step of determining the circumscribed rectangle of the 2D ellipse; using the circumscribed rectangle of the 2D ellipse as the colored area corresponding to the 3D ellipsoid, and rendering all the pixels within the colored area further comprises: Constructing a minimum circumscribed rectangle of the 2D ellipse using the major axis of the 2D ellipse as the long side of the rectangle and the minor axis of the 2D ellipse as the wide side of the rectangle, wherein the long side of the minimum circumscribed rectangle is parallel to the major axis of the 2D ellipse; The minimum circumscribed rectangle of the 2D ellipse is used as a colored area corresponding to the 3D ellipsoid, and the colors of all the pixels in the colored area are determined.

3. The method according to any one of claims 1 to 2, characterized in that The rendering of all pixels within the shading area further comprises: Traversing each of the pixel points in the shading area, and determining whether each of the pixel points is located in any of the 2D ellipses; When the pixel point is located in any one of the 2D ellipses, the pixel point located in any one of the 2D ellipses is used as a target pixel point, and all the 3D ellipsoids corresponding to all the 2D ellipses where the target pixel point is located are used as target 3D ellipsoids; Acquire the total number of the target 3D ellipsoids, the color of each of the target 3D ellipsoids, the transparency of each position on the target 3D ellipsoid that has a mapping relationship with the target pixel point, and the distance from each of the target 3D ellipsoids to the rendering camera; Determine a color contribution weight of each target 3D ellipsoid to the target pixel point according to the transparency of a position on each target 3D ellipsoid that has a mapping relationship with the target pixel point and the distance from each target 3D ellipsoid to a rendering camera; The color of the target pixel is determined according to the total number of the target 3D ellipsoids, the color of each of the target 3D ellipsoids, the color contribution weight of each of the target 3D ellipsoids to the target pixel, and the preset background color at the target pixel.

4. The method according to claim 3, characterized in that The determining the color of the target pixel point according to the total number of the target 3D ellipsoids, the color of each of the target 3D ellipsoids, the color contribution weight of each of the target 3D ellipsoids to the target pixel point, and the preset background color at the target pixel point further comprises: The color of the target pixel is determined according to the following formula: ； Wherein, C represents the color of the target pixel, N represents the total number of the target 3D ellipsoids, and c i represents the color of the i-th target 3D ellipsoid among all the target 3D ellipsoids, represents the transparency of the position on the i-th target 3D ellipsoid among all the target 3D ellipsoids that has a mapping relationship with the target pixel point, d i represents the distance from the i-th target 3D ellipsoid among all the target 3D ellipsoids to the rendering camera, w(d i , ) represents the color contribution weight of the i-th target 3D ellipsoid among all the target 3D ellipsoids to the target pixel point, and c0 represents the preset background color at the target pixel point.

5. The method according to claim 3, characterized in that: The rendering of all pixels within the shading area further comprises: When the pixel point is not located in any of the 2D ellipses, the pixel point not located in any of the 2D ellipses is regarded as a non-target pixel point; When it is determined that there is a preset background color at the non-target pixel point, the preset background color at the non-target pixel point is used as the color of the non-target pixel point.

6. The method according to claim 5, characterized in that After the pixel point not being located in any of the 2D ellipses is regarded as a non-target pixel point when the pixel point is not located in any of the 2D ellipses, the method further includes: When it is determined that the preset background color does not exist at the non-target pixel point, a rendering camera background color at the non-target pixel point is determined, and the rendering camera background color at the non-target pixel point is used as the color of the non-target pixel point.

7. The method according to claim 3, characterized in that The step of determining the color contribution weight of each of the target 3D ellipsoids at the target pixel point according to the transparency of each of the target 3D ellipsoids at the target pixel point and the distance of each of the target 3D ellipsoids at the target pixel point to the rendering camera further comprises: According to the transparency of each target 3D ellipsoid at the target pixel point and the distance of each target 3D ellipsoid at the target pixel point to the rendering camera, a preset method is used to determine the color contribution weight of each target 3D ellipsoid at the target pixel point, wherein the preset method is a direct weight accumulation method, an exponential weight accumulation method or a linear correction weight accumulation method.

8. A 3D Gaussian splash rendering device, characterized in that: The device comprises: An ellipsoid determination module, used to determine a 3D ellipsoid set corresponding to the 3D model in the 3D Gaussian splash, wherein the 3D ellipsoid set includes a plurality of 3D ellipsoids; An ellipsoid projection module, for mapping each of the 3D ellipsoids in the 3D ellipsoid set to an imaging plane of a rendering camera to obtain a 2D ellipse; A rectangle determination module, used to determine the circumscribed rectangle of the 2D ellipse; The pixel rendering module is used to use the circumscribed rectangle of the 2D ellipse as the coloring area corresponding to the 3D ellipsoid and render all pixel points in the coloring area.

9. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the 3D Gaussian splash rendering method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the 3D Gaussian splash rendering method according to any one of claims 1 to 7 is implemented.

Images