A three-dimensional reconstruction and rendering method, device, equipment and storage medium

Abstract

The application discloses a three-dimensional reconstruction and rendering method, device and equipment and a storage medium, relates to the technical field of computers, and comprises the following steps: generating initial three-dimensional primitives corresponding to the data structure of a sparse point cloud of a three-dimensional target to be constructed; performing rasterization rendering on the initial three-dimensional primitives to obtain a predicted image, performing multi-dimensional loss calculation on the predicted image based on a real image corresponding to the three-dimensional target to be constructed; adjusting target parameters of the initial three-dimensional primitives based on a total loss value, performing filling optimization on a three-dimensional model surface formed by the adjusted three-dimensional primitives; determining the obtained optimized three-dimensional primitives as new initial three-dimensional primitives, jumping to the step of performing rasterization rendering on the initial three-dimensional primitives, and stopping until a target optimization stop condition is met, and performing double-mode parallel rendering on the obtained target three-dimensional primitives to obtain a rendered three-dimensional target. The three-dimensional content is reconstructed and rendered at low cost.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a three-dimensional reconstruction and rendering method, apparatus, device, and storage medium. Background Technology

[0002] With the development of metaverse, digital twins and next-generation games, the business side has put forward extremely demanding "double standards" for the reconstruction and rendering of 3D scenes: it requires a real-time rendering frame rate of more than 60 FPS on mobile devices (which usually requires rasterization technology), and it also requires cinematic physical lighting effects (such as specular reflection and global illumination, which must rely on ray tracing technology).

[0003] In existing workflows, to meet this requirement, art teams typically need to maintain two sets of assets: a Gaussian point cloud for real-time display and a low-poly mesh (Proxy Mesh) for physics calculations. This not only increases production costs but also frequently leads to visual bugs such as "clipping" and "light leaks" caused by the misalignment of the two sets of assets. 3DGS (3D Gaussian Splatting) consists of blurry clouds, making it impossible for light rays and colliders to accurately determine boundaries, resulting in inaccurate shadow calculations and physical collisions. Converting Gaussians to meshes (such as MeshSplats) or directly attaching triangles often results in a "fish-scale" or "jagged" geometric surface. This is because a Gaussian ellipsoid is essentially a soft-edged volume, while a triangle is a hard-edged plane. Without strong constraints, triangles will intersect randomly, leading to a rough physical surface, incorrect reflection angles, and collision jitter. 2DGS (2D Gaussian Splatting) attempts to fuse ray tracing using "disks" as the basic primitives. However, the ray tracing hardware of existing GPUs (Graphics Processing Units) is specifically optimized for triangular primitives. The ray intersection calculation efficiency of disk primitives is extremely low, which cannot fully utilize the hardware computing power and makes it difficult to achieve real-time cinematic ray tracing effects.

[0004] As can be seen from the above, how to achieve low-cost reconstruction and rendering of 3D content is an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device, and storage medium for three-dimensional reconstruction and rendering, which can achieve the reconstruction and rendering of three-dimensional content at low cost. The specific solution is as follows: Firstly, this application provides a three-dimensional reconstruction and rendering method, including: Define a data structure for 3D primitives, obtain a sparse point cloud of the 3D target to be constructed, and generate initial 3D primitives corresponding to the data structure based on the sparse point cloud; the 3D primitives are 3D primitives combining Gaussian ellipsoids and micro polygons; the data structure includes a target representation layer and a target geometry layer; The initial 3D primitives are rasterized and rendered to obtain the corresponding predicted image. Based on the real image corresponding to the 3D target to be constructed, the predicted image is subjected to multidimensional loss calculation to obtain the corresponding total loss value. The multidimensional loss includes photometric loss, geometric consistency loss, normal orientation loss and neighborhood smoothing loss. Based on the total loss value and using the graphics processor to determine the parameter gradient, the target parameters of the initial 3D primitives are adjusted using the parameter gradient to obtain the corresponding adjusted 3D primitives. The surface of the 3D model composed of the adjusted 3D primitives is inspected, and a filling optimization operation is performed based on the inspection results to obtain the optimized 3D primitives. The optimized 3D primitives are determined as new initial 3D primitives, and the process jumps to the step of rasterizing the initial 3D primitives until the target optimization stop condition is met to obtain the target 3D primitives. The target 3D primitives are then rendered in parallel in dual modes to obtain the rendered 3D target.

[0006] Optionally, the process of defining a data structure for 3D primitives, obtaining a sparse point cloud of the 3D target to be constructed, and generating initial 3D primitives corresponding to the data structure based on the sparse point cloud includes: Define a data structure for 3D primitives; the data structure includes a target representation layer and a target geometry layer; the target representation layer includes Gaussian parameters; the Gaussian parameters include the center coordinates, rotation angle, size, opacity, and spherical harmonic coefficients of a Gaussian ellipsoid; the target geometry layer includes micropolygon parameters; the micropolygon parameters include vertex data and normals of the micropolygon; the center coordinates of the Gaussian ellipsoid coincide with the center point of the micropolygon; the normals of the micropolygon and the shortest axis direction of the Gaussian ellipsoid meet a preset approximate parallel condition; Obtain the sparse point cloud of the 3D target to be constructed, and generate initial 3D primitives with the same data structure based on the sparse point cloud.

[0007] Optionally, the step of rasterizing the initial 3D primitives to obtain the corresponding predicted image includes: The initial 3D primitives are rasterized and rendered based on the Gaussian parameters of the target representation layer in the initial 3D primitives to obtain the corresponding predicted image.

[0008] Optionally, the step of performing multidimensional loss calculation on the predicted image based on the real image corresponding to the three-dimensional target to be constructed, to obtain the corresponding total loss value, includes: The pixel difference between the predicted image and the real image corresponding to the three-dimensional target to be constructed is determined using the L1 loss function and the D-SSIM algorithm, and a first loss value is determined based on the obtained pixel difference value. The shape parameter difference between the shape parameters of the Gaussian ellipsoid and the micropolygon corresponding to the predicted image is determined based on the covariance matrix, and the second loss value is determined using the shape parameter difference. The direction of the shortest axis of the Gaussian ellipsoid corresponding to the predicted image is determined, and the target normal of the micropolygon corresponding to the predicted image is determined based on the normal map. A third loss value is determined based on the target normal and the direction. The surrounding neighboring primitives of each initial 3D primitive corresponding to the predicted image are determined, and a fourth loss value is determined based on the normal difference between each initial 3D primitive and its corresponding surrounding neighboring primitive. The total loss value is determined based on each loss value and its corresponding weight.

[0009] Optionally, the step of determining the parameter gradient based on the total loss value using a graphics processor, adjusting the target parameters of the initial 3D primitives using the parameter gradient to obtain the corresponding adjusted 3D primitives, inspecting the surface of the 3D model composed of the adjusted 3D primitives, and performing a filling optimization operation based on the inspection results to obtain optimized 3D primitives includes: Based on the total loss value, the parameter gradient is determined using the automatic differentiation algorithm of the graphics processor, and the Gaussian parameters and micro-polygon parameters of the initial 3D primitive are adjusted using the parameter gradient to obtain the corresponding adjusted 3D primitive. The surface of the three-dimensional model composed of the adjusted three-dimensional primitives is inspected. If the inspection results indicate that there are gaps or light leakage target areas on the surface of the three-dimensional model, then the adjusted three-dimensional primitives are split or cloned to obtain the filled three-dimensional primitives. The target region is filled and optimized based on the filled 3D primitives to obtain optimized 3D primitives.

[0010] Optionally, the step of performing dual-mode parallel rendering on the target 3D primitives to obtain the rendered 3D target includes: The target 3D primitives are rendered using Gaussian sputtering based on the Gaussian parameters of the target representation layer to obtain a first rendering result. The target 3D primitives are traced and rendered using the micro-polygon parameters of the target geometry layer and ray tracing technology to obtain a second rendering result; The rendered 3D target is determined based on the first rendering result and the second rendering result.

[0011] Optionally, the step of using the micro-polygon parameters of the target geometry layer of the target 3D primitive and ray tracing technology to perform tracking and rendering of the target 3D primitive to obtain a second rendering result includes: The micro-polygons of the target geometry layer of the target 3D primitive are mapped to the bottom-level acceleration result, and the micro-polygons of the target geometry layer of the target 3D primitive are determined using the bottom-level acceleration result; The intersection point of the ray and the micropolygon is determined by the ray intersection shader, and the opacity of the Gaussian ellipsoid in the target 3D primitive corresponding to the intersection point is determined. If the opacity is less than a preset threshold, the intersection point is ignored; If the opacity is not less than the preset threshold, the intersection point is determined as a valid physical collision point, and the physical lighting result corresponding to the valid physical collision point is determined. The second rendering result is then determined using the physical lighting result.

[0012] Secondly, this application provides a three-dimensional reconstruction and rendering apparatus, comprising: An initial primitive generation module is used to define the data structure of 3D primitives, obtain the sparse point cloud of the 3D target to be constructed, and generate each initial 3D primitive corresponding to the data structure based on the sparse point cloud; the 3D primitive is a 3D primitive combining Gaussian ellipsoids and micro polygons; the data structure includes a target representation layer and a target geometry layer; The loss calculation module is used to perform rasterization rendering on the initial 3D primitives to obtain the corresponding prediction image, and to perform multidimensional loss calculation on the prediction image based on the real image corresponding to the 3D target to be constructed to obtain the corresponding total loss value; the multidimensional loss includes photometric loss, geometric consistency loss, normal orientation loss and neighborhood smoothing loss; The model surface inspection module is used to determine the parameter gradient based on the total loss value and using the graphics processor, and to adjust the target parameters of the initial three-dimensional primitives using the parameter gradient to obtain the corresponding adjusted three-dimensional primitives. The module then inspects the surface of the three-dimensional model composed of the adjusted three-dimensional primitives and performs a filling optimization operation based on the inspection results to obtain the optimized three-dimensional primitives. The target primitive rendering module is used to determine the optimized 3D primitive as the new initial 3D primitive, and jump to the step of rasterizing the initial 3D primitive until the target optimization stop condition is met to obtain the target 3D primitive. The target 3D primitive is then rendered in parallel in dual modes to obtain the rendered 3D target.

[0013] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned three-dimensional reconstruction and rendering method.

[0014] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned three-dimensional reconstruction and rendering method.

[0015] This application defines a data structure for 3D primitives, obtains a sparse point cloud of the 3D target to be constructed, and generates initial 3D primitives corresponding to the data structure based on the sparse point cloud; the 3D primitives are 3D primitives combining Gaussian ellipsoids and micropolygons; the data structure includes a target representation layer and a target geometry layer; rasterization rendering is performed on the initial 3D primitives to obtain a corresponding predicted image; multidimensional loss calculation is performed on the predicted image based on the real image corresponding to the 3D target to be constructed to obtain a corresponding total loss value; the multidimensional loss includes photometric loss, geometric consistency loss, normal orientation loss, and neighborhood smoothing loss; based on the total... The loss value is calculated, and the parameter gradient is determined using the graphics processor. The target parameters of the initial 3D primitives are adjusted using the parameter gradient to obtain the corresponding adjusted 3D primitives. The surface of the 3D model composed of the adjusted 3D primitives is inspected, and a filling optimization operation is performed based on the inspection results to obtain the optimized 3D primitives. The optimized 3D primitives are determined as the new initial 3D primitives, and the process jumps to the step of rasterizing the initial 3D primitives until the target optimization stopping condition is met to obtain the target 3D primitives. The target 3D primitives are then rendered in parallel in dual modes to obtain the rendered 3D target.

[0016] As can be seen from the above, this application defines a two-level data structure that integrates Gaussian ellipsoids and micropolygons, enabling 3D primitives to possess both visual representation and physical interaction attributes from the outset. Rasterization rendering transforms the initial primitives into a visualized predicted image. Then, multi-dimensional loss calculations are performed, integrating photometric, geometric consistency, normal orientation, and neighborhood smoothing to quantify the error between the primitives and the real target. The gradient is determined based on the total loss value, and primitive parameters are adjusted. Surface inspection and filling optimization accurately identify and repair defects such as gaps and light leaks, allowing discrete micropolygons to form a continuous physical surface. Through multiple iterations, the total loss value is continuously reduced, resulting in higher accuracy of the obtained target 3D primitives. This dual-mode parallel rendering of the target 3D primitives ensures that the rendered 3D target is both real-time and highly accurate, giving the final 3D target both photorealistic visual fidelity and high-precision physical geometric properties, thus improving the efficiency of constructing complex 3D targets. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 This is a flowchart of a three-dimensional reconstruction and rendering method disclosed in this application; Figure 2 This is a schematic diagram of the structure of a three-dimensional reconstruction and rendering device disclosed in this application; Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Currently, in existing business processes, art teams typically need to maintain two sets of assets: a Gaussian point cloud for real-time display and a low-poly mesh for physical calculations. This not only increases production costs but also frequently leads to visual bugs such as "clipping" and "light leakage" caused by the incomplete alignment of the two sets of assets. To address this, this application provides a 3D reconstruction and rendering method that performs dual-mode parallel rendering of the target 3D primitives. This results in a 3D target with both real-time performance and high precision, allowing the final 3D target to possess both photorealistic visual fidelity and high-precision physical geometric properties, thereby improving the efficiency of constructing complex 3D targets.

[0021] See Figure 1 As shown, this embodiment of the invention discloses a three-dimensional reconstruction and rendering method, including: Step S11: Define the data structure of the three-dimensional primitives, obtain the sparse point cloud of the three-dimensional target to be constructed, and generate each initial three-dimensional primitive corresponding to the data structure based on the sparse point cloud; the three-dimensional primitives are three-dimensional primitives combining Gaussian ellipsoids and micro polygons; the data structure includes a target representation layer and a target geometry layer.

[0022] In this embodiment, a Surfel-Gaussian primitive data structure is defined, including a target geometry layer for describing the physical framework and a target presentation layer for describing the soft visual appearance. The target presentation layer includes Gaussian parameters; the Gaussian parameters include the center coordinates of the Gaussian ellipsoid, the rotation angle of the Gaussian ellipsoid, the length, width, height, opacity, and spherical harmonic coefficient of the Gaussian ellipsoid; the target geometry layer includes micro-polygon parameters; the micro-polygons include micro-triangles / micro-quadrilaterals; the micro-polygon parameters include the vertex data and normals of the micro-polygons.

[0023] Specifically, the process of defining a data structure for 3D primitives, obtaining a sparse point cloud of the 3D target to be constructed, and generating initial 3D primitives corresponding to the data structure based on the sparse point cloud includes: defining a data structure for 3D primitives; the data structure includes a target representation layer and a target geometry layer; the target representation layer includes Gaussian parameters; the Gaussian parameters include the center coordinates, rotation angle, size, opacity, and spherical harmonic coefficients of a Gaussian ellipsoid; the target geometry layer includes micro-polygon parameters; the micro-polygon parameters include vertex data and normals of the micro-polygon; the center coordinates of the Gaussian ellipsoid coincide with the center point of the micro-polygon; the normals of the micro-polygon and the shortest axis direction of the Gaussian ellipsoid meet a preset approximate parallel condition; obtaining a sparse point cloud of the 3D target to be constructed, and generating initial 3D primitives consistent with the data structure based on the sparse point cloud.

[0024] Step S12: Rasterize the initial 3D primitives to obtain the corresponding predicted image. Perform multidimensional loss calculation on the predicted image based on the real image corresponding to the 3D target to be constructed to obtain the corresponding total loss value. The multidimensional loss includes photometric loss, geometric consistency loss, normal orientation loss and neighborhood smoothing loss.

[0025] In this embodiment, the target geometry layer of the initial 3D primitive can be a tiny quadrilateral facing the camera, which can be divided into two triangles. The Gaussian parameters of the target representation layer in the initial 3D primitive are used to render the predicted image using rasterization. Specifically, the rasterization rendering of the initial 3D primitive to obtain the corresponding predicted image includes: performing rasterization rendering on the initial 3D primitive based on the Gaussian parameters of the target representation layer in the initial 3D primitive to obtain the corresponding predicted image.

[0026] It is understandable that the formula for the total loss function, which performs multidimensional loss calculation on the predicted image, is as follows: ; in, This is the total loss function; This refers to the photometric loss function within the total loss function; This refers to the geometric consistency loss function within the total loss function; This refers to the normal orientation loss function within the total loss function; This refers to the neighborhood smoothing loss function within the total loss function; , , These are the weights corresponding to the geometric consistency loss function, the normal orientation loss function, and the neighborhood smoothing loss function, respectively.

[0027] Furthermore, the photometric loss function uses the L1 loss function and the D-SSIM (Structural Similarity Index) algorithm to determine the pixel differences between the predicted image and the real image corresponding to the 3D target to be constructed; the geometric consistency loss function determines the shape parameter difference between the shape parameters of the Gaussian ellipsoid and the micropolygon corresponding to the predicted image based on the covariance matrix, and the formula for the geometric consistency loss function is as follows: ; in, Let the geometric consistency loss function be denoted as . Let be the covariance matrix corresponding to the Gaussian ellipsoid; This is the covariance matrix corresponding to the micropolygon; The value is the F-norm. Specifically, if the micro-polygon is a micro-triangle, the formula for the corresponding covariance matrix is ​​as follows: ; in, This is the covariance matrix corresponding to the micro-triangle; The spatial coordinates of the three vertices of the micro-triangle; Let be the geometric center of the micro-triangle, and the calculation formula is: ; , , Let be the spatial coordinates of the three vertices of the micro-triangle; T is the transpose. Similarly, if the micro-polygon is a micro-quadrilateral, the formula for the corresponding covariance matrix is ​​as follows: ; in, This is the covariance matrix corresponding to the micro quadrilateral; Let be the spatial coordinates of the four vertices of the microquadrilateral; Let be the geometric center of the micro quadrilateral, and the calculation formula is: ; , , , , where are the spatial coordinates of the four vertices of the micro quadrilateral; T is the transpose.

[0028] In this embodiment, the normal orientation loss function is determined based on the target normal of the micro-polygon corresponding to the predicted image and the direction of the shortest axis of the Gaussian ellipsoid corresponding to the predicted image, and the corresponding formula is as follows: ; in, The normal orientation loss function; The target normal of the micropolygon corresponding to the predicted image; The direction of the shortest axis of the Gaussian ellipsoid corresponding to the predicted image is given. The formula for the neighborhood smoothing loss function is as follows: ; in, This refers to the neighborhood smoothing loss function within the total loss function; For each initial 3D primitive i, search for the K nearest primitives in 3D space. These primitives constitute the neighborhood of the current primitive. The normal to the current initial 3D primitive; The normals to adjacent initial 3D primitives are defined. The total loss value is determined based on the loss values ​​corresponding to each of the above loss functions.

[0029] Specifically, the step of performing multidimensional loss calculation on the predicted image based on the real image corresponding to the 3D target to be constructed, to obtain the corresponding total loss value, includes: using the L1 loss function and the D-SSIM algorithm to determine the pixel difference between the predicted image and the real image corresponding to the 3D target to be constructed, and determining a first loss value based on the obtained pixel difference value; determining the shape parameter difference between the shape parameters of the Gaussian ellipsoid and the micropolygon corresponding to the predicted image based on the covariance matrix, and determining a second loss value based on the shape parameter difference; determining the direction of the shortest axis of the Gaussian ellipsoid corresponding to the predicted image, and determining the target normal of the micropolygon corresponding to the predicted image based on the normal map, and determining a third loss value based on the target normal and the direction; determining the surrounding neighboring primitives of each initial 3D primitive corresponding to the predicted image, and determining a fourth loss value based on the normal difference between each initial 3D primitive and its corresponding surrounding neighboring primitive; and determining the total loss value based on each loss value and its corresponding weight.

[0030] Step S13: Based on the total loss value and using the graphics processor to determine the parameter gradient, and using the parameter gradient to adjust the target parameters of the initial three-dimensional primitives to obtain the corresponding adjusted three-dimensional primitives, check the surface of the three-dimensional model composed of the adjusted three-dimensional primitives, and perform a filling optimization operation based on the obtained inspection results to obtain the optimized three-dimensional primitives.

[0031] In this embodiment, after obtaining the total loss value, the parameter gradient is determined based on the total loss value and using the Automatic Differentiation (AD) algorithm of the graphics processor. The Gaussian parameters and vertex positions of the micro-polygons of the initial 3D primitives are adjusted based on the parameter gradient to obtain the adjusted 3D primitives. If the parameter gradient exceeds the target gradient threshold and the transmittance of the surface of the 3D model composed of the adjusted 3D primitives is not 0, it indicates that there is light leakage through gaps. The adjusted 3D primitives are then split or cloned to fill and optimize the light leakage area to obtain the optimized 3D primitives.

[0032] Specifically, the process of determining parameter gradients based on the total loss value using a graphics processor, adjusting the target parameters of the initial 3D primitives using the parameter gradients to obtain corresponding adjusted 3D primitives, inspecting the surface of the 3D model composed of the adjusted 3D primitives, and performing a filling optimization operation based on the inspection results to obtain optimized 3D primitives includes: determining parameter gradients based on the total loss value using an automatic differentiation algorithm of a graphics processor, adjusting the Gaussian parameters and micro-polygon parameters of the initial 3D primitives using the parameter gradients to obtain corresponding adjusted 3D primitives; inspecting the surface of the 3D model composed of the adjusted 3D primitives, and if the inspection results indicate that there are gaps or light leaks in the target area on the 3D model surface, then performing a splitting or cloning operation on the adjusted 3D primitives to obtain filled 3D primitives; and performing a filling optimization operation on the target area based on the filled 3D primitives to obtain optimized 3D primitives.

[0033] Step S14: Determine the optimized 3D primitive as the new initial 3D primitive, and jump to the step of rasterizing the initial 3D primitive until the target optimization stop condition is met to obtain the target 3D primitive. Perform dual-mode parallel rendering on the target 3D primitive to obtain the rendered 3D target.

[0034] In this embodiment, after obtaining the optimized 3D primitives, they are designated as new initial 3D primitives, and the process jumps to the step of rasterizing and rendering the initial 3D primitives. This process is repeated until the total loss value does not exceed the target loss threshold, thus obtaining the target 3D primitive. The target loss threshold can be determined according to the actual situation. Then, Gaussian sputtering rendering is performed on the target 3D primitives using the Gaussian parameters of the target representation layer to ensure a frame rate greater than 60 FPS, resulting in a first rendering result. The micropolygon parameters of the target geometry layer of the target 3D primitives and APIs supporting real-time ray tracing (such as DXR / Vulkan RT) are used to trace and render the target 3D primitives, resulting in a second rendering result. The rendered 3D target is then determined based on each rendering result.

[0035] Specifically, the dual-mode parallel rendering of the target 3D primitives to obtain the rendered 3D target includes: performing Gaussian sputtering rendering on the target 3D primitives using the Gaussian parameters of the target representation layer to obtain a first rendering result; performing ray tracing rendering on the target 3D primitives using the micropolygon parameters of the target geometry layer and ray tracing technology to obtain a second rendering result; and determining the rendered 3D target based on the first rendering result and the second rendering result.

[0036] Understandably, when tracking and rendering the target 3D primitive, a low-level acceleration structure is constructed based on the micro-polygons of the target geometry layer of the target 3D primitive to quickly locate the micro-polygons of the target geometry layer of the target 3D primitive. A ray intersection shader is used to emit rays to collide with the micro-polygons to determine the intersection point of the rays and the micro-polygons, and to determine the opacity of the Gaussian ellipsoid of the target representation layer of the target 3D primitive corresponding to the intersection point. If the opacity is less than a preset threshold, the intersection point is ignored. The preset threshold can be set according to the actual situation. If the opacity is not less than the preset threshold, the intersection point is retained and determined as a valid physical collision point. Based on the valid collision point, cinematic physical lighting effects such as light reflection, refraction, global illumination, and shadows are calculated. Finally, the physical lighting calculation results of all valid collision points are integrated to obtain the second rendering result.

[0037] Specifically, the step of using the micro-polygon parameters of the target geometry layer of the target 3D primitive and ray tracing technology to perform tracking and rendering on the target 3D primitive to obtain a second rendering result includes: mapping the micro-polygons of the target geometry layer of the target 3D primitive to a bottom-level accelerated result; using the bottom-level accelerated result to determine the micro-polygons of the target geometry layer of the target 3D primitive; determining the intersection point of the ray and the micro-polygon based on the ray intersection shader, and determining the opacity of the Gaussian ellipsoid in the target 3D primitive corresponding to the intersection point; if the opacity is less than a preset threshold, then the intersection point is ignored; if the opacity is not less than the preset threshold, then the intersection point is determined as a valid physical collision point, and the physical lighting result corresponding to the valid physical collision point is determined, and the second rendering result is determined using the physical lighting result.

[0038] As can be seen from the above, this application defines a two-level data structure that integrates Gaussian ellipsoids and micropolygons, enabling 3D primitives to possess both visual representation and physical interaction attributes from the outset. Rasterization rendering transforms the initial primitives into a visualized predicted image. Then, multi-dimensional loss calculations are performed, integrating photometric, geometric consistency, normal orientation, and neighborhood smoothing to quantify the error between the primitives and the real target. The gradient is determined based on the total loss value, and primitive parameters are adjusted. Surface inspection and filling optimization accurately identify and repair defects such as gaps and light leaks, allowing discrete micropolygons to form a continuous physical surface. Through multiple iterations, the total loss value is continuously reduced, resulting in higher accuracy of the obtained target 3D primitives. This dual-mode parallel rendering of the target 3D primitives ensures that the rendered 3D target is both real-time and highly accurate, giving the final 3D target both photorealistic visual fidelity and high-precision physical geometric properties, thus improving the efficiency of constructing complex 3D targets.

[0039] Accordingly, see Figure 2As shown, this application also provides a three-dimensional reconstruction and rendering apparatus, including: The initial primitive generation module 11 is used to define the data structure of the three-dimensional primitives, obtain the sparse point cloud of the three-dimensional target to be constructed, and generate each initial three-dimensional primitive corresponding to the data structure based on the sparse point cloud; the three-dimensional primitives are three-dimensional primitives that combine Gaussian ellipsoids and micro polygons; the data structure includes a target representation layer and a target geometry layer; The loss calculation module 12 is used to perform rasterization rendering on the initial three-dimensional primitives to obtain the corresponding predicted image, and to perform multi-dimensional loss calculation on the predicted image based on the real image corresponding to the three-dimensional target to be constructed to obtain the corresponding total loss value; the multi-dimensional loss includes photometric loss, geometric consistency loss, normal orientation loss and neighborhood smoothing loss. The model surface inspection module 13 is used to determine the parameter gradient based on the total loss value and using the graphics processor, and to adjust the target parameters of the initial three-dimensional primitives using the parameter gradient to obtain the corresponding adjusted three-dimensional primitives, to inspect the surface of the three-dimensional model composed of the adjusted three-dimensional primitives, and to perform a filling optimization operation based on the inspection results to obtain the optimized three-dimensional primitives. The target primitive rendering module 14 is used to determine the optimized 3D primitive as the new initial 3D primitive, and jump to the step of rasterizing the initial 3D primitive until the target optimization stop condition is met to obtain the target 3D primitive. The target 3D primitive is then rendered in parallel in dual modes to obtain the rendered 3D target.

[0040] In some specific embodiments, the initial primitive generation module 11 may specifically include: A data structure definition unit is used to define the data structure of 3D primitives. The data structure includes a target representation layer and a target geometry layer. The target representation layer includes Gaussian parameters, which include the center coordinates, rotation angle, size, opacity, and spherical harmonic coefficients of a Gaussian ellipsoid. The target geometry layer includes micropolygon parameters, which include vertex data and normals of the micropolygon. The center coordinates of the Gaussian ellipsoid coincide with the center point of the micropolygon. The normals of the micropolygon and the shortest axis of the Gaussian ellipsoid meet a preset approximate parallel condition. An initial primitive generation unit is used to obtain a sparse point cloud of the three-dimensional target to be constructed, and to generate initial three-dimensional primitives with the same data structure based on the sparse point cloud.

[0041] In some specific embodiments, the loss calculation module 12 may specifically include: The initial primitive rendering unit is used to perform rasterization rendering on the initial three-dimensional primitives based on the Gaussian parameters of the target representation layer in the initial three-dimensional primitives to obtain the corresponding predicted image.

[0042] In some specific embodiments, the loss calculation module 12 may specifically include: The first loss value determination unit is used to determine the pixel difference between the predicted image and the real image corresponding to the three-dimensional target to be constructed using the L1 loss function and the D-SSIM algorithm, and to determine the first loss value based on the obtained pixel difference value. The second loss value determination unit is used to determine the shape parameter difference between the shape parameter of the Gaussian ellipsoid and the shape parameter of the micropolygon corresponding to the predicted image based on the covariance matrix, and to determine the second loss value using the shape parameter difference. The third loss value determination unit is used to determine the direction of the shortest axis of the Gaussian ellipsoid corresponding to the predicted image, and to determine the target normal of the micropolygon corresponding to the predicted image based on the normal map, and to determine the third loss value based on the target normal and the direction. The fourth loss value determination unit is used to determine the surrounding neighboring primitives of each initial three-dimensional primitive corresponding to the predicted image, and to determine the fourth loss value based on the normal difference between each initial three-dimensional primitive and its corresponding surrounding neighboring primitive. The total loss value determination unit is used to determine the total loss value based on each loss value and its corresponding weight.

[0043] In some specific embodiments, the model surface inspection module 13 may specifically include: The parameter adjustment unit is used to determine the parameter gradient based on the total loss value and using the automatic differentiation algorithm of the graphics processor, and to adjust the Gaussian parameters and micro-polygon parameters of the initial three-dimensional primitive using the parameter gradient to obtain the corresponding adjusted three-dimensional primitive. The primitive splitting unit is used to inspect the surface of the three-dimensional model composed of the adjusted three-dimensional primitives. If the inspection result indicates that there are gaps or light leakage target areas on the surface of the three-dimensional model, then the adjusted three-dimensional primitives are split or cloned to obtain the filled three-dimensional primitives. The region filling unit is used to perform filling optimization operations on the target region based on the filling 3D primitives to obtain optimized 3D primitives.

[0044] In some specific embodiments, the target primitive rendering module 14 may specifically include: The first primitive rendering unit is used to perform Gaussian sputtering rendering on the target three-dimensional primitive using the Gaussian parameters of the target representation layer of the target three-dimensional primitive to obtain a first rendering result. The second primitive rendering submodule is used to perform tracking and rendering of the target three-dimensional primitive using the micro-polygon parameters of the target geometry layer and ray tracing technology to obtain a second rendering result. The post-rendering target determination unit is used to determine the post-rendering 3D target based on the first rendering result and the second rendering result.

[0045] In some specific implementations, the second primitive rendering submodule may specifically include: The micro-polygon determination unit is used to map the micro-polygons of the target geometry layer of the target three-dimensional primitive to the underlying acceleration result, and to determine the micro-polygons of the target geometry layer of the target three-dimensional primitive using the underlying acceleration result. The opacity determination unit is used to determine the intersection point of the ray and the micro-polygon based on the ray intersection shader, and to determine the opacity of the Gaussian ellipsoid in the target 3D primitive corresponding to the intersection point; An intersection point ignoring unit is used to ignore the intersection point if the opacity is less than a preset threshold. The rendering result determination unit is used to determine the intersection point as a valid physical collision point if the opacity is not less than the preset threshold, and to determine the physical lighting result corresponding to the valid physical collision point, and to determine the second rendering result using the physical lighting result.

[0046] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the 3D reconstruction and rendering method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0047] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0048] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0049] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the 3D reconstruction and rendering methods disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0050] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed three-dimensional reconstruction and rendering method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0051] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0052] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0053] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0054] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0055] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A three-dimensional reconstruction and rendering method, characterized in that, include: Define a data structure for 3D primitives, obtain a sparse point cloud of the 3D target to be constructed, and generate initial 3D primitives corresponding to the data structure based on the sparse point cloud; The three-dimensional primitives are three-dimensional primitives that combine Gaussian ellipsoids and micropolygons; the data structure includes a target representation layer and a target geometry layer; The initial 3D primitives are rasterized and rendered to obtain the corresponding predicted image. Based on the real image corresponding to the 3D target to be constructed, the predicted image is subjected to multidimensional loss calculation to obtain the corresponding total loss value. The multidimensional loss includes photometric loss, geometric consistency loss, normal orientation loss and neighborhood smoothing loss. Based on the total loss value and using the graphics processor to determine the parameter gradient, the target parameters of the initial 3D primitives are adjusted using the parameter gradient to obtain the corresponding adjusted 3D primitives. The surface of the 3D model composed of the adjusted 3D primitives is inspected, and a filling optimization operation is performed based on the inspection results to obtain the optimized 3D primitives. The optimized 3D primitives are determined as new initial 3D primitives, and the process jumps to the step of rasterizing the initial 3D primitives until the target optimization stop condition is met to obtain the target 3D primitives. The target 3D primitives are then rendered in parallel in dual modes to obtain the rendered 3D target.

2. The three-dimensional reconstruction and rendering method according to claim 1, characterized in that, The defined data structure for 3D primitives is used to obtain a sparse point cloud of the 3D target to be constructed, and to generate initial 3D primitives corresponding to the data structure based on the sparse point cloud, including: Define a data structure for 3D primitives; the data structure includes a target representation layer and a target geometry layer; the target representation layer includes Gaussian parameters; the Gaussian parameters include the center coordinates, rotation angle, size, opacity, and spherical harmonic coefficients of a Gaussian ellipsoid; the target geometry layer includes micropolygon parameters; the micropolygon parameters include vertex data and normals of the micropolygon; the center coordinates of the Gaussian ellipsoid coincide with the center point of the micropolygon; the normals of the micropolygon and the shortest axis direction of the Gaussian ellipsoid meet a preset approximate parallel condition; Obtain the sparse point cloud of the 3D target to be constructed, and generate initial 3D primitives with the same data structure based on the sparse point cloud.

3. The three-dimensional reconstruction and rendering method according to claim 2, characterized in that, The step of rasterizing and rendering the initial 3D primitives to obtain the corresponding predicted image includes: The initial 3D primitives are rasterized and rendered based on the Gaussian parameters of the target representation layer in the initial 3D primitives to obtain the corresponding predicted image.

4. The three-dimensional reconstruction and rendering method according to claim 3, characterized in that, The step of performing multidimensional loss calculation on the predicted image based on the real image corresponding to the three-dimensional target to be constructed, to obtain the corresponding total loss value, includes: The pixel difference between the predicted image and the real image corresponding to the three-dimensional target to be constructed is determined using the L1 loss function and the D-SSIM algorithm, and a first loss value is determined based on the obtained pixel difference value. The shape parameter difference between the shape parameters of the Gaussian ellipsoid and the micropolygon corresponding to the predicted image is determined based on the covariance matrix, and the second loss value is determined using the shape parameter difference. The direction of the shortest axis of the Gaussian ellipsoid corresponding to the predicted image is determined, and the target normal of the micropolygon corresponding to the predicted image is determined based on the normal map. A third loss value is determined based on the target normal and the direction. The surrounding neighboring primitives of each initial 3D primitive corresponding to the predicted image are determined, and a fourth loss value is determined based on the normal difference between each initial 3D primitive and its corresponding surrounding neighboring primitive. The total loss value is determined based on each loss value and its corresponding weight.

5. The three-dimensional reconstruction and rendering method according to claim 2, characterized in that, The process involves determining parameter gradients based on the total loss value using a graphics processor, adjusting the target parameters of the initial 3D primitives using these gradients to obtain adjusted 3D primitives, inspecting the surface of the 3D model composed of the adjusted 3D primitives, and performing a filling optimization operation based on the inspection results to obtain optimized 3D primitives. This includes: Based on the total loss value, the parameter gradient is determined using the automatic differentiation algorithm of the graphics processor, and the Gaussian parameters and micro-polygon parameters of the initial 3D primitive are adjusted using the parameter gradient to obtain the corresponding adjusted 3D primitive. The surface of the three-dimensional model composed of the adjusted three-dimensional primitives is inspected. If the inspection results indicate that there are gaps or light leakage target areas on the surface of the three-dimensional model, then the adjusted three-dimensional primitives are split or cloned to obtain the filled three-dimensional primitives. The target region is filled and optimized based on the filled 3D primitives to obtain optimized 3D primitives.

6. The three-dimensional reconstruction and rendering method according to claim 2, characterized in that, The step of performing dual-mode parallel rendering on the target 3D primitives to obtain the rendered 3D target includes: The target 3D primitives are rendered using Gaussian sputtering based on the Gaussian parameters of the target representation layer to obtain a first rendering result. The target 3D primitives are traced and rendered using the micro-polygon parameters of the target geometry layer and ray tracing technology to obtain a second rendering result; The rendered 3D target is determined based on the first rendering result and the second rendering result.

7. The three-dimensional reconstruction and rendering method according to claim 6, characterized in that, The step of using the micro-polygon parameters of the target geometry layer of the target 3D primitives and ray tracing technology to trace and render the target 3D primitives to obtain a second rendering result includes: The micro-polygons of the target geometry layer of the target 3D primitive are mapped to the bottom-level acceleration result, and the micro-polygons of the target geometry layer of the target 3D primitive are determined using the bottom-level acceleration result; The intersection point of the ray and the micropolygon is determined by the ray intersection shader, and the opacity of the Gaussian ellipsoid in the target 3D primitive corresponding to the intersection point is determined. If the opacity is less than a preset threshold, the intersection point is ignored; If the opacity is not less than the preset threshold, the intersection point is determined as a valid physical collision point, and the physical lighting result corresponding to the valid physical collision point is determined. The second rendering result is then determined using the physical lighting result.

8. A three-dimensional reconstruction and rendering device, characterized in that, include: An initial primitive generation module is used to define the data structure of 3D primitives, obtain the sparse point cloud of the 3D target to be constructed, and generate each initial 3D primitive corresponding to the data structure based on the sparse point cloud; the 3D primitive is a 3D primitive combining Gaussian ellipsoids and micro polygons; the data structure includes a target representation layer and a target geometry layer; The loss calculation module is used to perform rasterization rendering on the initial 3D primitives to obtain the corresponding prediction image, and to perform multidimensional loss calculation on the prediction image based on the real image corresponding to the 3D target to be constructed to obtain the corresponding total loss value; the multidimensional loss includes photometric loss, geometric consistency loss, normal orientation loss and neighborhood smoothing loss; The model surface inspection module is used to determine the parameter gradient based on the total loss value and using the graphics processor, and to adjust the target parameters of the initial three-dimensional primitives using the parameter gradient to obtain the corresponding adjusted three-dimensional primitives. The module then inspects the surface of the three-dimensional model composed of the adjusted three-dimensional primitives and performs a filling optimization operation based on the inspection results to obtain the optimized three-dimensional primitives. The target primitive rendering module is used to determine the optimized 3D primitive as the new initial 3D primitive, and jump to the step of rasterizing the initial 3D primitive until the target optimization stop condition is met to obtain the target 3D primitive. The target 3D primitive is then rendered in parallel in dual modes to obtain the rendered 3D target.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the three-dimensional reconstruction and rendering method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the three-dimensional reconstruction and rendering method as described in any one of claims 1 to 7.

Images