3D Gaussian-based inverse rendering method, device, equipment and storage medium

Abstract

The disclosure provides a 3D Gaussian-based inverse rendering method, device, equipment and storage medium, and relates to the technical field of artificial intelligence, specifically to the technical field of computer vision, deep learning and the like, and can be applied to scenarios such as content generation based on artificial intelligence. The specific implementation scheme is as follows: based on a plurality of two-dimensional images of different perspectives of a target scene, three-dimensional modeling is performed on the target scene to obtain scene attributes of the target scene expressed based on a plurality of 3D Gaussian points; based on the scene attributes, material attributes and lighting parameters of the target scene, color values of each 3D Gaussian point of the target scene are obtained; based on the color values of each 3D Gaussian point, the target scene is projected into a two-dimensional space according to a plurality of perspectives respectively to obtain a plurality of mapping images; and based on the loss between the plurality of mapping images and the two-dimensional images of the plurality of perspectives, the scene attributes, material attributes and lighting parameters are optimized.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically computer vision, deep learning, and other technical fields, and can be applied to scenarios such as content generation based on artificial intelligence. Background Technology

[0002] Given a description of a 3D scene, the rendering process transforms the 3D scene into a 2D image. Inverse rendering, on the other hand, is the reverse process: given a series of 2D images, it reconstructs the 3D scene description. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, and storage medium for inverse rendering based on 3D Gaussian.

[0004] According to one aspect of this disclosure, a 3D Gaussian-based inverse rendering method is provided, comprising:

[0005] The target scene is modeled in three dimensions based on two-dimensional images from multiple perspectives of the target scene, and the scene attributes of the target scene are expressed based on multiple 3D Gaussian points.

[0006] Based on the scene attributes, material attributes, and lighting parameters of the target scene, the target scene is rendered to obtain the color values ​​of each 3D Gaussian point of the target scene.

[0007] Based on the color values ​​of each 3D Gaussian point, the target scene is projected into a two-dimensional space according to the multiple viewpoints to obtain multiple mapped images;

[0008] Based on the loss between the multiple mapped images and the multiple 2D images from different perspectives, the scene attributes, the material attributes, and the lighting parameters are optimized.

[0009] According to another aspect of this disclosure, a 3D Gaussian-based inverse rendering apparatus is provided, comprising:

[0010] The acquisition module is used to perform three-dimensional modeling of the target scene based on two-dimensional images from multiple perspectives of the target scene, and obtain the scene attributes of the target scene expressed based on multiple 3D Gaussian points;

[0011] The rendering module is used to render the target scene based on the scene attributes, material attributes, and lighting parameters of the target scene, and to obtain the color values ​​of each 3D Gaussian point of the target scene.

[0012] The mapping module is used to project the target scene into a two-dimensional space according to the multiple viewpoints based on the color values ​​of each 3D Gaussian point, so as to obtain multiple mapped images;

[0013] An optimization module is used to optimize the scene attributes, the material attributes, and the lighting parameters based on the loss between the multiple mapped images and the multiple 2D images from different viewpoints.

[0014] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0015] At least one processor; and

[0016] The memory is communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0018] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0019] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0020] In this embodiment, a 3D Gaussian explicit representation of scene attributes is used. Combined with the target scene's material attributes and lighting parameters, the target scene is rendered to obtain the color values ​​of each 3D Gaussian point. Based on these color values, the target scene is projected into a two-dimensional space from multiple viewpoints, resulting in multiple mapped images. The scene attributes, material attributes, and lighting parameters are optimized based on the loss between these mapped images and the two-dimensional images from multiple viewpoints, enabling accurate inverse rendering and scene reconstruction. Furthermore, the explicit representation is easy to understand and facilitates user interaction and editing of the target scene. This explicit representation eliminates the need for implicit-to-explicit conversion operations, ensuring compatibility with modern rendering engines and improving the convenience of real-time rendering, material editing, and relighting.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 This is a flowchart illustrating a 3D Gaussian-based inverse rendering method according to an embodiment of the present disclosure.

[0024] Figure 2 This is a schematic diagram of the information carried by a 3D Gaussian point according to an embodiment of the present disclosure;

[0025] Figure 3 This is a schematic flowchart illustrating the process of obtaining the color values ​​of 3D Gaussian points according to an embodiment of the present disclosure;

[0026] Figure 4 This is a flowchart illustrating the process of determining the normal of each 3D Gaussian point according to an embodiment of the present disclosure;

[0027] Figure 5 This is another flowchart illustrating the determination of the normal of each 3D Gaussian point according to an embodiment of the present disclosure;

[0028] Figure 6 This is a schematic diagram of obtaining a Boolean visibility mask according to an embodiment of the present disclosure;

[0029] Figure 7a This is a schematic diagram of the structural framework of an FCN (Fully Convolutional Network) according to an embodiment of the present disclosure;

[0030] Figure 7b This is a schematic diagram of the process of modeling indirect lighting according to an embodiment of the present disclosure;

[0031] Figure 8 This is a schematic diagram of illumination according to an embodiment of the present disclosure;

[0032] Figure 9 This is a schematic flowchart of an indirect optical tracing process according to an embodiment of the present disclosure;

[0033] Figure 10 This is a schematic diagram of the framework of a 3D Gaussian-based inverse rendering method according to an embodiment of the present disclosure;

[0034] Figure 11 This is a schematic diagram of the structure of a 3D Gaussian-based inverse rendering apparatus according to an embodiment of the present disclosure;

[0035] Figure 12 This is a block diagram of an electronic device used to implement the 3D Gaussian-based inverse rendering method of the present disclosure embodiments. Detailed Implementation

[0036] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0037] The terms “first,” “second,” etc., used in this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0038] In related technologies, inverse rendering can infer scene attributes from a given image, including geometry, lighting information, and scene materials. The primary approach is based on implicit fields for inverse rendering, combined with neural networks to represent scene attributes. However, the implicit field method suffers from implicit-explicit conversion errors, limiting its performance and effectiveness in real-time rendering, material editing, and relighting. Furthermore, it is incompatible with many modern engines such as WebGL (Web Graphics Library), Unreal Engine, and Unity (Unity3D).

[0039] In view of this, this disclosure proposes a 3D Gaussian-based inverse rendering method. This method uses explicit 3D Gaussian representation to express scene attributes, eliminating the need for implicit-explicit conversion and better supporting real-time rendering, material editing, and relighting. Specifically, the execution flow diagram of this method is shown below. Figure 1 As shown, it includes the following:

[0040] S101, Based on two-dimensional images of the target scene from multiple perspectives, perform three-dimensional modeling of the target scene to obtain scene attributes of the target scene expressed by multiple 3D Gaussian points.

[0041] In this embodiment of the disclosure, each location in the scene is represented using a Gaussian mixture model. Each location can be represented by a 3D Gaussian point. For example... Figure 2The image shows an example of the information carried by a 3D Gaussian point, which may include: Position, Covariance matrix, Opacity, and Spherical harmonics.

[0042] It is understood that, during implementation, the number of 3D Gaussian points used in the Gaussian mixture model to represent the target scene can be optimized. This disclosure focuses on the process of reconstructing the 3D attributes of the target scene based on 3G Gaussian.

[0043] In addition to the scene attributes described above, the target scene also needs to be optimized for material attributes and lighting parameters. As the names suggest, material attributes describe the materials used in the target scene. Lighting parameters describe the ambient lighting conditions of the target scene.

[0044] S102, based on the scene attributes, material attributes, and lighting parameters of the target scene, renders the target scene to obtain the color values ​​of each 3D Gaussian point of the target scene.

[0045] S103, based on the color values ​​of each 3D Gaussian point, projects the target scene into a two-dimensional space from multiple perspectives to obtain multiple mapped images.

[0046] In other words, a 3D scene is rendered to obtain an image on a two-dimensional plane. This allows for the optimization of the 3D scene description through comparison.

[0047] S104 optimizes scene attributes, material attributes, and lighting parameters based on the loss between multiple mapped images and 2D images from multiple viewpoints.

[0048] Once the convergence criteria are met, the optimization of scene attributes, material attributes, and lighting parameters is stopped. These convergence criteria include a loss less than a preset threshold or a preset number of iterations.

[0049] In this embodiment, a 3D Gaussian explicit representation of scene attributes is used. Combined with the target scene's material attributes and lighting parameters, the target scene is rendered to obtain the color values ​​of each 3D Gaussian point. Based on these color values, the target scene is projected into a two-dimensional space from multiple viewpoints, resulting in multiple mapped images. The scene attributes, material attributes, and lighting parameters are optimized based on the loss between these mapped images and the two-dimensional images from multiple viewpoints. The entire process, based on the explicit 3D Gaussian representation, accurately completes inverse rendering and scene reconstruction. Furthermore, the explicit representation is easy to understand and facilitates user interaction, allowing for editing of the target scene's materials. The explicit representation eliminates the need for implicit-to-explicit conversion operations, ensuring compatibility with modern rendering engines and improving the convenience of real-time rendering, material editing, and relighting.

[0050] In some embodiments, the target scene is modeled in three dimensions based on two-dimensional images from multiple perspectives of the target scene to obtain multiple 3D Gaussian points. This can be implemented by processing two-dimensional images from multiple perspectives based on Structure from Motion (SfM) technology to obtain multiple 3D Gaussian points.

[0051] In some embodiments, processing two-dimensional images from multiple perspectives based on Structure from Motion (SfM) technology to obtain multiple 3D Gaussian points can be implemented as follows: extracting feature points from each two-dimensional image; matching the same feature points between different two-dimensional images to obtain feature point pairs; estimating the camera motion based on the feature point pairs to obtain the camera pose; and reconstructing multiple 3D Gaussian points in three-dimensional space based on the camera pose and the feature point positions in the two-dimensional images.

[0052] The feature points may include at least one of corner points, edges, or other significant visual features.

[0053] In some embodiments, matching the same feature points between different two-dimensional images to obtain feature point pairs can be implemented by comparing the feature descriptors corresponding to feature points in different two-dimensional images. If the feature descriptors are the same, multiple corresponding feature points are identified as feature point pairs. The feature descriptor is used for a quantized representation of the image region surrounding the feature point. Some common three-dimensional feature descriptors may include:

[0054] 1) Geometric features: such as corners, edges, planes, etc.

[0055] 2) Surface features: such as normals, curvature, texture, etc.

[0056] 3) Shape descriptors: such as shape context, shape distribution, etc.

[0057] In some embodiments, the camera motion is estimated based on feature point pairs to obtain the camera pose, which can be implemented as follows: obtaining the essential matrix based on the feature point pairs, and obtaining the camera pose based on the essential matrix, wherein the essential matrix is ​​calculated based on the camera intrinsic and extrinsic parameters, as well as the baseline length and direction between the two cameras, to obtain the projection matrix between the two cameras for triangulation, thereby obtaining the camera pose.

[0058] Based on the aforementioned method to obtain multiple 3D Gaussian points, each 3D Gaussian point, in addition to storing its own scene attributes, also stores material attributes and lighting parameters in this embodiment for subsequent rendering.

[0059] In some embodiments, to better represent the 3D scene, the scene attributes for each 3D Gaussian point include:

[0060] a) Rotation and scaling parameters related to the covariance matrix of 3D Gaussian points;

[0061] The covariance matrix of a 3D Gaussian point is used to represent the shape attribute of the 3D Gaussian point, specifically the degree of extension or flattening of the 3D Gaussian point in different directions, in order to better fit the shape of objects in the scene.

[0062] In some embodiments, the relationship between the covariance matrix and the rotation parameter and scaling reference can be as shown in equation (1):

[0063] Σ=RSS T R T (1)

[0064] Where ∑ represents the covariance matrix, R represents the rotation parameter, and R T S represents the transpose of the rotation parameters; S represents the scaling parameters. T This represents the transpose matrix of the scaling parameter.

[0065] In implementation, the covariance matrix of the 3D Gaussian points can be decomposed using spectral decomposition to obtain rotation and scaling parameters, which serve as the representation of the scene attributes of the 3D Gaussian points in this embodiment. When the rotation and scaling parameters are used as scene attributes, the covariance matrix can be estimated by optimizing these parameters.

[0066] b) Opacity of 3D Gaussian points;

[0067] The opacity of a 3D Gaussian point is used to represent the visual transparency of that 3D Gaussian point, which can be understood as how much influence that 3D Gaussian point has on the final image.

[0068] c) Spherical harmonic coefficients of 3D Gaussian points;

[0069] d) Location information of 3D Gaussian points;

[0070] The position information of a 3D Gaussian point is used to represent the location of its center in three-dimensional space. This position information is required when projecting the 3D Gaussian point into two-dimensional space.

[0071] In practice, random values ​​can be used to initialize the scene attributes, material attributes, and lighting parameters of the 3D Gaussian point.

[0072] In this embodiment of the disclosure, the covariance matrix, opacity, spherical harmonic coefficient, and position information of the 3D Gaussian point are used as the scene attributes of the 3D Gaussian point. This can directly and explicitly describe the key information of the target scene, which is convenient for subsequent rendering, editing and relighting.

[0073] In some embodiments, the material properties of the target scene may include: albedo, metallicity, and roughness.

[0074] Albedo represents the proportion of photons reflected back at a 3D Gaussian point out of the total number of photons. Albedo is a three-dimensional real vector with values ​​ranging from 0 to 1.

[0075] Here, metallicity represents the number of photons reflected at the 3D Gaussian point and the number of photons entering the 3D Gaussian point.

[0076] Roughness represents the degree of surface roughness. The higher the surface roughness of the 3D Gaussian point, the stronger the light scattering.

[0077] Among them, metallicity and roughness are both scalars, and their values ​​range from 0 to 1.

[0078] In practice, random values ​​can be used to initialize the material properties of the 3D Gaussian point.

[0079] In this embodiment of the disclosure, albedo, metallicity, and roughness are introduced to represent the target scene. These are all attributes with clear physical meanings, which facilitates user understanding and editing of the target scene, and also enables the edge rendering engine to render it.

[0080] In some embodiments, to facilitate the establishment of lighting for the target scene, the lighting parameters of the target scene may include outgoing radiance and cube map.

[0081] Among them, radiant exitance refers to the amount of radiant energy emitted by a radiation source per unit area per unit time. It is used to represent indirect illumination and is related to the spherical harmonic coefficient of the 3D Gaussian point.

[0082] Cube mapping can be understood as mapping a cube structure using a combination of multiple textures, where each 2D texture forms a face of the cube, and it can be used to construct direct lighting.

[0083] In this embodiment, direction-aware outgoing radiance is stored in each 3D Gaussian matrix, representing indirect illumination using light transmission that approximates multiple reflections. The outgoing radiance and cubemap provide a clear understanding of the lighting conditions of the target scene. These two parameters enable the modeling of ambient lighting, facilitating accurate and rapid illumination estimation.

[0084] Based on the aforementioned details regarding the scene attributes, material attributes, and lighting parameters of the target scene, the color values ​​of each 3D Gaussian point in the target scene can be rendered as follows: Figure 3 As shown, it includes:

[0085] S301, based on the covariance matrix of each 3D Gaussian point, determine the normal of each 3D Gaussian point.

[0086] S302, Model the lighting of the target scene based on the lighting parameters to obtain the lighting model;

[0087] It should be noted that the execution order of steps S301 and S302 is not restricted.

[0088] S303 renders the target scene based on normals, lighting models, and material properties, obtaining the color values ​​of each 3D Gaussian point in the target scene.

[0089] In implementation, point-based BRDF (Bidirectional Reflectance Distribution Function) can be used for point-based rendering to obtain the color values ​​of each 3D Gaussian point. The rendering method that can render the 3D target scene and obtain the color values ​​of each 3D Gaussian point is applicable to the embodiments of this disclosure.

[0090] In this embodiment of the disclosure, an explicit 3D Gaussian representation is used to express scene attributes, material attributes, and lighting parameters. Point rendering is used to accurately calculate the color values ​​of each 3D Gaussian point in the target scene, thereby facilitating accurate estimation of the loss and improving the quality of the optimized scene description.

[0091] Based on the above rendering process, it can be seen that the inverse rendering process mainly involves normal estimation and lighting estimation. The following sections will explain normal estimation and lighting estimation respectively.

[0092] 1) Normal estimation

[0093] In this embodiment of the disclosure, the normal vector of each 3D Gaussian point is determined based on the covariance matrix of each 3D Gaussian point. Figure 4 Steps S401-S402 or Figure 5 Obtain the normal of the 3D Gaussian point using any of the steps S501-S502.

[0094] In some embodiments, the normal vector of each 3D Gaussian point is determined based on the covariance matrix of each 3D Gaussian point, and steps S401-S402 are performed for each 3D Gaussian point. Specifically, this can be implemented as follows:

[0095] S401 decomposes the covariance matrix of the 3D Gaussian points to obtain the eigenvalues ​​and eigenvectors of the covariance matrix.

[0096] The covariance matrix of 3D Gaussian points can be decomposed using the spectral decomposition method to obtain the eigenvalues ​​and eigenvectors of the covariance matrix.

[0097] S402, select the eigenvector corresponding to the smallest eigenvalue as the normal vector of the 3D Gaussian point.

[0098] For example, the covariance matrix of a 3D Gaussian point is decomposed using the spectral decomposition method, yielding eigenvector 1 corresponding to eigenvalue 1, eigenvector 2 corresponding to eigenvalue 2, ..., eigenvector n corresponding to eigenvalue n. If eigenvector 2 is the smallest eigenvalue, then eigenvector 2 is selected as the normal vector of the 3D Gaussian point.

[0099] In this embodiment of the disclosure, the eigenvector corresponding to the smallest eigenvalue of each 3D Gaussian is selected to represent its normal direction. Based on this method, the normal of the 3D Gaussian point can be determined quickly and accurately. This representation method can better capture the details and changes of the scene and provide more accurate inverse rendering results.

[0100] In some embodiments, to further simplify the calculation, the normal of each 3D Gaussian point is determined based on the covariance matrix of each 3D Gaussian point, and steps S501-S502 are performed for each 3D Gaussian point. Specifically, this can be implemented as follows:

[0101] S501 decomposes the covariance matrix of the 3D Gaussian points to obtain the eigenvectors of the covariance matrix.

[0102] The covariance matrix can be decomposed based on spectral decomposition.

[0103] S502, select the shortest eigenvector as the normal of the 3D Gaussian point.

[0104] Continuing the previous example, we obtain eigenvector 1 corresponding to eigenvalue 1, eigenvector 2 corresponding to eigenvalue 2, ..., eigenvector n corresponding to eigenvalue n. The length of eigenvector 1 is a, the length of eigenvector 2 is b, ..., the length of eigenvector n is n, where b < n < ... < a, that is, eigenvector 2 is the shortest, and eigenvector 2 is selected as the normal of the 3D Gaussian point.

[0105] In this embodiment of the disclosure, the shortest eigenvector of each 3D Gaussian point is used to represent its normal direction. Based on this method, the normal can be determined quickly and accurately. This representation method can better capture the details and changes of the scene and provide more accurate inverse rendering results.

[0106] Furthermore, to improve the accuracy of normal rendering, embodiments of this disclosure provide a directional occlusion scheme to optimize normals. This can be implemented by identifying whether a 3D Gaussian point is visible in each preset viewpoint direction from multiple perspectives, obtaining the identification result; and marking the corresponding 3D Gaussian point based on the identification result. That is, identifying whether each 3D Gaussian point is visible in each preset viewpoint direction, and marking the visibility of each 3D Gaussian point in each preset viewpoint direction using a marking method. Specifically, for each 3D Gaussian point, the following steps can be performed: determining the dot product of the normal of the 3D Gaussian point with each preset viewpoint direction from the multiple perspectives; for any preset viewpoint, determining whether the 3D Gaussian point is visible or invisible based on the dot product in that preset viewpoint. For example, if the dot product is greater than a preset threshold, it is marked as visible; if the dot product is less than or equal to the preset threshold, it is marked as invisible.

[0107] Furthermore, this can be understood as the possibility that the normals determined based on S401-S402 or S501-S502 and the actual normals may not be perfectly aligned in 3D Gaussian rendering. This is because when projecting the 3D Gaussian onto 2D screen pixels, it is not necessary to determine both sides of the normal. Therefore, to optimize the matching of the normal n and the rotation matrix R, the direction of each 3D Gaussian needs to be explicitly defined. Specifically, this is achieved through the step function U... + A Boolean visibility mask is obtained, and its expression is shown in equation (2):

[0108] M = U + (n·ω i (2)

[0109] Where n is the normal vector, ω i The i-th preset view direction among multiple viewpoints.

[0110] In one example, the preset threshold can be 0, in n·ω i If the value is less than or equal to 0, then through the step function U +After processing, a mask value of 0 indicates that the visible markers of 3D Gaussian points at the preset viewpoint are invisible; in n·ω i If the value is greater than 0, then through the step function U + After processing, if the Mask value is 1, it means that the 3D Gaussian point is visible in the preset viewpoint.

[0111] A diagram illustrating the process of obtaining a Boolean visibility mask is shown below. Figure 6 As shown, ω i，1 Let n1 be the preset viewpoint of the first 3D Gaussian point, and n1 be the normal direction of the first 3D Gaussian point. i，1 If the value is greater than 0, the mask of the first 3D Gaussian point can be determined to be 1; ω i，2 Let n2 be the preset viewpoint of the second 3D Gaussian point, and n2 be the normal direction of the second 3D Gaussian point. i，2 If the value is less than 0, then the mask of the second 3D Gaussian point can be determined to be 0.

[0112] In this embodiment of the disclosure, the normal is forced to orient in the correct direction based on this method, which provides visibility of 3D Gaussian points and can improve the accuracy of subsequent rendered images.

[0113] 2) Illumination estimation

[0114] In some embodiments of this disclosure, lighting modeling may include direct lighting modeling and indirect lighting modeling. Modeling the lighting of a target scene based on lighting parameters to obtain a lighting model can be implemented as follows:

[0115] Step A1: Model the direct lighting of the target scene. .

[0116] Existing lighting representation methods suffer from complexity and computational inefficiency. Therefore, this disclosure provides a simplified lighting representation method that utilizes convolutional networks to accurately estimate high-resolution environment maps. This method enables accurate estimation of direct lighting in a scene, thereby enhancing the effectiveness of lighting decomposition.

[0117] Furthermore, direct illumination is represented using mipmap cube maps. This involves using a high-resolution HDR image (e.g., 1024×2048) and pre-integrating at different mipmap levels based on roughness. However, training directly on the high-resolution cube map often leads to local optima due to the large number of trainable parameters and unstable gradients propagating through a large number of trainable 3D Gaussians. To address this issue, embodiments of this disclosure simplify the training process by introducing an additional compact neural network to bridge the gap between the desired high-resolution HDR image and the training entity.

[0118] Specifically, the direct lighting of the target scene can be further simplified as follows: input the cube map of the first resolution into the lighting modeling neural network to obtain the light map of the second resolution as the direct lighting; wherein, the first resolution is lower than the second resolution.

[0119] This can be understood as inputting a low-resolution cube map into a lighting modeling neural network to obtain a high-resolution light map, which is then used as direct lighting. Subsequently, the color values ​​of 3D Gaussian points can be rendered based on the high-resolution light map.

[0120] In this embodiment of the disclosure, the lighting modeling neural network can accurately and quickly estimate the direct lighting in the scene, thereby enhancing the effect of lighting decomposition.

[0121] In some embodiments, the lighting modeling neural network is a fully convolutional neural network, and each convolutional layer carries an activation function.

[0122] The fully convolutional neural network can be an FCN (Fully Convolutional Network), which can be a convolutional layer including the LeakyReLU activation function. Its structure is as follows: Figure 7a As shown, after two convolutional layers, the resolution of the feature map is doubled, resulting in a high-fidelity and smooth cubemap. Of course, in practice, the resolution can be doubled after several convolutional layers, which can be set according to actual needs, and this disclosure does not limit this.

[0123] Understandably, embodiments of this disclosure introduce a differentiable function through a fully convolutional neural network to generate high-resolution direct lighting. In one example, this function is expressed as shown in Equation (3) and is used to generate and optimize a cubemap of a first resolution.

[0124] g θ :R D1×a1×b1 →R D2×a2×b2 (3)

[0125] Among them, R D1×a1×b1 The first resolution cubemap is represented by D1, which represents the dimension of the first resolution cubemap, and a1×b1 represents the resolution of the first resolution cubemap. D2 represents the dimension of the second resolution lightmap, which is generally 3, representing the red, green, and blue channels. a2×b2 represents the resolution of the second resolution lightmap. The specific values ​​here can be adjusted as needed, and this embodiment does not limit them.

[0126] In specific implementation, g θIt starts with a learnable constant and is implemented as a fully convolutional network with upsampling layers.

[0127] In practice, an ambient lighting map (i.e., a second-resolution lighting map) is obtained through a fully convolutional neural network. This ambient lighting map is then converted into a cube map, which is used in the physically based rendering process to obtain the color values ​​of 3D Gaussian points.

[0128] In this embodiment of the disclosure, using a fully convolutional neural network can double the resolution of the feature map, thereby producing a high-fidelity and smooth light map, which can effectively perform light decomposition and improve the accuracy of the lighting obtained by inverse rendering.

[0129] In addition to direct lighting, this embodiment also implements modeling of indirect lighting.

[0130] Step A2: Model the indirect lighting of the target scene to obtain a lighting model described by direct and indirect lighting.

[0131] In this embodiment of the disclosure, the lighting model based on direct and indirect lighting descriptions can effectively represent the lighting conditions of the target scene, ensuring the accuracy of the lighting obtained by inverse rendering.

[0132] Due to the multiple reflections existing between 3D Gaussians, in order to address the difficulties in simulating indirect lighting, embodiments of this disclosure propose using an approximation method to model indirect lighting. This results in the light being modeled separately as direct light L. dir and indirect light L ind In particular, the lighting model for indirect lighting is shown in equation (4):

[0133] L i (ω i ,x)=V(ω i ,x)·L dir (ω i )+L ind (ω i ,x) (4)

[0134] Where V(ω) i (x) represents the visibility of light sources outside the bounding sphere of the scene represented by the 3D Gaussian point (i.e., the indirect lighting in the first part of the following text), L dir (ω i ) represents direct lighting (which can be understood as lighting from which reflected light can be received at the viewing angle), L ind (ω i ,x) represents the second part of indirect lighting, ω i This represents the preset viewpoint, and x represents the position information.

[0135] Based on the indirect lighting model of the aforementioned expression (3), in this embodiment of the disclosure, indirect lighting is decomposed into two components: represented by the spherical harmonic coefficient L of light. ind (ω i The occlusion mask V(ω) related to the viewpoint (x) and (ω) i ,x).

[0136] For each preset viewpoint in multiple perspectives, such as Figure 7b As shown, S701-S702 are executed respectively to model indirect lighting:

[0137] S701, for any 3D Gaussian point, select a combination of visible light sources at the 3D Gaussian point and at a preset viewing angle to form the first part of indirect lighting; wherein the light source that can directly illuminate or reflect onto the 3D Gaussian point in the preset viewing angle direction is a visible light source.

[0138] For direct light, the aforementioned occlusion mask V(ω) can be used. i The expression ,x) is used to indicate whether it is visible from the preset viewpoint.

[0139] In some embodiments, in order to calculate the position x of a 3D Gaussian point from the viewpoint ω i Indirect lighting L from the direction of entry ind (ω i ,x), rays can be sampled on the hemisphere using ray tracing. Therefore, obtaining the first part of indirect illumination can be implemented as follows: For each preset viewpoint among multiple viewpoints, perform the following separately: For any light source, determine the illumination situation of the light source at the 3D Gaussian point in the preset viewpoint direction based on the ray tracing method; if the illumination situation is that the light source is visible in the preset viewpoint direction, mark the light source at the 3D Gaussian point as a visible light source; if the illumination situation is that the light source is invisible in the preset viewpoint direction, mark the light source at the 3D Gaussian point as an invisible light source.

[0140] In this embodiment of the disclosure, the light source marker of the 3D Gaussian point is determined based on the visibility of the light source in the preset viewing direction, which can effectively estimate the indirect lighting situation and help improve the efficiency of reverse rendering.

[0141] However, the aforementioned method of sampling rays on a hemisphere via ray tracing results in an exponential increase in computational complexity for tracing and rendering as the number of reflections and samples increases. To reduce computational complexity and enable accurate indirect ray tracing, this disclosure proposes using trainable spherical harmonic coefficients to encode the viewpoint ω. i This allows for the simulation of the integrated color at point x. This method approximates indirect lighting while significantly reducing computational complexity. Specifically, as shown... Figure 8 As shown, it includes:

[0142] Step S801: Represent each 3D Gaussian point using the corresponding voxel.

[0143] In practice, the geometry can be simulated by filling a binary voxel mesh with the shape of the 3D Gaussian points.

[0144] Step S802: Perform ray tracing along the ray at the preset viewpoint.

[0145] The reflection direction of ray t at the preset viewing angle can be expressed based on equation (5):

[0146] t=(n·ω i )n-ω i (5)

[0147] Where t is the reflection direction of the ray from the preset viewpoint, n is the normal at the 3D Gaussian point, and ω i This is the preset viewing angle.

[0148] Step S803: If the first reflected ray of the ray collides with the obstacle voxel, determine that any light source passing through the obstacle voxel is an invisible light source in the reflection direction at the preset viewing angle.

[0149] Step S804: If the first reflected ray of the ray does not hit the obstacle voxel, determine that each light source is a visible light source in the reflection direction of the preset viewing angle.

[0150] Specifically, this can be understood as follows: if the first reflected ray of the ray collides with the obstacle voxel, the occlusion mask Mocc of the 3D Gaussian point is determined to be 0, that is, the light source is blocked and the direct light source cannot be perceived from the preset viewing angle; if the first reflected ray of the ray does not collide with the obstacle voxel, the occlusion mask Mocc of the 3D Gaussian point is determined to be 1, that is, it is not blocked.

[0151] Or as Figure 9 As shown, ellipses represent 3D Gaussian points, and squares represent voxels for each 3D Gaussian point. n1 and n2 are the normals corresponding to the respective 3D Gaussian points. Gray arrows indicate the path of the light source, and black arrows indicate the path of rays along the preset viewing angle. Starting from the preset viewing angle shown by the camera, after the light source reflects off the first voxel and collides with the second voxel, the portion of the light source that reflects off the second voxel in a direction parallel to the preset viewing angle will be invisible.

[0152] Therefore, for the same light source, some light rays are invisible and some light rays are visible. By marking the invisible light rays with a mask, the accuracy of distance ray tracing is improved.

[0153] In this embodiment, a ray tracing method based on voxels and a preset viewpoint is used, which can effectively simulate the indirect lighting effect in the scene. Since the geometry remains unchanged at this stage, the occlusion mask (Mocc) is directly calculated by tracing rays within the given geometry, without the need to use additional trainable parameters to predict the visibility of the light source.

[0154] S702, construct the second part of indirect lighting using the spherical harmonic coefficients of the 3D Gaussian point, so as to obtain the indirect lighting of the 3D Gaussian point at the preset viewing angle expressed by the first part of indirect lighting and the second part of indirect lighting.

[0155] In this embodiment of the disclosure, the combination of visible light sources at a preset viewing angle and at a 3D Gaussian point forms the first part of indirect lighting. The spherical harmonic coefficients of the 3D Gaussian point are used to construct the second part of indirect lighting. Thus, the indirect lighting at the 3D Gaussian point at the preset viewing angle can be expressed based on the first part of indirect lighting and the second part of indirect lighting. This method can reflect the situation of direct light and the situation of indirect light, thereby effectively modeling indirect lighting.

[0156] In summary, firstly, this disclosure employs 3D Gaussian representation as an explicit representation of the geometric and material properties of a scene. This representation uses a Gaussian mixture model to represent each location in the scene and uses Gaussian functions to parameterize normals and material properties. To accurately estimate normals, this disclosure utilizes the shortest eigenvector of each 3D Gaussian to represent its normal direction. This method provides accurate normal estimation and ensures accurate normal rendering by introducing a directional masking scheme. Regarding lighting, this disclosure designs a simplified lighting representation method. By utilizing a convolutional network to accurately estimate high-resolution environment maps, this disclosure improves the estimation accuracy of direct lighting and enhances the realism of the rendering results. To simulate indirect lighting effects in a scene, this disclosure employs a voxel-based indirect lighting tracing method. Direction-aware outgoing radiance is stored in each 3D Gaussian to approximate the light transmission of multiple reflections. This method effectively simulates the effects of indirect lighting and overcomes the challenges of indirect lighting modeling in the prior art. In summary, the technical framework of this disclosure includes using 3D Gaussian representation to express scene attributes, accurately estimating and optimizing normals, simplifying lighting representation, and voxel-based indirect lighting tracing. These steps are interconnected and together constitute the complete technical implementation solution provided by this invention, resulting in more accurate, realistic, and efficient rendering results.

[0157] 3) Overall framework

[0158] In some embodiments, based on normals, lighting models, and material properties, point-based BRDF rendering is performed on the target scene to obtain the overall framework diagram of the color values ​​of each 3D Gaussian point in the target scene, as shown below. Figure 10 As shown, it includes:

[0159] 1) The cube map of the first resolution is processed through the FCN model to obtain the light map of the second resolution as direct lighting. After the light map is converted into a cube map (not shown in the figure), it can be used for BDFR rendering.

[0160] 2) The covariance matrix of 3D Gaussian points is decomposed into rotation and scaling parameters, which are then combined with the position information and opacity of 3D Gaussian points to express some geometric properties of the 2D scene.

[0161] 3) Perform spectral decomposition on the covariance matrix to obtain eigenvalues ​​and eigenvectors; then determine the normal.

[0162] On the one hand, indirect ray tracing can be performed based on the normal vector, and then an indirect lighting model can be constructed based on the spherical harmonic coefficients; in Figure 10 To more intuitively represent the 3D scene, the emitted radiance of each 3D Gaussian point is estimated based on its spherical harmonic coefficient. The calculation method for the emitted radiance of the 3D Gaussian point is shown in equation (6):

[0163]

[0164] in, Let m be the spherical harmonic coefficients of a 3D Gaussian point, where m ∈ [-l, l]. 'l' represents any preset direction, and 'l' represents the illumination range.

[0165] In this embodiment of the disclosure, the emitted radiance can be used to simulate the radiation transmission process of light energy between these 3D Gaussian points based on the physical laws of energy propagation, and to approximately solve the light energy radiation distribution on the object surface, so as to explicitly describe the 3D scene and facilitate subsequent rendering, editing and relighting.

[0166] On the other hand, the dot product of the preset viewpoint and normal can be used to obtain the Boolean visibility mask of the 3D Gaussian point;

[0167] 4) Finally, the cube map of direct lighting, the indirect lighting model, albedo, metallicity, roughness, and normal input 3D point BRDF are rendered in points to obtain the color information of each 3D Gaussian point. The color information of each 3D point and the mask obtained by the Boolean visibility mask are multiplied and projected onto a two-dimensional plane to obtain the visibility-based color value of each 3D point.

[0168] In some embodiments, based on the color values ​​of each 3D Gaussian point, the target scene is projected into a two-dimensional space according to multiple viewpoints to obtain multiple mapped images, and steps B1-B2 are performed for any preset viewpoint among the multiple viewpoints:

[0169] Step B1: Optimize the color value of any 3D Gaussian point based on the visibility marker of the 3D Gaussian point at a preset viewing angle, so that the 3D Gaussian point that is not visible at the preset viewing angle remains invisible when projected into two-dimensional space.

[0170] Step B2: Project the optimized 3D Gaussian points in the target scene onto a two-dimensional space according to a preset viewpoint to obtain the mapping image corresponding to the preset viewpoint.

[0171] During implementation, 3DGS (3D Gaussian Splatting) can be used to project the optimized 3D Gaussian points into a two-dimensional space according to a preset viewpoint.

[0172] The overall process of obtaining the color values ​​of the 3D Gaussian points and then projecting them onto the two-dimensional plane to obtain the mapped image can be shown in Equation (7).

[0173]

[0174] Where i is any 3D Gaussian point among N 3D Gaussian points; M i α is the Boolean visibility mask for the i-th 3D Gaussian point; i Let i be the opacity of the i-th 3D Gaussian point; Let j be the color value of the i-th 3D Gaussian point, where j belongs to [1, i-1].

[0175] As mentioned above Figure 5 As shown, the color value of each 3D Gaussian point, processed based on visibility, is combined with scene attribute input and rendered using 3D Gaussian Splatting technology to obtain the color value at position (i, j) corresponding to the preset viewpoint. A mapping image is then constructed based on the color value at position (i, j). Here, (i, j) represents any position in the mapping image.

[0176] During implementation, for each preset viewpoint, a pixel-by-pixel mapped image in two-dimensional space can be rendered.

[0177] In this embodiment of the disclosure, the rendering effect of the mapped image obtained in this way conforms to the observation of the preset viewpoint, thereby improving the accuracy of the color value finally projected into the two-dimensional space.

[0178] 4) Regarding the loss

[0179] In some embodiments, the loss between multiple mapped images and two-dimensional images from multiple viewpoints is determined based on the following method:

[0180] Step E1: Determine the color loss between multiple mapped images and multiple viewpoint 2D images.

[0181] For a pair of mapped images and two-dimensional images, the color loss can be expressed as shown in equation (8):

[0182]

[0183] in, This represents the color value difference between the i-th pixel in the 2D image and the i-th pixel in the mapped image. N is the total number of pixels in the image.

[0184] It should be noted that the color loss of a pair of mapped images and two-dimensional images can be obtained based on the aforementioned MSE (Mean Square Error) or RMSE (Root Mean Square Error), and this disclosure does not limit the specific method used.

[0185] The final color loss can be represented by the mean, MSE, or RMSE of the color loss for each pair of images, but this disclosure does not limit this.

[0186] Step E2: Regularize multiple mapped images and multiple viewpoint 2D images with respect to preset parameters to obtain the regularization loss between each mapped image and the corresponding 2D image with respect to the preset parameters.

[0187] Step E3: Weighted summation of color loss and regularization loss for each preset parameter to obtain the loss between multiple mapped images and two-dimensional images from multiple viewpoints.

[0188] The preset parameters include at least one of the following: depth, normal, metallicity, roughness, and ambient light.

[0189] In this embodiment of the disclosure, the loss is calculated from multiple perspectives, including color and preset parameters. The loss obtained in this way can characterize the difference between the mapped image and the two-dimensional image from multiple perspectives, which helps to smooth various attributes through regularized loss and better optimize the description of the 3D scene.

[0190] In some embodiments, when the preset parameters include at least one of depth, normal, metallicity and roughness, the following is performed for any candidate parameter: a pixel-level spatial smoothing term is applied to each pixel in the two-dimensional space in two directions to align the mapped image and the corresponding two-dimensional image on the candidate parameter to obtain the regularization loss of the candidate parameter.

[0191] Specifically, with depth included in the preset parameters, the depth-related regularization loss between each mapped image and its corresponding 2D image is obtained, as shown in Equation (9):

[0192]

[0193] Where L(d) represents the depth-based regularization loss between the mapped image and the corresponding 2D image. This indicates the regularization of the mapped image with respect to depth; d represents the regularity of a 2D image with respect to depth. ij This represents the depth of the pixel at position (i, j). It is the gradient in the x-direction. It is the gradient in the y-direction, N is the total number of pixels, ij represents the pixel at position (i, j), and I ij Let C represent the pixel at position (i, j) in a two-dimensional image. ij This represents the pixel at position (i, j) in the mapped image.

[0194] Specifically, given that the preset parameters include normals, the regularized loss between each mapped image and its corresponding 2D image with respect to the normals is obtained, as shown in Equation (10):

[0195]

[0196] Where L(n) represents the regularization loss between the mapped image and the corresponding 2D image with respect to the normal. This indicates the regularity of the mapped image with respect to the normal. n represents the regularity of a 2D image with respect to its normal. ij The normal of the pixel at position (i, j) is represented by the normal. Other parameters are similar to those described above, and will not be repeated here in this embodiment.

[0197] Specifically, given that the preset parameters include metallicity, the regularized loss between each mapped image and its corresponding two-dimensional image with respect to metallicity is obtained, as shown in Equation (11):

[0198]

[0199] Where L(m) represents the regularization loss with respect to metallicity between the mapped image and the corresponding two-dimensional image. This indicates the regularization of the mapped image with respect to metallicity; The regularization of a two-dimensional image with respect to metallicity, m ij The value represents the metallicity of the pixel at position (i, j). Other parameters are similar to those described above, and will not be repeated here in this embodiment.

[0200] Specifically, given that roughness is a preset parameter, the regularized loss between each mapped image and its corresponding two-dimensional image with respect to roughness is obtained, as shown in Equation (12):

[0201]

[0202] Where L(r) represents the roughness-related regularization loss between the mapped image and the corresponding 2D image. This indicates the regularization of the mapped image with respect to roughness; The regularity of a two-dimensional image with respect to roughness, r ij This represents the roughness of the pixel at position (i, j). Other parameters are similar to those described above, and will not be repeated here in this embodiment.

[0203] In this embodiment of the disclosure, the preset parameters are regularized to ensure that the preset parameters have smooth and gradual changes on the surface, so as to ensure the visual consistency and realism of the target scene.

[0204] In some embodiments, when the preset parameters include ambient light, determining the regularization loss for ambient light can be implemented by: determining the average value of the cube map representing the direct illumination of the ambient light, determining the sum of the absolute values ​​of the differences between the color values ​​of each point in the cube map and the average value, and obtaining the regularization loss for ambient light; wherein the optimization objective includes minimizing the sum.

[0205] Among them, the regularized loss for ambient light Used to constrain neutral and white lighting in the scene. The average value of the cube map representing ambient light (i.e., the cube map of the second resolution) can be calculated, and the difference between the color value of each point and the average value can be minimized, as shown in Equation (13):

[0206]

[0207] in, The value is obtained by averaging the RGB channels of each pixel in the cube map. i,j This represents the color value of each pixel in the cube map.

[0208] In this embodiment, the average value of the cube map representing ambient light is first determined; then, based on the sum of the absolute values ​​of the differences between the color values ​​of each point in the cube map and the average value, the regularization loss for ambient light is obtained. This method is simple to operate and easy to implement.

[0209] Given preset parameters including depth, normal, metallicity, roughness, and ambient light, the color loss and the regularization loss for each preset parameter are weighted and summed to obtain the loss L between multiple mapped images and two-dimensional images from multiple viewpoints, as shown in Equation (14):

[0210]

[0211] Wherein, λ1, λ2, λ3, λ4, λ5, and λ6 represent the weights of each item, and other parameters are similar to those described above. This embodiment will not elaborate on them one by one.

[0212] Of course, one or more preset parameters can be selected to perform a weighted summation of the color loss and the regularization loss for each preset parameter to obtain the loss between multiple mapped images and two-dimensional images from multiple perspectives. This disclosure does not limit this.

[0213] During implementation, based on the loss between multiple mapped images and two-dimensional images from multiple perspectives, backpropagation is performed based on the loss to optimize scene attributes, material attributes, and lighting parameters.

[0214] In summary, this embodiment employs an explicit 3D Gaussian representation, which offers greater flexibility and accuracy compared to the implicit field representation used in related calculations. Furthermore, by adopting a directional masking scheme and representing normals with the shortest eigenvector, this embodiment addresses the limitations of normal estimation in related technologies, improving the accuracy and stability of normals. In addition, a simplified lighting representation method is designed, utilizing convolutional networks to accurately estimate the environment map, ensuring the accuracy of lighting decomposition. A voxel-based indirect lighting tracing method is also employed, effectively simulating indirect lighting effects in the scene. This embodiment can complete scene reconstruction and material decomposition end-to-end, and the results can be directly applied to existing graphics rendering engines without the need for implicit-explicit conversion.

[0215] Based on the same technical concept, this disclosure also proposes a 3D Gaussian-based inverse rendering device 1100, such as... Figure 11 As shown, it includes:

[0216] The acquisition module 1101 is used to perform three-dimensional modeling of the target scene based on two-dimensional images from multiple perspectives of the target scene, and obtain the scene attributes of the target scene expressed based on multiple 3D Gaussian points;

[0217] The rendering module 1102 is used to render the target scene based on the scene attributes, material attributes, and lighting parameters of the target scene to obtain the color values ​​of each 3D Gaussian point of the target scene.

[0218] The mapping module 1103 is used to project the target scene into a two-dimensional space according to the multiple viewpoints based on the color values ​​of each 3D Gaussian point, so as to obtain multiple mapped images;

[0219] The optimization module 1104 is used to optimize the scene attributes, the material attributes, and the lighting parameters based on the loss between the multiple mapped images and the two-dimensional images from the multiple viewpoints.

[0220] In some embodiments, the rendering module includes:

[0221] The normal vector determination unit is used to determine the normal vector of each 3D Gaussian point based on the covariance matrix of each 3D Gaussian point; and

[0222] The lighting modeling unit is used to model the lighting of the target scene based on the lighting parameters to obtain a lighting model;

[0223] The rendering unit is used to render the target scene based on the normal, the lighting model, and the material properties to obtain the color values ​​of each 3D Gaussian point of the target scene.

[0224] In some embodiments, the normal determination unit is specifically used for:

[0225] The covariance matrix of each 3D Gaussian point is decomposed to obtain the eigenvalues ​​and eigenvectors of the covariance matrix;

[0226] The eigenvector corresponding to the smallest eigenvalue is selected as the normal vector of the 3D Gaussian point.

[0227] In some embodiments, the normal determination unit is specifically used for:

[0228] The covariance matrix of each 3D Gaussian point is decomposed to obtain the eigenvector of the covariance matrix.

[0229] The shortest eigenvector is selected as the normal of the 3D Gaussian point.

[0230] In some embodiments, a first visibility determination module is further included, for:

[0231] In each preset viewing direction of the multiple viewpoints, the visibility of the 3D Gaussian point is identified to obtain the identification result;

[0232] Based on the recognition results, the corresponding 3D Gaussian points are marked.

[0233] In some embodiments, the mapping module is configured to:

[0234] Under various preset viewpoints, the color value of any 3D Gaussian point is optimized based on the visibility marker of the 3D Gaussian point in the preset viewpoint, so that 3D Gaussian points that are not visible in the preset viewpoint remain invisible in the two-dimensional space.

[0235] The optimized 3D Gaussian points in the target scene are projected into a two-dimensional space according to the preset viewpoint to obtain the mapping image corresponding to the preset viewpoint.

[0236] In some embodiments, the lighting modeling unit is specifically used for:

[0237] Model the direct lighting of the target scene; and,

[0238] Model the indirect lighting of the target scene to obtain the lighting model described by the direct lighting and the indirect lighting.

[0239] In some embodiments, the lighting modeling unit is specifically used for:

[0240] The cube map of the first resolution is input into the lighting modeling neural network to obtain the lighting map of the second resolution, which is used as the direct lighting.

[0241] The first resolution is lower than the second resolution.

[0242] In some embodiments, the illumination modeling neural network is a fully convolutional neural network, and each convolutional layer carries an activation function.

[0243] In some embodiments, the lighting modeling unit is specifically used for:

[0244] Under various preset viewing angles, for any 3D Gaussian point, a first part of indirect illumination is formed by combining the illumination of a visible light source at the preset viewing angle and at the 3D Gaussian point, wherein the light source that can directly illuminate or reflectively illuminate the 3D Gaussian point in the preset viewing angle direction is the visible light source; and

[0245] A second part of indirect illumination is constructed using the spherical harmonic coefficients of the 3D Gaussian point to obtain the indirect illumination of the 3D Gaussian point at the preset viewing angle, expressed by the first part of indirect illumination and the second part of indirect illumination.

[0246] In some embodiments, a second visibility determination module is further included, for:

[0247] Under various preset viewpoints from multiple perspectives, for any light source, the illumination of the light source at the 3D Gaussian point in the preset viewpoint direction is determined based on the ray tracing method.

[0248] When the lighting conditions are such that the light source is visible in the preset viewing angle direction, the light source marker of the 3D Gaussian point is set as the visible light source;

[0249] When the light source is not visible in the preset viewing angle direction under the illumination condition, the light source marker of the 3D Gaussian point is set to be an invisible light source.

[0250] In some embodiments, the second visibility determination module is specifically used for:

[0251] Each 3D Gaussian point is represented by a corresponding voxel;

[0252] Ray tracing is performed along the ray from the preset viewpoint;

[0253] If the first reflected ray of the ray collides with the obstacle voxel, any light source passing through the obstacle voxel is determined to be an invisible light source in the direction of reflection at the preset viewing angle;

[0254] If the first reflected ray of the ray does not hit the obstacle voxel, it is determined that the reflection direction of each light source at the preset viewing angle is a visible light source.

[0255] In some embodiments, the illumination parameters include emitted radiance, and further include an estimation module for:

[0256] Based on the spherical harmonic coefficients of each 3D Gaussian point, the emitted radiance of the 3D Gaussian point is estimated.

[0257] In some embodiments, the apparatus further includes an optimization module for:

[0258] Determine the color loss between the plurality of mapped images and the plurality of viewpoint 2D images; and;

[0259] Regularization is performed on the multiple mapped images and the multiple viewpoint 2D images respectively with respect to preset parameters to obtain the regularization loss between each mapped image and the corresponding 2D image with respect to the preset parameters;

[0260] The color loss and the regularization loss for each preset parameter are weighted and summed to obtain the loss between the multiple mapped images and the two-dimensional images from the multiple viewpoints.

[0261] The preset parameters include at least one of the following: depth, normal, metallicity, roughness, and ambient light.

[0262] In some embodiments, when the preset parameters include at least one of the optional parameters of depth, normal, metallicity, and roughness, the optimization module is executed for any candidate parameter, specifically for:

[0263] A pixel-level spatial smoothing term is applied to each pixel in the two-dimensional space in two directions to align the mapped image and the corresponding two-dimensional image on the candidate parameters to obtain the regularization loss of the candidate parameters.

[0264] In some embodiments, when the preset parameters include ambient light, the optimization module is specifically used for:

[0265] Determine the average value of the cube map representing the direct lighting of ambient light;

[0266] The sum of the absolute values ​​of the differences between the color values ​​of each point in the cube map and the average value is determined to obtain the regularization loss for the ambient light.

[0267] The optimization objective includes minimizing the accumulated sum.

[0268] In some embodiments, the scene properties for each 3D Gaussian point include:

[0269] The rotation and scaling parameters associated with the covariance matrix of the 3D Gaussian points;

[0270] The opacity of the 3D Gaussian point;

[0271] The spherical harmonic coefficients of the 3D Gaussian point;

[0272] The location information of the 3D Gaussian point.

[0273] In some embodiments, the material properties of the target scene include:

[0274] Albedo, metallicity, and roughness.

[0275] In some embodiments, the lighting parameters of the target scene include:

[0276] Outgoing radiance and cube map.

[0277] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0278] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0279] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0280] Figure 12A schematic block diagram of an example electronic device 1200 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0281] like Figure 12 As shown, device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1202 or a computer program loaded from storage unit 1208 into random access memory (RAM) 1203. The RAM 1203 may also store various programs and data required for the operation of device 1200. The computing unit 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0282] Multiple components in device 1200 are connected to I / O interface 1205, including: input unit 1206, such as keyboard, mouse, etc.; output unit 1207, such as various types of monitors, speakers, etc.; storage unit 1208, such as disk, optical disk, etc.; and communication unit 1209, such as network card, modem, wireless transceiver, etc. Communication unit 1209 allows device 1200 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0283] The computing unit 1201 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as the 3D Gaussian-based inverse rendering method. For example, in some embodiments, the 3D Gaussian-based inverse rendering method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1208. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1200 via ROM 1202 and / or communication unit 1209. When the computer program is loaded into RAM 1203 and executed by the computing unit 1201, one or more steps of the 3D Gaussian-based inverse rendering method described above can be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to perform a 3D Gaussian-based inverse rendering method by any other suitable means (e.g., by means of firmware).

[0284] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0285] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0286] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0287] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0288] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0289] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0290] Based on the aforementioned electronic devices, this disclosure also provides a vehicle that may include electronic devices, and may also include communication components, a display screen for realizing a human-machine interface, and an information collection device for collecting information about the surrounding environment, etc., wherein the communication components, the display screen, the information collection device and the electronic devices are communicatively connected.

[0291] According to embodiments of this disclosure, the electronic device can be integrated with the communication component, display screen, and information acquisition device, or it can be separately configured with the communication component, display screen, and information acquisition device.

[0292] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0293] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A 3D Gaussian-based inverse rendering method, comprising: The target scene is modeled in three dimensions based on two-dimensional images from multiple perspectives of the target scene, and the scene attributes of the target scene are expressed based on multiple 3D Gaussian points. Based on the scene attributes, material attributes, and lighting parameters of the target scene, the target scene is rendered to obtain the color values ​​of each 3D Gaussian point in the target scene; this includes: determining the normal of each 3D Gaussian point based on the covariance matrix of each 3D Gaussian point; and modeling the lighting of the target scene based on the lighting parameters to obtain a lighting model; and rendering the target scene based on the normal, the lighting model, and the material attributes to obtain the color values ​​of each 3D Gaussian point in the target scene. Based on the color values ​​of each 3D Gaussian point, the target scene is projected into a two-dimensional space according to the multiple viewpoints to obtain multiple mapped images; Based on the loss between the multiple mapped images and the multiple 2D images from different perspectives, the scene attributes, the material attributes, and the lighting parameters are optimized.

2. The method according to claim 1, wherein, The determination of the normal vector of each 3D Gaussian point based on the covariance matrix of each 3D Gaussian point includes: The covariance matrix of each 3D Gaussian point is decomposed to obtain the eigenvalues ​​and eigenvectors of the covariance matrix; The eigenvector corresponding to the smallest eigenvalue is selected as the normal vector of the 3D Gaussian point.

3. The method according to claim 1, wherein, The determination of the normal vector of each 3D Gaussian point based on the covariance matrix of each 3D Gaussian point includes: The covariance matrix of each 3D Gaussian point is decomposed to obtain the eigenvector of the covariance matrix; The shortest eigenvector is selected as the normal of the 3D Gaussian point.

4. The method according to any one of claims 1-3, further comprising: In each preset viewing direction of the multiple viewpoints, the visibility of the 3D Gaussian point is identified to obtain the identification result; Based on the recognition results, the corresponding 3D Gaussian points are marked.

5. The method according to claim 1, wherein, Based on the color values ​​of each 3D Gaussian point, the target scene is projected into a two-dimensional space according to the multiple viewpoints to obtain multiple mapped images, including: Under each preset view of the multiple viewpoints, the color value of any 3D Gaussian point is optimized based on the visibility marker of the 3D Gaussian point in the preset viewpoint, so that the 3D Gaussian point that is not visible in the preset viewpoint remains invisible in the two-dimensional space. The optimized 3D Gaussian points in the target scene are projected into a two-dimensional space according to the preset viewpoint to obtain the mapping image corresponding to the preset viewpoint.

6. The method according to claim 1, wherein, The step of modeling the lighting of the target scene based on the lighting parameters to obtain a lighting model includes: Model the direct lighting of the target scene; and, Model the indirect lighting of the target scene to obtain the lighting model described by the direct lighting and the indirect lighting.

7. The method according to claim 6, wherein, The modeling of the direct lighting of the target scene includes: The cube map of the first resolution is input into the lighting modeling neural network to obtain the lighting map of the second resolution, which is used as the direct lighting. The first resolution is lower than the second resolution.

8. The method according to claim 7, wherein, The illumination modeling neural network is a fully convolutional neural network, and each convolutional layer carries an activation function.

9. The method according to claim 6, wherein, The modeling of the indirect lighting of the target scene includes: Under each preset viewing angle of the multiple viewing angles, for any 3D Gaussian point, a first part of indirect lighting is formed by combining the illumination of a visible light source at the preset viewing angle and at the 3D Gaussian point, wherein the light source that can directly illuminate or illuminate the 3D Gaussian point in the direction of the preset viewing angle is the visible light source; and... A second part of indirect illumination is constructed using the spherical harmonic coefficients of the 3D Gaussian point to obtain the indirect illumination of the 3D Gaussian point at the preset viewing angle, expressed by the first part of indirect illumination and the second part of indirect illumination.

10. The method of claim 9, further comprising: Under each preset viewpoint of the multiple viewpoints, for any light source, the illumination of the light source at the preset viewpoint direction at the 3D Gaussian point is determined based on the ray tracing method; When the lighting conditions are such that the light source is visible in the preset viewing angle direction, the light source marker of the 3D Gaussian point is set as the visible light source; When the light source is not visible in the preset viewing angle direction under the illumination condition, the light source marker of the 3D Gaussian point is set to be an invisible light source.

11. The method according to claim 10, wherein, The method for determining the illumination of the light source at the preset viewing angle direction at the 3D Gaussian point using ray tracing includes: Each 3D Gaussian point is represented by a corresponding voxel; Ray tracing is performed along the ray from the preset viewpoint; If the first reflected ray of the ray collides with the obstacle voxel, any light source passing through the obstacle voxel is determined to be an invisible light source in the direction of reflection at the preset viewing angle; If the first reflected ray of the ray does not hit the obstacle voxel, it is determined that the reflection direction of each light source at the preset viewing angle is a visible light source.

12. The method according to claim 9, wherein, The illumination parameters include emitted radiance, and the method further includes: Based on the spherical harmonic coefficients of each 3D Gaussian point, the emitted radiance of the 3D Gaussian point is estimated.

13. The method of claim 1, further comprising determining a loss between the plurality of mapped images and the plurality of two-dimensional images from the plurality of viewpoints based on the following method: Determine the color loss between the multiple mapped images and the multiple viewpoint 2D images; as well as; Regularization is performed on the multiple mapped images and the multiple viewpoint 2D images respectively with respect to preset parameters to obtain the regularization loss between each mapped image and the corresponding 2D image with respect to the preset parameters; The color loss and the regularization loss for each preset parameter are weighted and summed to obtain the loss between the multiple mapped images and the two-dimensional images from the multiple viewpoints. The preset parameters include at least one of the following: depth, normal, metallicity, roughness, and ambient light.

14. The method according to claim 13, wherein, If the preset parameters include at least one of the optional parameters of depth, normal, metallicity, and roughness, perform the following for any candidate parameter: A pixel-level spatial smoothing term is applied to each pixel in the two-dimensional space in two directions to align the mapped image and the corresponding two-dimensional image on the candidate parameters to obtain the regularization loss of the candidate parameters.

15. The method according to claim 13, wherein, When the preset parameters include ambient light, the regularization loss is determined for the ambient light, including: Determine the average value of the cube map representing the direct lighting of ambient light; The sum of the absolute values ​​of the differences between the color values ​​of each point in the cube map and the average value is determined to obtain the regularization loss for the ambient light. The optimization objective includes minimizing the accumulated sum.

16. The method according to claim 1, wherein, The scene properties for each 3D Gaussian point include: The rotation and scaling parameters associated with the covariance matrix of the 3D Gaussian points; The opacity of the 3D Gaussian point; The spherical harmonic coefficients of the 3D Gaussian point; The location information of the 3D Gaussian point.

17. The method according to claim 1, wherein, The material properties of the target scene include: Albedo, metallicity, and roughness.

18. The method according to claim 1, wherein, The lighting parameters of the target scene include: Outgoing radiance and cube map.

19. A 3D Gaussian-based inverse rendering apparatus, comprising: The acquisition module is used to perform three-dimensional modeling of the target scene based on two-dimensional images from multiple perspectives of the target scene, and obtain the scene attributes of the target scene expressed based on multiple 3D Gaussian points; The rendering module is used to render the target scene based on the scene attributes, material attributes, and lighting parameters of the target scene, and to obtain the color values ​​of each 3D Gaussian point of the target scene. The rendering module includes: a normal determination unit, used to determine the normal of each 3D Gaussian point based on the covariance matrix of each 3D Gaussian point; a lighting modeling unit, used to model the lighting of the target scene based on the lighting parameters to obtain a lighting model; and a rendering unit, used to render the target scene based on the normal, the lighting model, and the material properties to obtain the color value of each 3D Gaussian point of the target scene. The mapping module is used to project the target scene into a two-dimensional space according to the multiple viewpoints based on the color values ​​of each 3D Gaussian point, so as to obtain multiple mapped images; An optimization module is used to optimize the scene attributes, the material attributes, and the lighting parameters based on the loss between the multiple mapped images and the multiple 2D images from different viewpoints.

20. The apparatus according to claim 19, wherein, The normal determination unit is specifically used for: The covariance matrix of each 3D Gaussian point is decomposed to obtain the eigenvalues ​​and eigenvectors of the covariance matrix; The eigenvector corresponding to the smallest eigenvalue is selected as the normal vector of the 3D Gaussian point.

21. The apparatus according to claim 19, wherein, The normal determination unit is specifically used for: The covariance matrix of each 3D Gaussian point is decomposed to obtain the eigenvector of the covariance matrix; The shortest eigenvector is selected as the normal of the 3D Gaussian point.

22. The apparatus according to any one of claims 19-21, further comprising a first visibility determination module, configured to: In each preset viewing direction of the multiple viewpoints, the visibility of the 3D Gaussian point is identified to obtain the identification result; Based on the recognition results, the corresponding 3D Gaussian points are marked.

23. The apparatus according to claim 19, wherein, The mapping module is used for: Under each preset view of the multiple viewpoints, the color value of any 3D Gaussian point is optimized based on the visibility marker of the 3D Gaussian point in the preset viewpoint, so that the 3D Gaussian point that is not visible in the preset viewpoint remains invisible in the two-dimensional space. The optimized 3D Gaussian points in the target scene are projected into a two-dimensional space according to the preset viewpoint to obtain the mapping image corresponding to the preset viewpoint.

24. The apparatus according to claim 19, wherein, The lighting modeling unit is specifically used for: Model the direct lighting of the target scene; and, Model the indirect lighting of the target scene to obtain the lighting model described by the direct lighting and the indirect lighting.

25. The apparatus according to claim 24, wherein, The lighting modeling unit is specifically used for: The cube map of the first resolution is input into the lighting modeling neural network to obtain the lighting map of the second resolution, which is used as the direct lighting. The first resolution is lower than the second resolution.

26. The apparatus according to claim 25, wherein, The illumination modeling neural network is a fully convolutional neural network, and each convolutional layer carries an activation function.

27. The apparatus according to claim 24, wherein, The lighting modeling unit is specifically used for: Under each preset viewing angle of the multiple viewing angles, for any 3D Gaussian point, a first part of indirect lighting is formed by combining the illumination of a visible light source at the preset viewing angle and at the 3D Gaussian point, wherein the light source that can directly illuminate or illuminate the 3D Gaussian point in the direction of the preset viewing angle is the visible light source; and... A second part of indirect illumination is constructed using the spherical harmonic coefficients of the 3D Gaussian point to obtain the indirect illumination of the 3D Gaussian point at the preset viewing angle, expressed by the first part of indirect illumination and the second part of indirect illumination.

28. The apparatus of claim 27, further comprising a second visibility determination module, configured to: Under each preset viewpoint of the multiple viewpoints, for any light source, the illumination of the light source at the preset viewpoint direction at the 3D Gaussian point is determined based on the ray tracing method; When the lighting conditions are such that the light source is visible in the preset viewing angle direction, the light source marker of the 3D Gaussian point is set as the visible light source; When the light source is not visible in the preset viewing angle direction under the illumination condition, the light source marker of the 3D Gaussian point is set to be an invisible light source.

29. The apparatus according to claim 28, wherein, The second visibility determination module is specifically used for: Each 3D Gaussian point is represented by a corresponding voxel; Ray tracing is performed along the ray from the preset viewpoint; If the first reflected ray of the ray collides with the obstacle voxel, any light source passing through the obstacle voxel is determined to be an invisible light source in the direction of reflection at the preset viewing angle; If the first reflected ray of the ray does not hit the obstacle voxel, it is determined that the reflection direction of each light source at the preset viewing angle is a visible light source.

30. The apparatus according to claim 27, wherein, The illumination parameters include emitted radiance, and the device further includes an estimation module for: Based on the spherical harmonic coefficients of each 3D Gaussian point, the emitted radiance of the 3D Gaussian point is estimated.

31. The apparatus of claim 19, further comprising an optimization module, configured to: Determine the color loss between the plurality of mapped images and the plurality of viewpoint 2D images; and; Regularization is performed on the multiple mapped images and the multiple viewpoint 2D images respectively with respect to preset parameters to obtain the regularization loss between each mapped image and the corresponding 2D image with respect to the preset parameters; The color loss and the regularization loss for each preset parameter are weighted and summed to obtain the loss between the multiple mapped images and the two-dimensional images from the multiple viewpoints. in, The preset parameters include at least one of the following: depth, normal, metallicity, roughness, and ambient light.

32. The apparatus according to claim 31, wherein, When the preset parameters include at least one of the optional parameters of depth, normal, metallicity, and roughness, the optimization module is executed for any candidate parameter, specifically for: A pixel-level spatial smoothing term is applied to each pixel in the two-dimensional space in two directions to align the mapped image and the corresponding two-dimensional image on the candidate parameters to obtain the regularization loss of the candidate parameters.

33. The apparatus according to claim 31, wherein, When the preset parameters include ambient light, the optimization module is specifically used for: Determine the average value of the cube map representing the direct lighting of ambient light; The sum of the absolute values ​​of the differences between the color values ​​of each point in the cube map and the average value is determined to obtain the regularization loss for the ambient light. The optimization objective includes minimizing the accumulated sum.

34. The apparatus according to claim 19, wherein, The scene properties for each 3D Gaussian point include: The rotation and scaling parameters associated with the covariance matrix of the 3D Gaussian points; The opacity of the 3D Gaussian point; The spherical harmonic coefficients of the 3D Gaussian point; The location information of the 3D Gaussian point.

35. The apparatus according to claim 19, wherein, The material properties of the target scene include: Albedo, metallicity, and roughness.

36. The apparatus according to claim 19, wherein, The lighting parameters of the target scene include: Outgoing radiance and cube map.

37. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-18.

38. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-18.

39. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-18.

Images