Network training method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The application provides a network training method and device, electronic equipment, computer readable storage medium and computer program product. The method comprises: generating a normal map, a depth map and a reflection intensity map based on a three-dimensional Gaussian representation of a training image through an initial Gaussian rendering network; determining at least one loss of a reflection intensity loss, a depth loss and a normal estimation loss based on at least one of the normal map, the depth map and the reflection intensity map; and training the initial Gaussian rendering network based on the at least one loss until a training target is reached to obtain a Gaussian rendering network. Through the application, the image rendering effect of a neural network in implementing a new view synthesis task can be improved.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a network training method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] Currently, volumetric rendering techniques, such as 3D Gaussian rendering, have significantly improved the quality of new perspective compositing tasks. However, cross-view 3D Gaussian rendering faces difficulties in accurately rendering and geometrically reconstructing glossy areas when dealing with specular reflections in object images, resulting in poor image quality from the generated new perspectives. Summary of the Invention

[0003] This application provides a network training method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the image rendering effect of neural networks in performing new perspective synthesis tasks.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a network training method, the method comprising:

[0006] Determine the target accuracy type corresponding to the network training task;

[0007] Using an initial Gaussian rendering network, normal maps, depth maps, and reflection intensity maps are generated based on the 3D Gaussian representation of the training images;

[0008] Based on at least one of the normal map, the depth map, and the reflection intensity map, determine at least one of the reflection intensity loss, depth loss, and normal estimation loss;

[0009] Based on the at least one loss, the initial Gaussian rendering network is trained until the training objective is achieved, thus obtaining the Gaussian rendering network.

[0010] This application provides a network training device, the device comprising:

[0011] The generation module is used to generate normal maps, depth maps, and reflection intensity maps based on the 3D Gaussian representation of the training images through an initial Gaussian rendering network.

[0012] The loss determination module is used to determine at least one of the reflection intensity loss, depth loss and normal estimation loss based on at least one of the normal map, the depth map and the reflection intensity map.

[0013] A training module is used to train the initial Gaussian rendering network based on the at least one loss until the training objective is achieved, thereby obtaining the Gaussian rendering network.

[0014] Optionally, the loss determination module is further configured to:

[0015] Based on the reflection intensity map and the normal map, determine the total variation loss and the high reflection loss;

[0016] The reflection intensity loss is determined by combining the total variation loss and the high reflection loss.

[0017] Optionally, the loss determination module is further configured to:

[0018] Based on the normal map, the normal vector adjustment value corresponding to the three-dimensional Gaussian point in the three-dimensional Gaussian representation is determined; the normal vector adjustment value is used to adjust during training so that the normal vector corresponding to the three-dimensional Gaussian point estimated by the initial Gaussian rendering network approximates the true normal vector.

[0019] The normal vector adjustment value is rendered to obtain a normal vector rendering map;

[0020] Based on the rendered normal vector map, determine the normal estimation loss.

[0021] Optionally, the loss determination module is further configured to:

[0022] Determine the nearest image corresponding to the training image;

[0023] Depth estimation is performed based on the training image, the nearest image, and camera parameters to obtain a depth estimation map corresponding to the training image;

[0024] The correlation loss between the depth estimation map and the depth map is determined as the depth loss.

[0025] Optionally, the loss determination module is further configured to:

[0026] The depth estimation map is obtained by using a pre-trained depth estimation network based on the training image, the nearest image, and camera parameters; the depth estimation network is obtained through self-supervised training.

[0027] Optionally, the network training device further includes a densification module, which is further configured to determine at least one of the reflection intensity loss, depth loss, and normal estimation loss based on at least one of the normal map, the depth map, and the reflection intensity map before:

[0028] Generate a three-dimensional Gaussian representation of the training images;

[0029] Based on the reflection intensity map and camera parameters, the target three-dimensional space in the three-dimensional Gaussian representation is determined; the densification threshold corresponding to the target three-dimensional space is reduced to increase the number of three-dimensional Gaussian points in the target three-dimensional space.

[0030] Optionally, the training module is further configured to:

[0031] A base color map is generated based on the three-dimensional Gaussian representation using the initial Gaussian rendering network.

[0032] The training rendering image is obtained by rendering based on the normal map, the depth map, the reflection intensity map and the base color map;

[0033] The color loss is determined based on the training and ground truth rendering images;

[0034] By combining the color loss with the at least one loss, a total loss is determined, and the network parameters of the initial Gaussian rendering network are adjusted according to the total loss until the training objective is achieved, thus obtaining the Gaussian rendering network.

[0035] This application provides an electronic device, the electronic device comprising:

[0036] Memory is used to store executable instructions for a computer;

[0037] The processor, when executing computer-executable instructions stored in the memory, implements the network training method provided in the embodiments of this application.

[0038] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the network training method provided in this application when executed by a processor.

[0039] This application provides a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implements the network training method provided in this application.

[0040] The embodiments of this application have the following beneficial effects:

[0041] An initial Gaussian rendering network is used to generate a normal map, depth map, and reflection intensity map based on the 3D Gaussian representation of the training image. Based on at least one of the normal map, depth map, and reflection intensity map, at least one loss among reflection intensity loss, depth loss, and normal estimation loss is determined. The initial Gaussian rendering network is trained based on at least one loss until the training objective is achieved, resulting in the Gaussian rendering network. In this way, during the training process of the initial Gaussian rendering network, at least one loss among reflection intensity loss, depth loss, and normal estimation loss is used to regularize the normal rendering capability and / or depth rendering capability of the initial Gaussian rendering network under the guidance of reflection intensity, thereby improving the accuracy of the depth map and normal map generated by the Gaussian rendering network and thus improving the image rendering effect of the new perspective synthesis task. Attached Figure Description

[0042] Figure 1(a) is a schematic diagram of the reflective surface of a real object;

[0043] Figure 1(b) is the normal diagram of 3D Gaussian rendering using current related technologies;

[0044] Figure 1(c) is a depth map of 3D Gaussian rendering using current related technologies;

[0045] Figure 1(d) shows the effect of the new perspective obtained by using the current related technology to perform 3D Gaussian rendering;

[0046] Figure 1(e) is a rendering of the new perspective obtained by performing 3D Gaussian rendering in an embodiment of this application;

[0047] Figure 2 This is a schematic diagram of an optional network training method provided in an embodiment of this application;

[0048] Figure 3(a) is a schematic diagram of the process of determining the total change loss in the network training method provided in the embodiments of this application;

[0049] Figure 3(b) is a schematic diagram of the process of determining high reflection loss in the network training method provided in the embodiments of this application;

[0050] Figure 4(a) is a schematic diagram of an optional process for locating the target in three-dimensional space in the network training method provided in the embodiments of this application;

[0051] Figure 4(b) is a schematic diagram of a view from one perspective provided in an embodiment of this application;

[0052] Figure 4(c) is a schematic diagram of the target three-dimensional space corresponding to the region in the view provided in the embodiment of this application;

[0053] Figure 5This is a schematic diagram comparing the rendering effect of the Gaussian rendering network obtained by the network training method provided in this application embodiment with current related technologies.

[0054] Figure 6 This is a schematic diagram of an optional structure of the network training device provided in the embodiments of this application;

[0055] Figure 7 This is a schematic diagram of an optional structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0058] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0059] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0060] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0061] Before providing a further detailed description of the embodiments of this application, the nouns and terms used in the embodiments of this application are explained, and the nouns and terms used in the embodiments of this application shall be interpreted as follows:

[0062] 1) Novel Perspective Synthesis: Given a source image, a source pose, and a target pose, this is an image processing task that renders and generates an image corresponding to the target pose.

[0063] 2) 3D Gaussian is a technique used in computer vision and graphics to describe and render 3D scenes. It uses a series of 3D Gaussian functions to represent points or voxels in the scene, recording information such as position, covariance matrix, transparency, and color. 3D Gaussian can be considered a flexible and expressive method for scene representation; it not only achieves smooth data transitions by transforming discrete data points or voxels into visual effects but also preserves fine-grained details in the scene.

[0064] 3) 3D Gaussian Splatting (3DGS): 3DGS is used to create and render 3D scenes. Each splatter contains information about its position, color, size, and transparency. When these splatter points are combined, they can highly restore the realism of the scene.

[0065] 4) Multi-View Stereo (MVS) reconstruction is a technique that reconstructs a 3D model by matching images from multiple perspectives. It utilizes images of the same object taken from different angles by multiple cameras, calculates feature points in the images, and then recovers the object's 3D geometry. Typically, the basic steps of MVS 3D reconstruction include: Image acquisition: Using multiple cameras to capture images of the same object from different angles, obtaining multi-view images. Image matching: Matching the multi-view images to find common feature points. Triangulation: Using the matched feature points and the principles of triangulation, calculating the object's 3D coordinates. Point cloud generation: Converting the calculated 3D coordinates into point cloud data, forming a preliminary 3D representation of the object. Mesh generation: Converting the point cloud data into a triangular mesh to further refine the 3D model. Texture mapping: Mapping the texture of the images onto the triangular mesh to generate the final 3D model.

[0066] MVS reconstruction technology benefits from advancements in volumetric rendering techniques in the reconstruction of 3D scenes and objects. Gaussian rendering (3DGS) and its 2D and 4D variants, as a type of volumetric rendering, can explicitly represent scenes using learned Gaussian primitives, demonstrating real-time compositing speed while ensuring high-fidelity rendering. However, when reflections exist in the scene, compositing new perspectives of reflective objects is very challenging because reflections on 3D surfaces are view-dependent, violating cross-view consistency. Therefore, current 3DGS technology, when applied to new perspective compositing tasks, can only capture weak specular reflections in new perspectives, making it difficult to accurately reconstruct reflective surfaces, resulting in poor quality new perspective composite images of reflective areas.

[0067] Specifically, the following issues may exist in synthesizing new perspectives of reflective objects with high quality:

[0068] 1) Current 3DGS networks have difficulty accurately estimating normal maps and depth maps when processing reflective surfaces. Taking the real object image in Figure 1(a) as an example, there is a reflective area shown in red on the object surface. The corresponding rendering area of ​​this reflective area in the normal map shown in Figure 1(b) and the depth map shown in Figure 1(c) is relatively blurry, which reduces the accuracy of geometric reconstruction and depth rendering based on normal maps and depth maps.

[0069] 2) Current 3DGS networks tend to learn the geometry of collapse on reflective surfaces with strong specular reflection or indirect light, as shown in Figure 1(d). The reflective surface within the red box exhibits a collapse effect compared to the real image in Figure 1(a).

[0070] This application provides a network training method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can realize a 3DGS network for reflection perception by training a 3DGS network, and then use the trained 3DGS network to achieve high-quality new perspective synthesis tasks. For example, the effect of 3D Gaussian rendering obtained by the 3DGS network trained according to this application embodiment is shown in Figure 1(e). It can be seen that the rendering effect of the 3DGS network trained according to this application embodiment on the reflection area is closer to a realistic reflection surface image than current related technologies, greatly improving the quality of 3DGS rendering and new perspective synthesis tasks.

[0071] The network training method provided in this application can be applied to electronic devices. In some embodiments, the electronic device may include a terminal or a server. For example, the terminal may include a personal portable mobile device (such as a mobile phone, tablet, laptop, etc.) or an in-vehicle terminal; the server may include an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, network services, and big data and artificial intelligence platforms. The specific choice depends on the actual situation, and this application does not limit the choice.

[0072] refer to Figure 2 , Figure 2 This is a schematic diagram of an optional flowchart of the network training method provided in an embodiment of this application. For example... Figure 2 As shown, the network training method provided in this application embodiment can be implemented by executing the process S101-S103, as follows:

[0073] S101. Using the initial Gaussian rendering network, generate normal maps, depth maps, and reflection intensity maps based on the 3D Gaussian representation of the training images.

[0074] In step S101, the three-dimensional geometric information of objects in the image is inferred from the training image at the given original viewpoint, and is used as a three-dimensional Gaussian representation. For example, the three-dimensional Gaussian representation may include the shape and position of the object. Thus, using the three-dimensional Gaussian representation obtained from the 3D reconstruction and the given target viewpoint, a target training image at the target viewpoint is generated through three-dimensional Gaussian rendering.

[0075] In some embodiments, the three-dimensional Gaussian representation of the training image may include a three-dimensional Gaussian point cloud. Each Gaussian point (also called a Gaussian sphere) in the three-dimensional Gaussian point cloud includes corresponding attribute information. For example, the attribute information of the Gaussian point may include: the point's coordinates, covariance matrix (characterizing the shape of the Gaussian point), opacity, spherical harmonic function (used to calculate the color of the Gaussian point at any viewpoint), reflection intensity, normal, and depth information, etc.

[0076] The initial Gaussian rendering network generates a 3D Gaussian representation of the training image based on the original viewpoint. Then, based on the normal, depth information and reflection intensity of each Gaussian point, it renders the normal map, depth map and reflection intensity map of the 3D Gaussian representation from the target viewpoint.

[0077] S102. Based on at least one of the normal map, depth map, and reflection intensity map, determine at least one of the reflection intensity loss, depth loss, and normal estimation loss.

[0078] In step S102, based on at least one of the normal map, depth map, and reflection intensity map, at least one loss among reflection intensity loss, depth loss, and normal estimation loss can be determined. Furthermore, based on at least one of the reflection intensity loss, depth loss, and normal estimation loss, the normal map, depth map, and reflection intensity map generated by the initial Gaussian rendering network can be regularized, thereby improving the initial Gaussian rendering network's ability to generate high-quality normal maps, depth maps, and reflection intensity maps.

[0079] In some embodiments, the calculation of reflection intensity loss can be performed by determining the total variation (TV) loss and the high reflection loss based on the reflection intensity map and the normal map; the reflection intensity loss can then be determined by combining the total variation loss and the high reflection loss.

[0080] Specifically, the reflective and high-reflective regions can be located based on the reflectivity map; the total variation loss can be determined based on the reflective region and the normal map; the high-reflective loss can be determined based on the high-reflective region and the normal map; and the reflectivity loss can be determined based on the total variation loss and the high-reflective loss.

[0081] 3DGS, as an explicit representation method, is prone to getting stuck in local optima due to overfitting, especially on the glossy surfaces of reflective objects. This can lead to a collapsed rendering of reflective surfaces and poor geometric reconstruction. This application embodiment improves the geometric reconstruction performance of the 3DGS network by regularizing the surface geometry. This application embodiment utilizes the total variation loss to achieve regularization of the surface geometry. For example, as shown in Figure 3(a), the reflection strength map in Figure 3(a) represents the smoothness and reflectivity of the object's surface. Areas with high reflection strength appear brighter visually. For instance, in the reflection strength map of Figure 3(a), the white areas in the 3D object can represent surfaces with high reflection strength. The colors in the normal map of Figure 3(a) represent the direction of the surface normals. Normals are used to determine the orientation of the surface and are crucial for calculating lighting and reflection. Changes in color in the normal map represent changes in the normal direction of the object's surface. By calculating the total variation loss between the reflection strength map and the normal map, geometric smoothing can be achieved by applying the total variation loss, thus improving the reconstruction quality. For example, the reflection area can be located by calculating the mean reflection intensity in the reflection intensity map, as shown in the following formula:

[0082]

[0083] In the formula, d represents the direction of the light ray, and β(d) represents the reflection intensity along direction d. This represents the average value of the reflection intensity β(d). This represents the reflection area obtained from the localization. Based on the reflection area... The TV loss can be calculated using the following formula:

[0084]

[0085] in, and These represent the spatial gradients of the normal N in the x and y directions, respectively. These gradients measure the degree of change of the normal in the image plane, and ⊙ represents the Hadamard product. The total TV loss L is calculated... TV By training an initial Gaussian rendering network based on the total TV loss, the blurring problem of the normal map of the reflection region in Figure 1(b) can be alleviated.

[0086] For example, Figure 3(b) shows a high-reflective surface used to represent regions on an object's surface that exhibit strong specular reflection. In the viewing direction, the normal of the high-reflective surface should be opposite to the viewing direction to achieve a more accurate reflection effect. That is, the normal vector of the high-reflective region should be close to the opposite direction of the viewing direction. Based on this physical property, the learning of 3D Gaussians can be further constrained.

[0087] In some embodiments, the high-reflectivity region is located by setting a predefined parameter τ. For example, the high-reflectivity region can be represented as {d}. i |β(d i )≥β τ}, that is, a set of all conditions β(d) i )≥β τ Direction d i The set consisting of β. τ This represents the τ-th quantile value of the reflection intensity (i.e., the τ-th percentile value). This is for areas with high reflection, such as d... i To optimize this, this embodiment of the application constrains the angle between the normal vector of the region and the viewing direction to 180°, resulting in the following formula for calculating high reflection loss:

[0088]

[0089] Here, the summation symbol represents the summation of all β(d) satisfying the condition. i )≥β τ Accumulate i, β(d) i ) represents direction d i The reflection intensity; β τIt is a threshold used to determine which areas are high reflectivity areas. <N(d i ), d i > Represents the normal N(d) i ) and the direction of observation d i The dot product of θ and y = θ, which measures the angle between the normal and the viewing direction. ||N(d) i )|| and ||d i )|| are the magnitudes (lengths) of the normal and the view direction, respectively, used to ensure that the dot product is normalized to prevent the loss function from being affected by the vector length, 1+β(d i This is used to adjust the weight of each item in the overall loss function based on the reflection intensity. hr This represents high reflectivity loss, with an optimization target close to -1. This symbolizes that for high reflectivity areas facing the observation direction, the observation direction will be directly downward from the upward normal.

[0090] Understandably, by applying a total variation loss to achieve geometric smoothing and combining the inherent physical properties of specular reflection with the viewing direction to calculate the high reflectivity loss, this strategy encourages 3DGS networks to generate smoother normal maps, reduce large variance, and improve reconstruction quality.

[0091] In some embodiments, the depth loss can be calculated by determining the nearest image corresponding to the training image by calculating the camera translation distance; depth estimation is performed based on the training image, the nearest image, and camera parameters to obtain a depth estimation map corresponding to the training image; and the correlation loss between the depth estimation map and the depth map is determined as the depth loss. In this way, the depth map generated by the initial Gaussian rendering network can be regularized based on the depth loss, thereby improving the image quality and performance of the depth map generated by the initial Gaussian rendering network through continuous training.

[0092] Experiments revealed that the collapse of the reconstructed reflection area in current related technologies is due to the inability of 3D Gaussian methods to accurately locate the object's surface. Therefore, Gaussian-based methods are prone to overfitting the mirror effect and reconstructing the mirror with hollow geometry, resulting in reflection area collapse. This application's embodiment can guide the 3D Gaussian depth as a plane by generating a depth estimation map, thereby regularizing the depth map drawn by the 3D Gaussian method and avoiding surface collapse.

[0093] In some embodiments, a depth estimation map can be obtained by performing depth estimation based on training images, recent images, and camera parameters using a pre-trained depth estimation network. The depth estimation network can be trained in a self-supervised manner, thus avoiding manual or human annotation. For example, a U-Net architecture depth estimation network can be implemented based on camera parameters, performing depth estimation from both the original and target viewpoints to obtain a depth estimation map. This resulting depth estimation map contains accurate depth information, providing effective supervision for the regularization of the 3D Gaussian depth map, thereby obtaining more accurate geometric information and better view synthesis results from the target viewpoint.

[0094] For example, the depth estimation map can be represented as: D′ v =f φ (C gt C φ ,P). Among them, f φ Represents a U-Net network with parameter φ, C gt Representing the training image, C φ Let P represent the most recent image and P represent camera parameters. For example, correlation loss can include correlation-based Pearson loss, and depth loss can be calculated using the following formula:

[0095]

[0096] Where D(d) represents the depth map generated by the initial Gaussian rendering network, D v ' represents the depth estimation map generated by the depth estimation network, Cov represents the calculated covariance between the depth map and the depth estimation map, and σ represents the variance operator. σ(D v σ(D(d)) is used to normalize the covariance to obtain the correlation coefficient, which helps to measure the correlation between two depth maps. depth (D v Let D(d) represent the depth loss. The closer the depth loss is to 0, the lower the correlation between the depth map and the estimated depth map, indicating a potential problem with the depth estimation. Therefore, adjusting the network parameters based on the depth loss can improve the accuracy and consistency of the depth estimation.

[0097] In some embodiments, for calculating the normal estimation loss, the normal vector adjustment value corresponding to the 3D Gaussian point in the 3D Gaussian representation can be determined based on the normal map; the normal vector adjustment value is used to adjust during training to approximate the true normal vector of the 3D Gaussian point in the 3D Gaussian representation; the normal vector adjustment value is rendered to obtain a normal vector rendering map; and the normal estimation loss is determined based on the normal vector rendering map.

[0098] For example, the normal vector adjustment value can be a preset, learnable normal vector adjustment value. Through the initial Gaussian rendering network, normal vector estimation is performed based on the geometric information of each 3D Gaussian point in the 3D Gaussian representation, resulting in the normal vector n for each 3D Gaussian point. i This application embodiment sets a learnable normal vector adjustment value Δn. i Δn can be continuously adjusted during network training. i The value of n makes n i The shortest axis approximates the three-dimensional Gaussian point. Therefore, the normal vector can be adjusted by the value Δn. i As a regularization term, it encourages the initial Gaussian rendering network to estimate the normal vector more closely to the true normal vector. This application embodiment, through Δn... i Rendering yields its 2D texture, also known as the normal vector rendering map, which is used for subsequent constraints, as follows:

[0099]

[0100] Where, N d ΔN(d) represents the number of three-dimensional Gaussian points along direction d, where ΔN(d) is the number of points along direction d. i The 2D texture. ΔN(d) can represent the change in normal vector in direction d. To improve the accuracy of the 3D Gaussian normal vector, Δn can be regularized using the following normal vector regularization loss function. i The loss is estimated by the normal vector:

[0101] L Δn =||ΔN(d)||2

[0102] Where ||ΔN(d)||2 represents the 2-norm of ΔN(d), L Δn This represents the normal estimation loss. By applying the regularized loss function to the normal vectors described above, we obtain the normal estimation loss, which encourages more precise convergence of the 3D Gaussian normal vectors, enabling more accurate learning of the normal vectors and improving the quality of 3D reconstruction.

[0103] S103. Based on at least one loss, train the initial Gaussian rendering network until the training objective is achieved, and obtain the Gaussian rendering network.

[0104] In S103, an initial Gaussian rendering network can be used to generate a base color map based on the 3D Gaussian representation of the training image. Rendering is then performed based on the normal map, depth map, reflection intensity map, and base color map to obtain a training rendering map. The color loss is determined based on the training rendering map and the ground truth rendering map. Combining the color loss with at least one other loss, the total loss is determined, and the network parameters of the initial Gaussian rendering network are adjusted based on the total loss until the training objective is achieved, thus obtaining the Gaussian rendering network.

[0105] For example, for each training round, the initial Gaussian rendering network can generate a normal map, a depth map, a reflection intensity map, and a color map based on the 3D Gaussian representation of the training image. A learnable environment map is generated based on the depth map and the normal map. The learning environment map, the reflection intensity map, and the color map are then rendered to obtain the training rendering map corresponding to each training round.

[0106] In some embodiments, an initial Gaussian rendering network is used to generate a ray d along a preset rendering direction for a 3D Gaussian representation of a training image. This ray d passes through N in the 3D Gaussian representation. d A three-dimensional Gaussian point, through N d The weighted sum of three-dimensional Gaussian points is used to calculate the base color C(d) of each pixel by integrating the color ci(d) of the corresponding three-dimensional Gaussian point with the spherical harmonic function of that point. Then, tile-based rasterization is used to blend the colors ci, which are sorted by their respective depths zi. The final base color C(d) and depth D(d) are determined as follows:

[0107]

[0108] Where, ω i Represents weight, α j Represents transmittance, α j The opacity of a 3D Gaussian point can be used to determine its dimensionality. i The calculations yielded the following results:

[0109]

[0110] Where, x i This represents the distance between the Gaussian center and the ray-following direction d. Each 3D Gaussian point has a distance of n from the normal vector n. i and reflection intensity β i The correlation can be obtained through the following formula:

[0111]

[0112] In this embodiment, the normal vector n is determined by approximating the shortest axis of a three-dimensional Gaussian point. i And through learnable Δn i Used as a regularization term to encourage n i It is closer to the true normal vector. Therefore, the final rendered RGB image, which is also the training rendered image, can be represented as:

[0113] C=(1-β(d))C(d)+β(d)ε(d r )

[0114]

[0115] Where ε is a learnable environment map, which, exemplarily, may include an HDR map and a cube map. ε can be obtained by sampling through the reflection direction dr. Here, for ease of learning reflection colors, this embodiment assumes that all objects are reflective in ε. Then, the color loss can be defined as the color loss between the rendering and its base true value, as follows:

[0116] L rgb =λL1(C,C gt )+(1-λ)L DSSIM (C,C gt )

[0117] Wherein, the weight λ is used to balance the L1 loss and L DSSIM loss.

[0118] Based on color loss and at least one other loss, the total loss can be calculated using the following formula:

[0119] L=λ rgb L rgb +λ depth L depth +λ TV L TV +λ hr L hr +λ Δn L Δn

[0120] Among them, the total change loss L TV High reflection loss L hr and depth loss L depth This is a regularization term used to smooth the surface and avoid overfitting and sinkholes. The normal estimation loss L... Δn This is used to learn the normal vector more accurately. This can be achieved by adjusting the color loss L... rgb Total change loss L TV High reflection loss L hr Depth loss L depth Loss L estimated by normal Δn Different weights are preset, and the total loss is calculated by weighted summation. For example, the weight λ for color loss... rgb It can be set to 1. λ TV It can be set to 1e-2; λ hr It can be set to 1e-4; λ depth It can be set to 1e-1 or 1e-3, λ Δn It can be set to 1e-3, etc., and the specific choice depends on the actual situation. This application does not limit the specific choice.

[0121] The network parameters of the initial Gaussian rendering network are adjusted using the total loss. Based on the adjusted initial Gaussian rendering network, iterative training is performed using the training process described in the embodiments of this application until the preset training objective is reached, thus completing the training and obtaining the Gaussian rendering network. In this way, the Gaussian rendering network can be used to realize the rendering process in new perspective compositing tasks.

[0122] It is understood that, in this embodiment of the application, an initial Gaussian rendering network is used to generate a normal map, a depth map, and a reflection intensity map based on the 3D Gaussian representation of the training image; based on at least one of the normal map, depth map, and reflection intensity map, at least one loss among reflection intensity loss, depth loss, and normal estimation loss is determined; based on at least one loss, the initial Gaussian rendering network is trained until the training objective is achieved, thus obtaining the Gaussian rendering network. This achieves regularization of the normal rendering capability and / or depth rendering capability of the initial Gaussian rendering network during the training process, guided by reflection intensity, through at least one loss among reflection intensity loss, depth loss, and normal estimation loss, thereby improving the accuracy of the depth map and normal map generated by the trained Gaussian rendering network and thus improving the image rendering effect of the new perspective synthesis task.

[0123] In some embodiments, specular reflection, as a view-dependent form of information, presents a challenge in capturing its patterns for a limited number of explicit 3D Gaussians. This application proposes a reflection-guided densification strategy to generate more 3D Gaussians, enhancing the densification of 3D Gaussians in reflective 3D regions to better capture specular reflection information. Since 2D masks cannot be directly utilized for 3D Gaussian densification, this application employs multi-view ray tracing technology to supplement more 3D Gaussian points for learning specular reflection patterns, guided by reflection intensity.

[0124] In some embodiments, after generating a normal map, a depth map, and a reflection intensity map based on the 3D Gaussian representation of the training image through an initial Gaussian rendering network, and before determining at least one loss among the reflection intensity loss, depth loss, and normal estimation loss based on at least one of the normal map, depth map, and reflection intensity map, the target 3D space in the 3D Gaussian representation of the training image can be determined according to the reflection intensity map and camera parameters; the densification threshold of the target 3D space is reduced to increase the number of 3D Gaussian points in the target 3D space.

[0125] For example, in this embodiment, M bounding boxes of the 2D reflection map are first calculated, representing the highly reflective areas of the object. Then, as shown in Figure 4(a), light rays are emitted from the four 2D vertices of the bounding box in each view using camera parameters. One view can be shown in Figure 4(b). These light rays will intersect with the light rays emitted from other views. These four intersection points can locate a 3D space containing a set of important 3D Gaussians that significantly contribute to specular reflection, i.e., the target 3D space in Figure 4(a). The region corresponding to the target 3D space in the view can be shown as the red box region in Figure 4(c). By reducing the densification threshold of the target 3D space, for example, by halving the densification threshold, the densification of 3D Gaussian points in the target 3D space is enhanced. In this case, the 3D Gaussians in the target 3D space are more likely to split into more Gaussian points and adapt to specular reflection, thereby making the learning process of new viewpoint synthesis easier.

[0126] Understandably, this involves splitting the mirror information into more 3D Gaussian points specifically for the reflective areas. With a sufficient number of 3D Gaussian points, the mirror effect of the reflective areas can be more easily reconstructed, resulting in more accurate composite results from new perspectives.

[0127] In some embodiments, the 3DGS network (Ours) obtained by the network training method provided in this application can achieve the same rendering effect as current related 3DGS networks (such as Ref-NeRF, Ref-Neus, Gaussian Shader, 3DGS-DR). Figure 5 As shown, for the reflective surface of an object, the new perspective synthesis effect of the 3DGS network trained in this embodiment is clearer and closer to the real scene (Ground Truth, GT).

[0128] This application provides a network training device, such as... Figure 6 As shown, the network training device 1 may include:

[0129] The generation module 11 is used to generate a normal map, a depth map, and a reflection intensity map based on the three-dimensional Gaussian representation of the training image through an initial Gaussian rendering network.

[0130] The loss determination module 12 is used to determine at least one of the reflection intensity loss, depth loss and normal estimation loss based on at least one of the normal map, the depth map and the reflection intensity map.

[0131] Training module 13 is used to train the initial Gaussian rendering network based on the at least one loss until the training objective is achieved, thereby obtaining the Gaussian rendering network.

[0132] Optionally, the loss determination module is further configured to:

[0133] Based on the reflection intensity map and the normal map, determine the total variation loss and the high reflection loss;

[0134] The reflection intensity loss is determined by combining the total variation loss and the high reflection loss.

[0135] In some embodiments, the loss determination module 12 is further configured to:

[0136] Based on the normal map, the normal vector adjustment value corresponding to the three-dimensional Gaussian point in the three-dimensional Gaussian representation is determined; the normal vector adjustment value is used to adjust during training so that the normal vector corresponding to the three-dimensional Gaussian point estimated by the initial Gaussian rendering network approximates the true normal vector.

[0137] The normal vector adjustment value is rendered to obtain a normal vector rendering map;

[0138] Based on the rendered normal vector map, determine the normal estimation loss.

[0139] In some embodiments, the loss determination module 12 is further configured to:

[0140] Determine the nearest image corresponding to the training image;

[0141] Depth estimation is performed based on the training image, the nearest image, and camera parameters to obtain a depth estimation map corresponding to the training image;

[0142] The correlation loss between the depth estimation map and the depth map is determined as the depth loss.

[0143] In some embodiments, the loss determination module 12 is further configured to:

[0144] The depth estimation map is obtained by using a pre-trained depth estimation network based on the training image, the nearest image, and camera parameters; the depth estimation network is obtained through self-supervised training.

[0145] In some embodiments, the network training apparatus 1 further includes a densification module, which is further configured to determine at least one of the reflection intensity loss, depth loss, and normal estimation loss based on at least one of the normal map, the depth map, and the reflection intensity map before:

[0146] Generate a three-dimensional Gaussian representation of the training images;

[0147] Based on the reflection intensity map and camera parameters, the target three-dimensional space in the three-dimensional Gaussian representation is determined; the densification threshold corresponding to the target three-dimensional space is reduced to increase the number of three-dimensional Gaussian points in the target three-dimensional space.

[0148] In some embodiments, the training module 13 is further configured to:

[0149] A base color map is generated based on the three-dimensional Gaussian representation using the initial Gaussian rendering network.

[0150] The training rendering image is obtained by rendering based on the normal map, the depth map, the reflection intensity map and the base color map;

[0151] The color loss is determined based on the training and ground truth rendering images;

[0152] By combining the color loss with the at least one loss, a total loss is determined, and the network parameters of the initial Gaussian rendering network are adjusted according to the total loss until the training objective is achieved, thus obtaining the Gaussian rendering network.

[0153] It should be noted that the description of the above device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0154] This application provides an electronic device, such as... Figure 7 As shown, the electronic device 3 may include: a memory 32 and a processor 33. The memory 32 and the processor 33 are connected via a communication bus 34; the memory 32 is used to store executable instructions; the processor 33 is used to implement the network training method provided in this application embodiment when executing the executable instructions stored in the memory 32.

[0155] This application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored and, when executed by a processor, will cause the processor to execute any of the network training methods provided in this application.

[0156] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0157] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0158] As an example, executable instructions may, but do not necessarily, correspond to files in the file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0159] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0160] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A network training method, characterized in that, include Using an initial Gaussian rendering network, normal maps, depth maps, and reflection intensity maps are generated based on the 3D Gaussian representation of the training images; Based on at least one of the normal map, the depth map, and the reflection intensity map, determine at least one of the reflection intensity loss, depth loss, and normal estimation loss; Based on the at least one loss, the initial Gaussian rendering network is trained until the training objective is achieved, thus obtaining the Gaussian rendering network.

2. The method according to claim 1, characterized in that, Based on at least one of the normal map, the depth map, and the reflection intensity map, the reflection intensity loss is determined, including: Based on the reflection intensity map and the normal map, determine the total variation loss and the high reflection loss; The reflection intensity loss is determined by combining the total variation loss and the high reflection loss.

3. The method according to claim 1, characterized in that, Determining the normal estimation loss based on at least one of the normal map, the depth map, and the reflection intensity map includes: Based on the normal map, the normal vector adjustment value corresponding to the three-dimensional Gaussian point in the three-dimensional Gaussian representation is determined; the normal vector adjustment value is used to adjust during training so that the normal vector corresponding to the three-dimensional Gaussian point estimated by the initial Gaussian rendering network approximates the true normal vector. The normal vector adjustment value is rendered to obtain a normal vector rendering map; Based on the rendered normal vector map, determine the normal estimation loss.

4. The method according to claim 1, characterized in that, Determining depth loss based on at least one of the normal map, the depth map, and the reflection intensity map includes: Determine the nearest image corresponding to the training image; Depth estimation is performed based on the training image, the nearest image, and camera parameters to obtain a depth estimation map corresponding to the training image; The correlation loss between the depth estimation map and the depth map is determined as the depth loss.

5. The method according to claim 4, characterized in that, The step of performing depth estimation based on the training image, the nearest image, and camera parameters to obtain a depth estimation map corresponding to the training image includes: The depth estimation map is obtained by using a pre-trained depth estimation network based on the training image, the nearest image, and camera parameters; the depth estimation network is obtained through self-supervised training.

6. The method according to any one of claims 1-5, characterized in that, Before determining at least one of the reflection intensity loss, depth loss, and normal estimation loss based on at least one of the normal map, the depth map, and the reflection intensity map, the method further includes: Generate a three-dimensional Gaussian representation of the training images; Based on the reflection intensity map and camera parameters, the target three-dimensional space in the three-dimensional Gaussian representation is determined; the densification threshold corresponding to the target three-dimensional space is reduced to increase the number of three-dimensional Gaussian points in the target three-dimensional space.

7. The method according to any one of claims 1-5, characterized in that, The step of training the initial Gaussian rendering network based on the at least one loss until the training objective is achieved, thereby obtaining the Gaussian rendering network, includes: A color map is generated based on the three-dimensional Gaussian representation using the initial Gaussian rendering network. The training rendering image is obtained by rendering based on the normal map, the depth map, the reflection intensity map, and the color map; The color loss is determined based on the training and ground truth rendering images; By combining the color loss with the at least one loss, a total loss is determined, and the network parameters of the initial Gaussian rendering network are adjusted according to the total loss until the training objective is achieved, thus obtaining the Gaussian rendering network.

8. A network training device, characterized in that, The device includes: The generation module is used to generate normal maps, depth maps, and reflection intensity maps based on the 3D Gaussian representation of the training images through an initial Gaussian rendering network. The loss determination module is used to determine at least one of the reflection intensity loss, depth loss and normal estimation loss based on at least one of the normal map, the depth map and the reflection intensity map. A training module is used to train the initial Gaussian rendering network based on the at least one loss until the training objective is achieved, thereby obtaining the Gaussian rendering network.

9. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method described in any one of claims 1 to 7.

11. A computer program product comprising computer-executable instructions, characterized in that, When the computer-executable instructions or computer program are executed by a processor, they implement the method according to any one of claims 1 to 7.

Images