Training method of rendering network, rendering method and related device

Abstract

The present disclosure provides a method for training a rendering network, a method for rendering, and related devices, relating to the technical field of artificial intelligence, in particular to the technical field of computer vision, deep learning and the like, and can be applied to scenarios such as content generation based on artificial intelligence. The specific implementation scheme is: a spherical depth estimation network based on a three-dimensional Gaussian rendering network processes a reference view and a source view to obtain a spherical depth map of the reference view; the reference view and the source view are panoramic views; a three-dimensional Gaussian model based on the three-dimensional Gaussian rendering network processes the spherical depth map and the reference view to obtain a three-dimensional Gaussian point; the three-dimensional Gaussian point is rendered to a target view angle to obtain a rendering result; a loss value is determined based on the target true value of the spherical depth map, the rendering result and the target view angle; and the model parameters of the three-dimensional Gaussian rendering network are optimized based on the loss value.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, particularly to computer vision, deep learning and other technical fields, and can be applied to scenarios such as content generation based on artificial intelligence. Background Technology

[0002] 3DGS (3D Gaussian Splatting) is an emerging 3D scene representation and rendering technology. The core advantage of 3DGS lies in its ability to provide fast, high-quality 3D scene reconstruction and rendering.

[0003] 3DGS is rapidly developing and being applied in multiple fields. For example, it has demonstrated enormous application potential in areas such as virtual reality, digital human reconstruction, AIGC (Artificial Intelligence Generated Content), and autonomous driving. Summary of the Invention

[0004] This disclosure provides a method for training a rendering network, a rendering method, and related apparatus.

[0005] According to one aspect of this disclosure, a method for training a rendering network is provided, comprising:

[0006] A spherical depth estimation network based on a 3D Gaussian rendering network processes a reference view and a source view to obtain a spherical depth map of the reference view; the reference view and the source view are panoramic images.

[0007] A 3D Gaussian model based on a 3D Gaussian rendering network processes a spherical depth map and a reference view to obtain 3D Gaussian points.

[0008] Render the 3D Gaussian points to the target viewpoint to obtain the rendering result;

[0009] The loss value is determined based on the spherical depth map, the rendering results, and the ground truth value of the target from the target's perspective;

[0010] Model parameters of a 3D Gaussian rendering network are optimized based on loss values.

[0011] According to another aspect of this disclosure, a rendering method is provided, applied to the three-dimensional Gaussian rendering network of the foregoing aspect, comprising:

[0012] Obtain the first and second panoramic images;

[0013] Input the first and second panoramic images into a 3D Gaussian rendering network to obtain 3D Gaussian points;

[0014] Render 3D Gaussian points to a specified viewpoint to obtain the rendering result.

[0015] According to another aspect of this disclosure, a training apparatus for rendering a network is provided, comprising:

[0016] The first processing module is used to process the reference view and the source view based on the spherical depth estimation network of the 3D Gaussian rendering network to obtain the spherical depth map of the reference view; the reference view and the source view are panoramic images.

[0017] The second processing module is used to process the spherical depth map and reference view of the 3D Gaussian model based on the 3D Gaussian rendering network to obtain 3D Gaussian points.

[0018] The first rendering module is used to render three-dimensional Gaussian points to the target viewpoint to obtain the rendering result;

[0019] The loss determination module is used to determine the loss value based on the spherical depth map, the rendering results, and the target ground truth from the target's perspective.

[0020] The optimization module is used to optimize the model parameters of the 3D Gaussian rendering network based on the loss value.

[0021] According to another aspect of this disclosure, a rendering apparatus is provided for use with the aforementioned three-dimensional Gaussian rendering network, the apparatus comprising:

[0022] The acquisition module is used to acquire the first panoramic image and the second panoramic image;

[0023] The fifth processing module is used to input the first panoramic image and the second panoramic image into the 3D Gaussian rendering network to obtain 3D Gaussian points;

[0024] The second rendering module is used to render three-dimensional Gaussian points to a specified viewpoint to obtain the rendering result.

[0025] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0026] At least one processor; and

[0027] The memory is communicatively connected to the at least one processor; wherein,

[0028] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0029] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0030] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0031] In this embodiment, the reference view and source view are panoramic images. By predicting the depth of the panoramic image, three-dimensional Gaussian points are reconstructed to render a new perspective image. Throughout the process, a panoramic image is used to achieve depth prediction and three-dimensional scene reconstruction, providing a novel and generalizable three-dimensional Gaussian rendering method.

[0032] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0033] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0034] Figure 1 This is a flowchart illustrating the training method of a rendering network according to the first embodiment of this disclosure;

[0035] Figure 2 This is a schematic diagram of a three-dimensional Gaussian rendering network according to a second embodiment of the present disclosure;

[0036] Figure 3 This is another schematic diagram of a three-dimensional Gaussian rendering network according to a third embodiment of the present disclosure;

[0037] Figure 4 This is a schematic flowchart illustrating the process of obtaining a spherical depth map of a reference view according to the fourth embodiment of this disclosure;

[0038] Figure 5 This is a schematic flowchart illustrating the process of obtaining the first cost volume according to the fifth embodiment of this disclosure;

[0039] Figure 6 This is a schematic diagram of the training process of a three-dimensional Gaussian rendering network according to the sixth embodiment of this disclosure;

[0040] Figure 7 This is yet another schematic diagram of a three-dimensional Gaussian rendering network according to a third embodiment of the present disclosure;

[0041] Figure 8 This is a schematic flowchart of obtaining a spherical depth map according to the seventh embodiment of this disclosure;

[0042] Figure 9 This is a schematic flowchart illustrating the process of obtaining the second cost volume according to the eighth embodiment of this disclosure;

[0043] Figure 10This is another schematic diagram of the training process of a three-dimensional Gaussian rendering network according to the ninth embodiment of this disclosure;

[0044] Figure 11 This is a flowchart illustrating the rendering method according to the tenth embodiment of this disclosure;

[0045] Figure 12 This is a schematic diagram of the structure of a training apparatus for a rendering network according to the eleventh embodiment of this disclosure;

[0046] Figure 13 This is a schematic diagram of the structure of a rendering apparatus according to the twelfth embodiment of the present disclosure;

[0047] Figure 14 This is a block diagram of an electronic device used to implement the training method / rendering method of the rendering network in the embodiments of this disclosure. Detailed Implementation

[0048] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0049] The terms “first,” “second,” etc., used in this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or apparatus is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0050] In related technologies, methods for synthesizing new perspectives in panoramic images typically require lengthy single-scene optimization processes. This limits the application of 3DGS sputtering in wide-baseline panoramic images. Due to the sparse viewpoints in wide-baseline panoramic images, single-scene optimization methods easily lead to 3DGS overfitting to the input viewpoint, resulting in poor synthesis of new perspectives.

[0051] Generalizable 3D Gaussian rendering is a technique for reconstructing and rendering 3D scenes from multiple perspectives. At its core, this technique uses 3D Gaussian points to explicitly represent points in the scene and generates high-quality 3D scene images through the rendering of these Gaussian points.

[0052] Among related technologies, generalizable 3DGS methods are mainly geared towards ordinary perspective inputs. For panoramic images (spherical images), these methods still need to be optimized.

[0053] In view of this, embodiments of this disclosure provide a novel joint learning framework (hereinafter referred to as a 3D Gaussian rendering network), which is a generalizable 3DGS, mainly comprising a large-scale spherical depth estimation network and a 360-degree generalizable Gaussian decoder for 360-degree images. Embodiments of this disclosure can support 3DGS rendering of various types of panoramic images, such as spherical panoramic images, histogram panoramic images, etc.

[0054] like Figure 1 The diagram shown is a flowchart illustrating the training method for the rendering network provided in this embodiment of the present disclosure, including the following:

[0055] S101, a spherical depth estimation network based on a 3D Gaussian rendering network processes the reference view and the source view to obtain a spherical depth map of the reference view; the reference view and the source view are panoramic images.

[0056] In implementation, when there are panoramic images from multiple camera perspectives, each camera perspective panoramic image can be used as a reference view, and the panoramic images from other camera perspectives can be used as source views. For example, if the panoramic images from three camera perspectives are A, B, and C, then with A as the reference view, B and C are used as source views. Conversely, with B as the reference view, A and C are used as source views. This disclosure uses a pair of reference views and source views as an example for illustration; the processing method for each pair of reference views and source views is the same.

[0057] In the spherical depth map, the value of each pixel is the predicted depth value, which corresponds one-to-one with the pixel value in the reference view.

[0058] S102, based on the 3D Gaussian model of the 3D Gaussian rendering network, processes the spherical depth map and reference view to obtain 3D Gaussian points.

[0059] The 3D Gaussian model reconstructs 3D Gaussian points in 3D space based on the input. The attributes of these 3D Gaussian points include their covariance matrix, spherical harmonic coefficients, position, and opacity. Using the reconstructed 3D Gaussian points, panoramic images from any viewpoint can be rendered.

[0060] S103 renders the 3D Gaussian points to the target viewpoint to obtain the rendering result.

[0061] The target perspective can be determined according to actual needs, and this disclosure does not limit it.

[0062] S104. Based on the spherical depth map, rendering results, and the target ground truth from the target's perspective, determine the loss value.

[0063] The target ground truth can be understood as the realistic rendering effect of the 3D scene from the target's perspective, in order to facilitate supervised training.

[0064] S105, Optimizes model parameters of 3D Gaussian rendering network based on loss value.

[0065] like Figure 2 As shown, the generalizable 3D Gaussian rendering network provided in this embodiment includes a spherical depth estimation network and a 3D Gaussian model.

[0066] The reference view and source view are input into the spherical depth estimation network to obtain a spherical depth map. The spherical depth map is then combined with the reference view... Figure 1 By inputting a 3D Gaussian model, the system can render the target's perspective. The spherical depth map, the rendered result, and the corresponding ground truth value of the target can be compared to optimize the model parameters based on the loss value.

[0067] In this embodiment, the reference view and source view are panoramic images. By predicting the depth of the panoramic image, 3D Gaussian points are reconstructed to render a new perspective image. This provides a novel, generalizable 3D Gaussian rendering method. This method supports 3D Gaussian rendering with panoramic images as input, preserving as much information as possible in the 3D scene and improving rendering quality. Furthermore, compared to single-scene optimization, the model parameter optimization process in this embodiment is more efficient. In summary, this embodiment provides an efficient and highly accurate 3D Gaussian rendering mode.

[0068] In some embodiments, in order to achieve high-quality depth estimation and improve the rendering quality of new perspective images, as described in this disclosure, Figure 3 As shown, the spherical depth estimation network includes a first cost volume prediction network, an initial depth estimation network, and a first fusion module.

[0069] based on Figure 3 The spherical depth estimation network shown above, the aforementioned process of obtaining the spherical depth map of the reference view, can be implemented as follows: Figure 4 As shown:

[0070] S401, the first cost volume is obtained by processing the first rectangular projection map of the reference view and the second rectangular projection map of the source view based on the first cost volume prediction network.

[0071] The reference view can be converted from spherical coordinates to planar coordinates, and the resulting rectangular projection is used as the first rectangular projection.

[0072] Similarly, the source view can be converted from spherical coordinates to planar coordinates, and the resulting rectangular projection view can be used as the second rectangular projection view.

[0073] In stereo vision and multi-view stereo vision, cost volume is a core concept used to represent the matching cost under different depth assumptions. It helps calculate the depth information of each pixel in the image. The first cost volume in step S401, as the name suggests, is the cost volume determined by the prediction network based on the first cost volume.

[0074] S402, Based on the initial depth estimation network, process the cube projection map of the reference view to obtain the initial spherical depth.

[0075] The initial spherical depth is the preliminary predicted depth information. In practice, any network capable of estimating depth is applicable to the embodiments of this disclosure.

[0076] S403, the initial spherical depth and the first cost volume are fused based on the first fusion module to obtain the spherical depth map of the reference view.

[0077] In practice, the first cost volume can also be used to predict the spherical depth map. However, predicting the spherical depth map based solely on the first cost volume still results in significant depth errors, and in most outdoor 360-degree scenes, depth supervision is lacking. Therefore, in this embodiment, the depth prior provided by the initial depth estimation network is further combined with the first cost volume prediction network to improve the accuracy of depth estimation. It is understood that the entire depth estimation process relies primarily on the rectangular projections of the reference and source views, which minimizes the lossy compression caused by splitting the cube projection for depth estimation. This provides high-quality depth estimation data for Gaussian point reconstruction of the 3D Gaussian model, ultimately improving the rendering quality of the model.

[0078] In some embodiments, the first cost volume may be obtained by performing the following operations based on the first cost volume prediction network:

[0079] Step A1: Extract the first feature of the first rectangular projection image and the second feature of the second rectangular projection image.

[0080] In implementation, the first cost volume prediction network includes an encoder. By inputting a first rectangular projection map of the reference view and a second rectangular projection map of the source view into the encoder, a first feature of the reference view and a second feature of the source view can be obtained.

[0081] Step A2: Based on the first feature and the second feature, determine the first feature matching cost of multiple pixels in the reference view under multiple depth assumptions, as the first cost volume; the first feature matching cost is determined based on the similarity between the first feature vector of the multiple pixels in the first feature and the second feature vector of the second feature.

[0082] In implementation, the first cost volume prediction network may include a matching cost determination module for determining the first feature matching cost.

[0083] The first feature matching cost consists of the matching costs of multiple individual pixels, which can be understood as a multi-dimensional matrix, i.e., an initial cost volume. In implementation, for each pixel within the first rectangular projection map of the reference view, the following operations can be performed to obtain the matching cost corresponding to that pixel:

[0084] Step A21: Perform uniform sampling within a preset depth sampling range to obtain multiple depth hypotheses for the pixel.

[0085] In practice, the preset depth sampling range can vary depending on the reconstructed scene. For example, in an indoor scene, the range is [1m, 10m]; in an outdoor scene, the sampling range can be appropriately expanded. In specific implementation, the sampling range for different scenes can be set based on prior knowledge, and this embodiment does not limit this.

[0086] The number of depth hypotheses for each pixel can also be flexibly set according to requirements. Assuming that each pixel is sampled D times within the corresponding preset depth sampling range, then each pixel corresponds to D depth hypotheses, where D is a positive integer greater than 1.

[0087] Step A22: Obtain the projection point corresponding to the pixel in the source view.

[0088] In practice, the pixel points of the same location point in the panoramic images from different perspectives can be obtained by using the transformation relationship between the planar coordinates, spherical coordinates and Cartesian coordinates in the camera coordinate system of the rectangular projection view.

[0089] In both the first rectangular projection of the reference view and the second rectangular projection of the source view, planar coordinates are used to represent the positions of different pixels. The conversion relationship between planar coordinates and spherical coordinates is shown in expression (1):

[0090]

[0091] In expression (1), (u, v) represent the planar coordinate position in the rectangular projection (such as the first or second rectangular projection). (θ, φ) represent the polar angle and azimuth angle in spherical coordinates. W and H represent the width and height of the rectangular projection.

[0092] The spherical coordinate system can be transformed to the Cartesian coordinate system to obtain the position expression of each pixel in the rectangular projection map in the camera coordinate system, as shown in expression (2):

[0093]

[0094] In expression (2), (r, θ, φ) are spherical coordinates; r is the radial distance, which is the straight-line distance from the pole of the spherical coordinate system to the point. In the spherical coordinate system, this distance is fixed and represents the straight-line distance from the point to the origin; xyz are the camera coordinates.

[0095] Because the source view and the reference view belong to different camera perspectives within the same scene, the reference view and the source view can be associated based on the transformation relationship between the different camera perspectives, as shown in expression (3):

[0096]

[0097] In expression (3), i represents the camera at the reference viewpoint. This represents the position coordinates of the reference view's perspective, where W represents the transformation relationship between different camera perspectives, and j represents the source view's perspective. This indicates the position of the source view's perspective.

[0098] Using the mapping relationship described above, the same location point in the scene can be located in both the first and second rectangular projection maps. This means obtaining the projection point of a pixel in the first rectangular projection map into the second rectangular projection map.

[0099] Step A23: Obtain the feature vector of the pixel from the first feature as the first feature vector; and obtain the feature vector of the projection point of the pixel in the second rectangular projection map of the source view from the second feature as the second feature vector.

[0100] Step A24: Determine the similarity between the first feature vector and the second feature vector, as the matching cost for that pixel.

[0101] For each pixel (u, v) in the reference view ref Assuming its depth is t i (i = 1, 2, ..., D, i.e., the i-th depth assumption), find the corresponding pixels (u, v) in the source view based on the spherical projection. src Next, calculate (u, v). ref and (u, v) srcThe similarity between local features at (u, v, i), i is the similarity between the first and second feature vectors, and is considered as the value of the cost volume (first feature matching cost) at (u, v, i). Assuming the length of the feature vector is F, the process produces a 4D cost volume V after D depth hypothetical scans.

[0102] If the cost volume is too large, it will affect the inference speed. To improve the rendering speed of this network, in some embodiments, the first cost volume prediction network performs the following operations to obtain the first cost volume, such as... Figure 5 As shown, it includes the following steps:

[0103] S501, extract the first feature of the first rectangular projection map and the second feature of the second rectangular projection map.

[0104] S502, based on the first feature and the second feature, determine the first feature matching cost of multiple pixels in the reference view under multiple depth assumptions; the first feature matching cost is determined based on the similarity between the first feature vector of the multiple pixels in the first feature and the second feature vector of the second feature.

[0105] The operations of steps S501 and S502 are the same as those of steps A1 and A2 mentioned above, and will not be repeated here.

[0106] In some embodiments, the cosine similarity between feature vectors can be used as the similarity between two feature vectors. Alternatively, the similarity can be determined as shown in expression (4):

[0107]

[0108] In expression (4), C represents the similarity between two feature vectors, and F i The first feature vector represents the feature vector of the corresponding pixel in the first feature of the first rectangular projection of the reference view. Let F represent the second eigenvector mentioned above; F in the denominator is the length of the eigenvector.

[0109] S503, perform dimensionality reduction on the first feature matching cost to obtain the first cost volume.

[0110] In this embodiment, the first feature matching cost is calculated for each pixel under multiple depth assumptions, which can accurately and comprehensively describe the matching cost between the reference view and the source view, thereby improving the accuracy of the cost volume. Further dimensionality reduction operations can reduce the size of the cost volume and improve the rendering efficiency of the entire network.

[0111] In some embodiments, a three-dimensional convolutional neural network (3DCNN) can be used to process the first feature matching cost to obtain a first cost volume.

[0112] 3DCNN can effectively reduce the cost volume while retaining key information within it, thereby improving overall processing efficiency while ensuring the accuracy and efficiency of spherical depth map prediction.

[0113] Furthermore, in order to further improve processing efficiency, in this embodiment, the dimensionality reduction operation can also be implemented as follows: for the first feature matching cost, the mean operation is performed on the dimension of the first feature vector to obtain the first cost volume.

[0114] As explained above, for each pixel at each hypothetical depth, the matching cost is (u, v, i), and the feature vector length is F. Therefore, the final scale of the first feature matching cost is W*H*D*F. In implementation, the mean value of the F dimension can be calculated to make F equal to 1, resulting in a final scale of W*H*D*1 for the first cost volume.

[0115] This method not only effectively reduces cost and file size but also preserves key information and maintains the prediction accuracy of the spherical depth map. Furthermore, the averaging method is simpler and faster, effectively improving the rendering efficiency of the entire 3D Gaussian rendering network.

[0116] As explained above, while the first cost volume prediction network can estimate depth, it cannot solve the spherical distortion problem of rectangular projections. Furthermore, many outdoor scenes lack reference depths for supervised training. Therefore, relying solely on the first cost volume prediction network still results in significant depth errors and severely limits its applicability. Consequently, an initial depth estimation network is introduced in conjunction with the first cost volume prediction network.

[0117] Specifically, the initial spherical depth is obtained by processing the cube projection map of the reference view based on the initial depth estimation network, which can be implemented as follows:

[0118] Step B1: Perform depth estimation on the cube projection map to obtain a depth point cloud map.

[0119] In some embodiments, depth estimation can be performed on the cube projection map based on a geometric 3D vision network (DUSt3R) to obtain a depth point cloud map.

[0120] DUSt3R is a novel, dense, and unconstrained paradigm for stereo 3D reconstruction that requires no prior knowledge of camera calibration or viewpoint pose. It transforms the pairwise reconstruction problem into a regression of point maps, relaxing the hard constraints of traditional projection camera models. When multiple images are provided (e.g., six images in a cube projection), a simple yet effective global alignment strategy is proposed to represent all paired point maps within a common reference frame. DUSt3R's network architecture is based on standard Transformer encoders and decoders, leveraging powerful pre-trained models.

[0121] In this embodiment of the disclosure, DUSt3R can provide a good depth prior to provide a high-quality depth point cloud map, thereby improving the estimation accuracy of the spherical depth map and ultimately improving the accuracy of generalizable 3D Gaussian rendering.

[0122] In implementation, the initial depth estimation network can adopt a mature network model that can effectively provide depth priors (such as DUSt3R). During the optimization of the 3D Gaussian rendering network, the model parameters of this initial depth estimation network are frozen. That is, during the training phase, the model parameters of the initial depth estimation network do not participate in the optimization, thereby improving the efficiency of model training while ensuring the accuracy of depth estimation.

[0123] Step B2: Project the depth point cloud map onto the sphere of the reference view to obtain the initial spherical depth.

[0124] In this embodiment of the disclosure, by estimating the depth point cloud map, a good depth prior is provided for the first cost volume. By projecting it onto a sphere, the first cost volume and the depth prior can be better integrated. The combination of the two can provide each other with effective depth information, so as to transform the coarse initial spherical depth into a high-precision spherical depth map through the first cost volume.

[0125] Based on the initial spherical depth and the first cost volume, the initial spherical depth and the first cost volume are fused using the first fusion module to obtain a spherical depth map of the reference view, which can be implemented as follows:

[0126] Step C1: Fuse the initial spherical depth and the first cost volume to obtain the first fused feature;

[0127] During implementation, the initial spherical depth map and the first cost volume can be merged using concat (stitching).

[0128] Step C2: Perform depth prediction on the first fused feature to obtain a spherical depth map.

[0129] In implementation, depth prediction can be performed on the first fused features based on the first two-dimensional convolutional neural network (2D CNN) to obtain a spherical depth map. 2D CNN is a neural network architecture specifically designed for processing image data, and it has shown great potential in the fields of computer vision and image processing.

[0130] By employing 2D CNN for fusion, its local awareness and parameter sharing enable 2D CNN to efficiently capture local features in the first cost volume and initial spherical depth, reduce the number of network parameters, and improve the rendering efficiency of the 3D Gaussian rendering network.

[0131] In summary, the first fusion module, by fusing the first cost volume and the initial spherical depth and performing depth prediction, can effectively utilize the first cost volume and the initial spherical depth to estimate the spherical depth map, thereby improving rendering efficiency and accuracy.

[0132] In the entire 3D Gaussian rendering network, it is necessary to first estimate the spherical depth map, and then reconstruct the 3D Gaussian points from the 3D Gaussian model by combining the reference view.

[0133] In this embodiment of the disclosure, the 3D Gaussian model includes an encoder and a decoder. Therefore, processing a spherical depth map and a reference view based on a 3D Gaussian rendering network to obtain 3D Gaussian points can be implemented as follows: inputting the reference view into the encoder of the 3D Gaussian model, and inputting a second fusion feature into the decoder of the 3D Gaussian model, to obtain the 3D Gaussian points output by the decoder; wherein the second fusion feature is obtained by fusing the spherical depth map, the first feature, and the second feature.

[0134] This disclosure provides key feature information for a panoramic view of a 3D Gaussian model by introducing a first feature and a second feature, thereby improving the accuracy of reconstructing 3D Gaussian points from the 3D Gaussian model.

[0135] Specifically, a concat operation can be performed on the first feature, the second feature, and the spherical depth map to obtain the second fused feature.

[0136] Here, the reference view input to the encoder primarily uses the RGB color values ​​of its individual pixels. Three-dimensional Gaussian points are reconstructed using a three-dimensional Gaussian model.

[0137] Then, the reconstructed 3D Gaussian points can be rendered to any viewpoint to facilitate optimization of model parameters based on the loss.

[0138] In implementation, the target ground truth values ​​used to determine the loss may include depth ground truth, photometric ground truth, and color ground truth. Depth ground truth is used to optimize the depth accuracy predicted by the spherical depth estimation network. Photometric and color ground truth values ​​are used to enable the entire network's inference to achieve a realistic visual effect in rendering images from any viewpoint.

[0139] Therefore, determining the loss value based on the spherical depth map, the rendering result, and the ground truth of the target viewpoint can be implemented as follows: determining the loss value based on the depth loss between the depth ground truth of the spherical depth map and the target ground truth, the luminance loss between the luminance value of the rendering result and the luminance ground truth of the target ground truth, and the color value between the color value of the rendering result and the color ground truth of the ground truth.

[0140] In practice, the loss is calculated as shown in expression (5):

[0141] L joint =2β·t·(L depth +L photo )+(1-2β·t)·L rgb (5)

[0142] In expression (5), t represents the number of training steps, L depth It is a depth loss, L photo It is a loss of luminosity, L rgb β is the loss of RGB (red, green, blue) color values; β is a hyperparameter set before training.

[0143] In this embodiment of the disclosure, the ability to evaluate the model based on multiple aspects such as depth and rendering effect can improve the model rendering accuracy at different stages.

[0144] In summary, the architecture of a 3D Gaussian rendering network provided in this disclosure embodiment is as follows: Figure 6 As shown:

[0145] The first rectangular projection map of the reference view and the second rectangular projection map of the source view are input into a first cost volume prediction network. The encoder of this network (not shown in the figure) extracts the first feature of the first rectangular projection map and the second feature of the second rectangular projection map. Then, through... Figure 6 Multi-view matching in the model identifies the pixels of the same location point in two rectangular projection maps, thereby finding the first feature vector of that location point in the first feature and the second feature vector in the second feature. For each identical location point, the similarity between its corresponding first and second feature vectors is calculated, resulting in a first matching cost (not shown in the figure). Then, the first cost volume is obtained by dimensionality reduction of the first matching cost.

[0146] At the same time, such as Figure 6 As shown, in another branch, a cube projection map is extracted from the reference view and fed into the initial depth prediction network (e.g., ...). Figure 6 The initial spherical depth is obtained by using DUSt3R (in the diagram). Then, the initial spherical depth and the first cost volume are concatenated (not shown in the figure) and input into a 2DCNN (i.e., ...). Figure 6 The first fusion module (Fusion) in the process is used to perform fusion to predict the spherical depth map.

[0147] The predicted spherical depth map and related features are concatenated and then combined with a reference view and input into a 3D Gaussian Decoder to reconstruct 3D Gaussian points in the scene.

[0148] Then, the 3D Gaussian points are rendered to the target viewpoint to obtain the rendering result from the new viewpoint. Figure 6 (Not shown in the image).

[0149] Finally, the loss can be calculated based on expression (5), and the model parameters of the 3D Gaussian rendering network can be optimized accordingly. It should be noted that when DUSt3R is used in the initial deep network, the model parameters can be frozen and not participate in the optimization process to improve the model training efficiency.

[0150] remove Figure 6 In addition to the architecture shown, this disclosure also provides another three-dimensional Gaussian rendering network.

[0151] like Figure 7 As shown, the spherical depth estimation network includes a second cost volume prediction network, a monocular depth estimation network, and a second fusion module.

[0152] based on Figure 7 The architecture, based on a spherical depth estimation network using a 3D Gaussian rendering network, processes the reference view and the source view to obtain a spherical depth map of the reference view. This can be implemented as follows: Figure 8 As shown:

[0153] S801, based on the monocular depth estimation network, the cube projection map of the reference view is processed to obtain the monocular features extracted by the intermediate layer of the monocular depth estimation network.

[0154] In implementation, the monocular depth estimation network can adopt the monocular depth estimation algorithm of DepthAnythingV2. The features of its last layer can be used as monocular features; alternatively, features from other layers can be selected as monocular features depending on the specific circumstances.

[0155] S802, based on the second cost volume prediction network, the monocular features, the first rectangular projection map of the reference view, and the second rectangular projection map of the source view are processed to obtain the second cost volume.

[0156] Similar to what has been described above, predicting a spherical depth map based solely on the second cost volume still results in significant depth errors. Furthermore, in outdoor 360-degree scenes, depth supervision is often unavailable. Therefore, embodiments of this disclosure introduce monocular features extracted by a monocular depth estimation network to provide depth feature information, thereby providing effective depth priors and improving the estimation accuracy of the final spherical depth map.

[0157] S803, based on the second fusion module, perform depth prediction on the second cost volume to obtain a spherical depth map of the reference view.

[0158] In this embodiment of the disclosure, based on the second cost volume prediction network, the depth prior provided by the monocular depth estimation network is further combined to improve the accuracy of depth estimation. It is also understood that the entire depth estimation process relies primarily on the rectangular projections of the reference view and the source view, which can minimize the lossy compression caused by splitting the cube projection for depth estimation, thereby providing high-quality depth estimation data for Gaussian point reconstruction of the 3D Gaussian model and ultimately improving rendering quality.

[0159] In implementation, the monocular depth extracted by the monocular depth estimation network carries key features for depth estimation. Therefore, these features can be fused into the features extracted by the second cost volume prediction network to improve the accuracy of the second cost extraction network in determining the cost volume. Thus, based on the second cost volume prediction network processing the monocular features, the first rectangular projection map of the reference view, and the second rectangular projection map of the source view, the second cost volume is obtained, which can be implemented as follows: Figure 9 As shown:

[0160] S901, extract the first feature of the first rectangular projection map and the second feature of the second rectangular projection map.

[0161] Similar to the first cost volume prediction network, the second cost volume prediction network can extract a first feature from the first rectangular projection map and a second feature from the second rectangular projection map based on the encoder.

[0162] S902, the monocular features are fused into the first feature to obtain the third feature; and the monocular features are fused into the second feature to obtain the fourth feature.

[0163] In practice, monocular features can be concatenated to the first and second features to obtain the third and fourth features.

[0164] S903, based on the third feature and the fourth feature, determine the second feature matching cost of multiple pixels in the reference view under multiple depth assumptions; the second feature matching cost is determined based on the similarity between the third feature vector of the multiple pixels in the third feature and the fourth feature vector of the fourth feature.

[0165] Similar to the first feature matching cost, the second feature matching cost consists of the matching costs of multiple pixels in the reference view. The second feature matching cost can be understood as a multi-dimensional matrix, i.e., an initial cost volume. In implementation, for each pixel within the first rectangular projection map of the reference view, the following operations can be performed to obtain the matching cost corresponding to that pixel:

[0166] Step A21: Perform uniform sampling within a preset depth sampling range to obtain multiple depth hypotheses for the pixel.

[0167] In practice, the preset depth sampling range can vary depending on the reconstructed scene. For example, in an indoor scene, the range is [1m, 10m]; in an outdoor scene, the sampling range can be appropriately expanded. In specific implementation, the sampling range for different scenes can be set based on prior knowledge, and this embodiment does not limit this.

[0168] The number of depth hypotheses for each pixel can also be flexibly set according to requirements. Assuming that each pixel is sampled D times within the corresponding preset depth sampling range, then each pixel corresponds to D depth hypotheses, where D is a positive integer greater than 1.

[0169] Step A22: Obtain the projection point corresponding to the pixel in the source view.

[0170] In practice, the pixel points of the same location point in the panoramic images from different viewpoints can be obtained by using the transformation relationship between the planar coordinates, spherical coordinates, and Cartesian coordinates in the camera coordinate system of the rectangular projection view. For the specific implementation process, please refer to the mapping relationship shown in expressions (1)-(3) above, which will not be repeated here.

[0171] Using the mapping relationships shown in expressions (1)-(3) above, the same location point in the scene can be located in both the first and second rectangular projection maps. That is, the projection point of the pixel in the first rectangular projection map is obtained in the second rectangular projection map.

[0172] Step A23: Obtain the feature vector of the pixel from the third feature as the third feature vector; and obtain the feature vector of the projection point of the pixel in the second rectangular projection map of the source view from the fourth feature as the fourth feature vector.

[0173] Step A24: Determine the similarity between the third and fourth feature vectors, which will be used as the matching cost for that pixel.

[0174] The method for calculating the similarity between the third and fourth feature vectors can be referred to the aforementioned expression (4), and will not be repeated here in this embodiment.

[0175] For each pixel (u, v) in the reference view ref Assuming its depth is t i (i = 1, 2, ..., D, i.e., the i-th depth assumption), find the corresponding pixels (u, v) in the source view based on the spherical projection. src Next, calculate (u, v). ref and (u, v) src The similarity between local features at (u, v, i), i is the similarity between the first and second feature vectors, and is considered as the value of the cost volume at (u, v, i). Assuming the length of the feature vector is F, the process produces a 4D cost volume V after D hypothetical depth scans.

[0176] If the cost is too large, it will affect the inference speed. To improve rendering speed, the cost of second feature matching can be compressed. See S904 for details.

[0177] S904, perform dimensionality reduction on the second feature matching cost to obtain the second cost volume.

[0178] In this embodiment, for each pixel, a monocular depth extracted by a monocular depth estimation network is used to determine the matching cost. The first feature matching cost is calculated for multiple pixels under multiple depth assumptions, which can accurately and comprehensively describe the matching cost of the reference view and the source view, thereby improving the accuracy of the cost volume. Further dimensionality reduction operations can reduce the size of the cost volume and improve the rendering efficiency of the entire network.

[0179] In some embodiments, the dimensionality reduction operation on the second feature matching cost can be implemented by using a second three-dimensional convolutional neural network to process the second feature matching cost and obtain the second cost volume.

[0180] 3DCNN can effectively reduce the cost volume while retaining key information within it, thereby improving overall processing efficiency while ensuring the accuracy of spherical depth map prediction.

[0181] In addition, to further improve processing efficiency, in this embodiment, the dimensionality reduction operation can also be implemented as follows: for the second feature matching cost, the average value operation is performed on the dimension of the third feature vector to obtain the first cost volume.

[0182] As explained above, for each pixel at each hypothetical depth, the matching cost is (u, v, i), and the feature vector length is F. Therefore, the final scale of the second feature matching cost is W*H*D*F. In implementation, the mean value over the F dimension can be calculated to make F equal to 1, resulting in a final scale of W*H*D*1 for the second cost volume.

[0183] This method not only effectively reduces cost and file size but also preserves key information and maintains the prediction accuracy of the spherical depth map. Furthermore, the averaging method is simpler and faster, effectively improving the rendering efficiency of the entire 3D Gaussian rendering network.

[0184] In this embodiment of the disclosure, the 3D Gaussian model includes an encoder and a decoder. Therefore, the 3D Gaussian model based on the 3D Gaussian rendering network processes a spherical depth map and a reference view to obtain 3D Gaussian points. This can be implemented by: inputting the reference view into the encoder of the 3D Gaussian model and inputting a third fusion feature into the decoder of the 3D Gaussian model to obtain the 3D Gaussian points output by the decoder; wherein the third fusion feature is obtained by fusing the spherical depth map, a third feature, and a fourth feature.

[0185] This disclosure provides key feature information and depth information of the panoramic view for the 3D Gaussian model by introducing third and fourth features, so as to improve the accuracy of reconstructing 3D Gaussian points from the 3D Gaussian model.

[0186] The third feature, the fourth feature, and the spherical depth map can be concatted to obtain the third fused feature.

[0187] Here, the reference view input to the encoder primarily uses the RGB color values ​​of its individual pixels. Three-dimensional Gaussian points are reconstructed using a three-dimensional Gaussian model.

[0188] Then, the reconstructed 3D Gaussian points can be rendered to any viewpoint to facilitate optimization of model parameters based on the loss.

[0189] In summary, the architecture of the second 3D Gaussian rendering network provided in this embodiment is as follows: Figure 10 As shown:

[0190] The first rectangular projection map of the reference view and the second rectangular projection map of the source view are input into the second cost volume prediction network, and the encoder of the network (not shown in the figure) extracts the first feature of the first rectangular projection map and the second feature of the second rectangular projection map.

[0191] At the same time, such as Figure 10 As shown, in another branch, the reference view is split into cube projection maps, which are then fed into a monocular depth estimation network to obtain the features of its last layer output as monocular features.

[0192] The monocular feature is fused (e.g., concat) into the first and second features to obtain the third and fourth features.

[0193] Then, the pixels of the same location point in the scene are determined in the two rectangular projection maps, thereby finding the third feature vector of that location point in the third feature and the fourth feature vector in the fourth feature. For each same location point, the similarity between its corresponding third feature vector and fourth feature vector is calculated, thus obtaining the second feature matching cost. Then, by reducing the dimensionality of the second feature matching cost (not shown in the figure), the second cost volume is obtained.

[0194] The second cost volume is then fed into a 2D CNN for processing to predict the spherical depth map.

[0195] The predicted spherical depth map, the third feature, and the fourth feature are concatenated and then combined with the reference view and input into the 3D Gaussian model to reconstruct the 3D Gaussian points in the scene.

[0196] Then, the 3D Gaussian points are rendered to the target viewpoint to obtain the rendering result under the new viewpoint (not shown in the figure).

[0197] Finally, the loss can be calculated based on expression (5), and the model parameters of the 3D Gaussian rendering network can be optimized accordingly.

[0198] Furthermore, regardless Figure 6 The 3D Gaussian rendering network shown is still Figure 10 Another 3D Gaussian rendering network is shown. The 3D Gaussian model used in this embodiment is the 360Depth-shifter Gaussian Decoder. Its goal is to predict 3D Gaussian points in 3D space. The attributes of these 3D Gaussian points include the covariance matrix, spherical harmonic coefficients, position, and opacity. These 3D Gaussian points can be used to render new 360-degree view images. The 360Depth-shifter Gaussian Decoder uses explicit 3D Gaussian point representations, making the rendering process more efficient and scalable. It takes a spherical depth map and an RGB image of a reference view as input. The 3D Gaussian model is based on an encoder-decoder module built on a U-Net (Convolutional Networks for Biomedical Image Segmentation) to predict each 3D Gaussian point, resulting in a set G of 3D Gaussian points. i ={g k =(μ k , ∑ k α k S k )}, where μ kIt is the mean, representing the position of the center point of the 3D Gaussian; ∑ k It is the covariance matrix; α k It is opacity; S k It is the spherical harmonic coefficient. And, μ k This is obtained by direct conversion from a spherical depth map. Based on, for example... Figure 6 and Figure 10 The improved spherical depth shown allows for higher quality Gaussian properties, resulting in higher quality panoramic views from new perspectives.

[0199] Based on the same technical concept, this disclosure also provides a rendering method applied to a 3D Gaussian rendering network trained as described above, such as... Figure 11 As shown, it includes the following:

[0200] S1101, Obtain the first panoramic image and the second panoramic image;

[0201] In the first and second panoramic images, one panoramic image serves as the reference view, while the other panoramic image serves as the source view and participates in the reasoning process.

[0202] S1102, input the first panoramic image and the second panoramic image into the 3D Gaussian rendering network to obtain 3D Gaussian points.

[0203] The specific structure and inference process of the 3D Gaussian rendering network have been explained above and will not be repeated here. For example, in implementation, it can be adopted... Figure 6 or Figure 9 Inference is performed using any of the provided 3D Gaussian rendering networks to obtain 3D Gaussian points constructed based on the first and second panoramic images.

[0204] S1103 renders 3D Gaussian points to a specified viewpoint to obtain the rendering result.

[0205] In this embodiment of the disclosure, rangeable 3D Gaussian sputtering based on panoramic images can be supported. Panoramic images can retain more image details, reduce image loss at other rendering times, and improve rendering quality.

[0206] In related technologies, for example, generalizable spherical NeRF (Neural Radiance Fields) methods (such as PanoGRF (Generalizable Spherical Radiance Fields for Wide-baseline Panoramas)) require a large number of points to be collected, have slow inference and rendering speeds, and require pre-trained depth estimation networks.

[0207] Other generalizable 3DGS methods (such as pixel sputtering and multiview) Figure 3 Efficient 3D Gaussian scattering models (mvsplat) are perspective-oriented, not designed for panoramic images. They rely on perspective projection for coordinate transformations, ignoring the spherical nature of panoramic images. Using these methods requires first breaking the panoramic image down into perspective views; however, this breakdown is achieved through approximate interpolation, resulting in a lossy transformation.

[0208] The 3D Gaussian rendering network provided in this disclosure utilizes the full-view characteristic of panoramic images. It employs a spherical cost-volume to improve coarse depth information, which is then processed by a 3D Gaussian model to output Gaussian points aligned with the input viewpoint. This process decomposes the Gaussian generation process and the novel viewpoint rendering process, allowing real-time rendering at high frame rates (≥100 FPS). Experiments have verified that rendering at a 256*256 resolution takes approximately 5*10 seconds. -3 It can support 200 FPS.

[0209] Furthermore, this disclosure utilizes cost-volume fusion of monocular features provided by a monocular depth estimation network or depth priors predicted by DUSt3R to infer more accurate scene geometric depth (i.e., spherical depth map). Based on the optimized geometric depth, more accurate 3D-GS properties can be inferred, resulting in the rendering of more accurate new-view panoramic images.

[0210] The method provided in this disclosure can be applied to products or projects that require real-time 3D graphics rendering and computer vision integration. This method offers a novel, generalizable panoramic approach. Figure 3The D-GS framework enables inference at ≥5 FPS and real-time rendering at ≥100 FPS. Furthermore, this method achieves state-of-the-art results on datasets such as Matterport3D (Learning from RGB-D Data in Indoor Environments, a large-scale RGB-D dataset), Front3D (3D Furnished Rooms with layOuts and semaNTics), and Carla (Learning to Act), and is capable of providing real-time rendering for immersive experiences. Therefore, it can be applied to fields requiring real-time 3D graphics and computer vision technologies, such as virtual reality (VR), augmented reality (AR), autonomous driving, robot navigation, 3D modeling, and rendering.

[0211] Based on the same technical concept, this disclosure also proposes a training device 1200 for rendering networks, such as... Figure 12 As shown, it includes:

[0212] The first processing module 1201 is used to process the reference view and the source view based on the spherical depth estimation network of the 3D Gaussian rendering network to obtain the spherical depth map of the reference view; the reference view and the source view are panoramic images.

[0213] The second processing module 1202 is used to process the spherical depth map and reference view of the 3D Gaussian model based on the 3D Gaussian rendering network to obtain 3D Gaussian points.

[0214] The first rendering module 1203 is used to render three-dimensional Gaussian points to the target viewpoint to obtain the rendering result;

[0215] The loss determination module 1204 is used to determine the loss value based on the spherical depth map, the rendering result, and the target ground truth from the target viewpoint.

[0216] Optimization module 1205 is used to optimize the model parameters of the 3D Gaussian rendering network based on the loss value.

[0217] In some embodiments, the spherical depth estimation network includes a first cost volume prediction network, an initial depth estimation network, and a first fusion module; the first processing module includes:

[0218] The first processing unit is used to process the first rectangular projection map of the reference view and the second rectangular projection map of the source view based on the first cost volume prediction network to obtain the first cost volume.

[0219] The second processing unit is used to process the cube projection map of the reference view based on the initial depth estimation network to obtain the initial spherical depth.

[0220] The fusion unit is used to perform fusion processing on the initial spherical depth and the first cost volume based on the first fusion module to obtain a spherical depth map of the reference view.

[0221] In some embodiments, the first processing unit is specifically used for:

[0222] Extract the first feature of the first rectangular projection image and the second feature of the second rectangular projection image;

[0223] Based on the first feature and the second feature, the first feature matching cost of multiple pixels in the reference view is determined under multiple depth assumptions; the first feature matching cost is determined based on the similarity between the first feature vector of the multiple pixels in the first feature and the second feature vector of the second feature.

[0224] The first feature matching cost is reduced in dimensionality to obtain the first cost volume.

[0225] In some embodiments, the first processing unit is specifically used for:

[0226] For the first feature matching cost, the mean value is calculated along the dimension of the first feature vector to obtain the first cost volume.

[0227] In some embodiments, the first processing unit is specifically used for:

[0228] A three-dimensional convolutional neural network is used to process the first feature matching cost to obtain the first cost volume.

[0229] In some embodiments, the second processing unit is specifically used for:

[0230] Depth estimation is performed on the cube projection map to obtain a depth point cloud map;

[0231] Projecting the depth point cloud onto the sphere of the reference view yields the initial spherical depth.

[0232] In some embodiments, the second processing unit is specifically used for:

[0233] Depth estimation of a cube projection map is performed based on a geometric 3D vision network to obtain a depth point cloud map.

[0234] In some embodiments, the fusion unit is specifically used for:

[0235] The initial spherical depth and the first cost volume are fused to obtain the first fused feature;

[0236] Depth prediction is performed on the first fusion feature to obtain a spherical depth map.

[0237] In some embodiments, the fusion unit is specifically used for:

[0238] A spherical depth map is obtained by performing depth prediction on the first fused feature based on the first two-dimensional convolutional network.

[0239] In some embodiments, during the optimization of the 3D Gaussian rendering network, the model parameters of the initial depth estimation network are frozen and remain unchanged.

[0240] In some embodiments, the second processing module is specifically used for:

[0241] The reference view is input into the encoder of the three-dimensional Gaussian model, and the second fusion feature is input into the decoder of the three-dimensional Gaussian model to obtain the three-dimensional Gaussian points output by the decoder;

[0242] The second fusion feature is obtained by fusing the spherical depth map, the first feature, and the second feature.

[0243] In some embodiments, the spherical depth estimation network includes a second cost volume prediction network, a monocular depth estimation network, and a second fusion module; the first processing module includes:

[0244] The third processing unit is used to process the cube projection map of the reference view based on the monocular depth estimation network to obtain the monocular features extracted by the intermediate layer of the monocular depth estimation network.

[0245] The fourth processing unit is used to process the monocular features, the first rectangular projection map of the reference view, and the second rectangular projection map of the source view based on the second cost volume prediction network to obtain the second cost volume;

[0246] The prediction unit is used to perform depth prediction on the second cost volume based on the second fusion module to obtain a spherical depth map of the reference view.

[0247] In some embodiments, the fourth processing unit is specifically used for:

[0248] Extract the first feature of the first rectangular projection image and the second feature of the second rectangular projection image;

[0249] The monocular features are fused into the first feature to obtain the third feature; and the monocular features are fused into the second feature to obtain the fourth feature.

[0250] Based on the third and fourth features, the second feature matching cost of multiple pixels in the reference view is determined under multiple depth assumptions; the second feature matching cost is determined based on the similarity between the third feature vector of the third feature and the fourth feature vector of the fourth feature of the multiple pixels.

[0251] The second feature matching cost is reduced in dimensionality to obtain the second cost volume.

[0252] In some embodiments, the fourth processing unit is specifically used for:

[0253] For the second feature matching cost, the mean value is calculated along the third feature vector dimension to obtain the second cost volume.

[0254] In some embodiments, the fourth processing unit is specifically used for:

[0255] The second feature matching cost is processed using a second three-dimensional convolutional neural network to obtain the second cost volume.

[0256] In some embodiments, the second processing module is specifically used for:

[0257] The reference view is input into the encoder of the three-dimensional Gaussian model, and the third fusion feature is input into the decoder of the three-dimensional Gaussian model to obtain the three-dimensional Gaussian points output by the decoder;

[0258] The third fusion feature is obtained by fusing the spherical depth map, the third feature, and the fourth feature.

[0259] In some embodiments, the loss determination module includes:

[0260] The loss determination unit is used to determine the loss value based on the depth loss between the spherical depth map and the depth ground truth in the target, the luminance loss between the luminance value of the rendered result and the luminance ground truth in the target, and the color value between the color value of the rendered result and the color ground truth in the target.

[0261] Based on the same technical concept, this disclosure also provides a rendering device 1300, applied to the aforementioned three-dimensional Gaussian rendering network, such as... Figure 13 As shown, the device includes:

[0262] The acquisition module 1301 is used to acquire the first panoramic image and the second panoramic image;

[0263] The fifth processing module 1302 is used to input the first panoramic image and the second panoramic image into the three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points;

[0264] The second rendering module 1303 is used to render three-dimensional Gaussian points to a specified viewpoint to obtain the rendering result.

[0265] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0266] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0267] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0268] Figure 14 A schematic block diagram of an example electronic device 1400 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0269] like Figure 14 As shown, device 1400 includes a computing unit 1401, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1402 or a computer program loaded from storage unit 1408 into random access memory (RAM) 1403. The RAM 1403 may also store various programs and data required for the operation of device 1400. The computing unit 1401, ROM 1402, and RAM 1403 are interconnected via bus 1404. Input / output (I / O) interface 1405 is also connected to bus 1404.

[0270] Multiple components in device 1400 are connected to I / O interface 1405, including: input unit 1406, such as a keyboard, mouse, etc.; output unit 1407, such as various types of displays, speakers, etc.; storage unit 1408, such as a disk, optical disk, etc.; and communication unit 1409, such as a network card, modem, wireless transceiver, etc. Communication unit 1409 allows device 1400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0271] The computing unit 1401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1401 performs the various methods and processes described above, such as the training / rendering methods for rendering networks. For example, in some embodiments, the training / rendering methods for rendering networks may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1408. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1400 via ROM 1402 and / or communication unit 1409. When the computer program is loaded into RAM 1403 and executed by the computing unit 1401, one or more steps of the training / rendering methods for rendering networks described above may be performed. Alternatively, in other embodiments, computing unit 1401 may be configured by any other suitable means (e.g., by means of firmware) to perform a training method / rendering method for the rendering network.

[0272] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0273] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0274] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0275] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0276] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0277] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0278] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0279] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for training a rendering network, comprising: A spherical depth estimation network based on a 3D Gaussian rendering network processes a reference view and a source view to obtain a spherical depth map of the reference view; The reference view and the source view are panoramic views; The 3D Gaussian model based on the 3D Gaussian rendering network processes the spherical depth map and the reference view to obtain 3D Gaussian points; The three-dimensional Gaussian points are rendered to the target viewpoint to obtain the rendering result; Based on the spherical depth map, the rendering result, and the target ground truth from the target viewpoint, the loss value is determined; The model parameters of the 3D Gaussian rendering network are optimized based on the loss value; The spherical depth estimation network includes a first cost volume prediction network, an initial depth estimation network, and a first fusion module; The spherical depth estimation network based on the 3D Gaussian rendering network processes the reference view and the source view to obtain a spherical depth map of the reference view, including: Based on the first cost volume prediction network, the first rectangular projection map of the reference view and the second rectangular projection map of the source view are processed to obtain the first cost volume; The initial spherical depth is obtained by processing the cube projection map of the reference view based on the initial depth estimation network. The initial spherical depth and the first cost volume are fused based on the first fusion module to obtain the spherical depth map of the reference view.

2. The method according to claim 1, wherein, The first cost volume is obtained by processing the first rectangular projection map of the reference view and the second rectangular projection map of the source view based on the first cost volume prediction network, including: Extract the first feature of the first rectangular projection image and the second feature of the second rectangular projection image; Based on the first feature and the second feature, a first feature matching cost is determined for multiple pixels in the reference view under multiple depth assumptions; the first feature matching cost is determined based on the similarity between the first feature vector of the multiple pixels in the first feature and the second feature vector in the second feature. The first feature matching cost is reduced in dimensionality to obtain the first cost volume.

3. The method according to claim 2, wherein, The step of performing a dimensionality reduction operation on the first feature matching cost to obtain the first cost volume includes: For the first feature matching cost, the mean value is calculated along the dimension of the first feature vector to obtain the first cost volume.

4. The method according to claim 2, wherein, The step of performing a dimensionality reduction operation on the first feature matching cost to obtain the first cost volume includes: The first feature matching cost is processed using a three-dimensional convolutional neural network to obtain the first cost volume.

5. The method according to claim 1, wherein, The process of processing the cube projection map of the reference view based on the initial depth estimation network to obtain the initial spherical depth includes: Depth estimation is performed on the cube projection map to obtain a depth point cloud map; The initial spherical depth is obtained by projecting the depth point cloud onto the sphere of the reference view.

6. The method according to claim 5, wherein, The process of performing depth estimation on the cube projection map to obtain a depth point cloud map includes: Depth estimation is performed on the cube projection map based on a geometric 3D vision network to obtain the depth point cloud map.

7. The method according to claim 1, wherein, The step of fusing the initial spherical depth and the first cost volume based on the first fusion module to obtain the spherical depth map of the reference view includes: By fusing the initial spherical depth and the first cost volume, a first fused feature is obtained; Depth prediction is performed on the first fused feature to obtain the spherical depth map.

8. The method according to claim 7, wherein, The step of performing depth prediction on the first fused feature to obtain the spherical depth map includes: The spherical depth map is obtained by performing depth prediction on the first fused feature based on the first two-dimensional convolutional network.

9. The method according to claim 1, wherein, During the optimization of the 3D Gaussian rendering network, the model parameters of the initial depth estimation network are frozen and remain unchanged.

10. The method according to claim 2, wherein, The 3D Gaussian model based on the 3D Gaussian rendering network processes the spherical depth map and the reference view to obtain 3D Gaussian points, including: The reference view is input into the encoder of the three-dimensional Gaussian model, and the second fusion feature is input into the decoder of the three-dimensional Gaussian model to obtain the three-dimensional Gaussian points output by the decoder; The second fusion feature is obtained by fusing the spherical depth map, the first feature, and the second feature.

11. A method for training a rendering network, comprising: A spherical depth estimation network based on a 3D Gaussian rendering network processes a reference view and a source view to obtain a spherical depth map of the reference view; The reference view and the source view are panoramic views; The 3D Gaussian model based on the 3D Gaussian rendering network processes the spherical depth map and the reference view to obtain 3D Gaussian points; The three-dimensional Gaussian points are rendered to the target viewpoint to obtain the rendering result; Based on the spherical depth map, the rendering result, and the target ground truth from the target viewpoint, the loss value is determined; The model parameters of the 3D Gaussian rendering network are optimized based on the loss value; The spherical depth estimation network includes a second cost volume prediction network, a monocular depth estimation network, and a second fusion module; The spherical depth estimation network based on the 3D Gaussian rendering network processes the reference view and the source view to obtain a spherical depth map of the reference view, including: Based on the monocular depth estimation network, the cube projection map of the reference view is processed to obtain the monocular features extracted by the intermediate layer of the monocular depth estimation network; Based on the second cost volume prediction network, the monocular features, the first rectangular projection map of the reference view, and the second rectangular projection map of the source view are processed to obtain the second cost volume; Based on the second fusion module, a depth prediction is performed on the second cost volume to obtain a spherical depth map of the reference view.

12. The method according to claim 11, wherein, The second cost volume prediction network processes the monocular features, the first rectangular projection map of the reference view, and the second rectangular projection map of the source view to obtain the second cost volume, including: Extract the first feature of the first rectangular projection image and the second feature of the second rectangular projection image; The monocular feature is fused into the first feature to obtain the third feature; and the monocular feature is fused into the second feature to obtain the fourth feature; Based on the third feature and the fourth feature, a second feature matching cost is determined for multiple pixels in the reference view under multiple depth assumptions; the second feature matching cost is determined based on the similarity between the third feature vector of the multiple pixels in the third feature and the fourth feature vector in the fourth feature. The second feature matching cost is reduced in dimensionality to obtain the second cost volume.

13. The method according to claim 12, wherein, The step of performing a dimensionality reduction operation on the second feature matching cost to obtain the second cost volume includes: For the second feature matching cost, the mean operation is performed on the third feature vector dimension to obtain the second cost volume.

14. The method according to claim 12, wherein, The step of performing a dimensionality reduction operation on the second feature matching cost to obtain the second cost volume includes: The second feature matching cost is processed using a second three-dimensional convolutional neural network to obtain the second cost volume.

15. The method according to claim 12, wherein, The 3D Gaussian model based on the 3D Gaussian rendering network processes the spherical depth map and the reference view to obtain 3D Gaussian points, including: The reference view is input into the encoder of the three-dimensional Gaussian model, and the third fusion feature is input into the decoder of the three-dimensional Gaussian model to obtain the three-dimensional Gaussian points output by the decoder; The third fusion feature is obtained by fusing the spherical depth map, the third feature, and the fourth feature.

16. The method according to any one of claims 1-15, wherein, The step of determining the loss value based on the spherical depth map, the rendering result, and the target ground truth from the target viewpoint includes: The loss value is determined based on the depth loss between the spherical depth map and the depth ground truth in the target ground truth, the luminance loss between the luminance value of the rendered result and the luminance ground truth in the target ground truth, and the color value of the rendered result and the color ground truth in the target ground truth.

17. A rendering method, applied to a 3D Gaussian rendering network trained by the method as described in any one of claims 1-16, the method comprising: Obtain the first and second panoramic images; The first panoramic image and the second panoramic image are input into the three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points; The three-dimensional Gaussian points are rendered to a specified viewpoint to obtain the rendering result.

18. A training apparatus for rendering a network, comprising: The first processing module is used to process the reference view and the source view based on the spherical depth estimation network of the 3D Gaussian rendering network to obtain the spherical depth map of the reference view. The reference view and the source view are panoramic views; The second processing module is used to process the spherical depth map and the reference view based on the three-dimensional Gaussian model of the three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points; The first rendering module is used to render the three-dimensional Gaussian points to the target viewpoint to obtain the rendering result; The loss determination module is used to determine the loss value based on the spherical depth map, the rendering result, and the target ground truth from the target viewpoint; An optimization module is used to optimize the model parameters of the 3D Gaussian rendering network based on the loss value; The spherical depth estimation network includes a first cost volume prediction network, an initial depth estimation network, and a first fusion module; The first processing module includes: The first processing unit is configured to process the first rectangular projection map of the reference view and the second rectangular projection map of the source view based on the first cost volume prediction network to obtain the first cost volume. The second processing unit is used to process the cube projection map of the reference view based on the initial depth estimation network to obtain the initial spherical depth; The fusion unit is used to perform fusion processing on the initial spherical depth and the first cost volume based on the first fusion module to obtain a spherical depth map of the reference view.

19. The apparatus according to claim 18, wherein, The first processing unit is specifically used for: Extract the first feature of the first rectangular projection image and the second feature of the second rectangular projection image; Based on the first feature and the second feature, determine the first feature matching cost of multiple pixels in the reference view under multiple depth assumptions; The first feature matching cost is determined based on the similarity between the first feature vector of the plurality of pixels in the first feature and the second feature vector in the second feature; The first feature matching cost is reduced in dimensionality to obtain the first cost volume.

20. The apparatus according to claim 19, wherein, The first processing unit is specifically used for: For the first feature matching cost, the mean value is calculated along the dimension of the first feature vector to obtain the first cost volume.

21. The apparatus according to claim 19, wherein, The first processing unit is specifically used for: The first feature matching cost is processed using a three-dimensional convolutional neural network to obtain the first cost volume.

22. The apparatus according to claim 18, wherein, The second processing unit is specifically used for: Depth estimation is performed on the cube projection map to obtain a depth point cloud map; The initial spherical depth is obtained by projecting the depth point cloud onto the sphere of the reference view.

23. The apparatus according to claim 22, wherein, The second processing unit is specifically used for: Depth estimation is performed on the cube projection map based on a geometric 3D vision network to obtain the depth point cloud map.

24. The apparatus according to claim 18, wherein, The fusion unit is specifically used for: By fusing the initial spherical depth and the first cost volume, a first fused feature is obtained; Depth prediction is performed on the first fused feature to obtain the spherical depth map.

25. The apparatus according to claim 24, wherein, The fusion unit is specifically used for: The spherical depth map is obtained by performing depth prediction on the first fused feature based on the first two-dimensional convolutional network.

26. The apparatus according to claim 18, wherein, During the optimization of the 3D Gaussian rendering network, the model parameters of the initial depth estimation network are frozen and remain unchanged.

27. The apparatus according to claim 19, wherein, The second processing module is specifically used for: The reference view is input into the encoder of the three-dimensional Gaussian model, and the second fusion feature is input into the decoder of the three-dimensional Gaussian model to obtain the three-dimensional Gaussian points output by the decoder; The second fusion feature is obtained by fusing the spherical depth map, the first feature, and the second feature.

28. A training apparatus for rendering a network, comprising: The first processing module is used to process the reference view and the source view based on the spherical depth estimation network of the 3D Gaussian rendering network to obtain the spherical depth map of the reference view. The reference view and the source view are panoramic views; The second processing module is used to process the spherical depth map and the reference view based on the three-dimensional Gaussian model of the three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points; The first rendering module is used to render the three-dimensional Gaussian points to the target viewpoint to obtain the rendering result; The loss determination module is used to determine the loss value based on the spherical depth map, the rendering result, and the target ground truth from the target viewpoint; An optimization module is used to optimize the model parameters of the 3D Gaussian rendering network based on the loss value; The spherical depth estimation network includes a second cost volume prediction network, a monocular depth estimation network, and a second fusion module; The first processing module includes: The third processing unit is used to process the cube projection map of the reference view based on the monocular depth estimation network to obtain the monocular features extracted by the intermediate layer of the monocular depth estimation network. The fourth processing unit is used to process the monocular features, the first rectangular projection map of the reference view, and the second rectangular projection map of the source view based on the second cost volume prediction network to obtain the second cost volume; The prediction unit is used to perform depth prediction on the second cost volume based on the second fusion module to obtain a spherical depth map of the reference view.

29. The apparatus according to claim 28, wherein, The fourth processing unit is specifically used for: Extract the first feature of the first rectangular projection image and the second feature of the second rectangular projection image; The monocular feature is fused into the first feature to obtain the third feature; and the monocular feature is fused into the second feature to obtain the fourth feature; Based on the third feature and the fourth feature, a second feature matching cost is determined for multiple pixels in the reference view under multiple depth assumptions; the second feature matching cost is determined based on the similarity between the third feature vector of the multiple pixels in the third feature and the fourth feature vector in the fourth feature. The second feature matching cost is reduced in dimensionality to obtain the second cost volume.

30. The apparatus according to claim 29, wherein, The fourth processing unit is specifically used for: For the second feature matching cost, the mean operation is performed on the third feature vector dimension to obtain the second cost volume.

31. The apparatus according to claim 29, wherein, The fourth processing unit is specifically used for: The second feature matching cost is processed using a second three-dimensional convolutional neural network to obtain the second cost volume.

32. The apparatus according to claim 29, wherein, The second processing module is specifically used for: The reference view is input into the encoder of the three-dimensional Gaussian model, and the third fusion feature is input into the decoder of the three-dimensional Gaussian model to obtain the three-dimensional Gaussian points output by the decoder; The third fusion feature is obtained by fusing the spherical depth map, the third feature, and the fourth feature.

33. The apparatus according to any one of claims 18-32, wherein, The loss determination module includes: The loss determination unit is used to determine the loss value based on the depth loss between the spherical depth map and the depth ground truth in the target ground truth, the luminance loss between the luminance value of the rendered result and the luminance ground truth in the target ground truth, and the color value of the rendered result and the color ground truth in the target ground truth.

34. A rendering apparatus, applied to a three-dimensional Gaussian rendering network trained by the apparatus as described in any one of claims 18-33, the apparatus comprising: The acquisition module is used to acquire the first panoramic image and the second panoramic image; The fifth processing module is used to input the first panoramic image and the second panoramic image into the three-dimensional Gaussian rendering network to obtain three-dimensional Gaussian points; The second rendering module is used to render the three-dimensional Gaussian points to a specified viewpoint to obtain the rendering result.

35. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-17.

36. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-17.

37. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-17.

Images